Compound AI: Thinking in Systems, Not Models

AI today isn't about models. Or rather, it's not just about models. Modern AI systems are often complex, modular systems made up of multiple interconnected components. Some components contain or connect to AI models, while others perform non-AI functions. This is known as Compound AI.

Platforms like deepset Cloud are uniquely equipped to meet the needs of AI teams looking to design, build, test, and deploy products built with Compound AI.

What is Compound AI?

Compound AI is a design principle championed by the Berkeley AI Research (BAIR) lab, as described in their blog post about “the shift from models to Compound AI systems.” 

"We define a Compound AI System as a system that tackles AI tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools." – BAIR.

This approach is not new. Products and tools that use AI have long been designed as modular systems, graphs, or pipelines. That’s why we’ve built our own generative AI orchestration tools as modular toolboxes that provide the building blocks to create and maintain modern AI tools. These building blocks or components are then combined into increasingly complex pipelines or graphs.

Components: Building blocks of Compound AI

Components typically have one well-defined task. This means that they can be evaluated and improved in isolation from the rest of the system. It also makes it easy to replace components when, for example, a newer, faster, or cheaper version becomes available. As a result, Compound AI systems are more flexible than static configurations.

Composability: the de facto standard in AI today

The basic retrieval augmented generation (RAG) setup, shown above, is a common application of Compound AI today. RAG combines a retrieval component with a generative component to create a system that is both creative and grounded in a controlled database. But this basic RAG setup (which can be extended indefinitely) is just one of many Compound system designs. A few examples:

• Advanced RAG setups still use components for retrieval and generation, but they also contain other components, for instance to rerank the retriever’s results according to some product-specific criteria, before passing them on to the LLM.
• Agents use multiple highly specialized components that are invoked by the controlling system as needed. For example, an agentic system might include a Web browser and access to a proprietary database. Depending on the context required to perform a task, the agent will invoke the browser, the database, or both.
• Hybrid retrieval is another example of a popular modular setup that has been in use for years. It uses two components to retrieve information, most commonly one that is keyword-based and one that understands semantics. In this setup, the two retrievers can complement each other when one falls short (for example, when the semantic model doesn't know the language of a query).
• Multimodality allows us to process different types of data – such as text, images, video files, and spreadsheets – in a single system, increasing the insights that can be gained. Composability allows us to easily combine models optimized for different data types in a single system.
• Finally, the concept of modularity can also be applied to indexing pipelines: These are the systems that preprocess incoming files and prepare them for semantic search with a language model. Thanks to Compound AI, these pipelines can be extended at will. For example, you could add a component that parses documents for names and locations and adds this information to the database as metadata during indexing (this is commonly known as “named entity recognition” or NER).

Advantages of using Compound AI

The LLM space is evolving rapidly. New, more powerful models – both commercial and open source – are emerging all the time. To truly harness the generative and computational power of these models, companies must learn how to use them. This means not only demonstrating the power of generative AI technology, but also integrating it into products in a seamless and value-adding way.

Composability lets you add models and other components to your system as you need them. And it lets you remove the parts you no longer need, or swap them out for a better version. Finally, it lets you combine complex, billion-parameter language models with other, simpler components, for example a keyword-based retriever like BM25.

During evaluation, you can look at each of your components in isolation, breaking down the gargantuan task of evaluating a complex AI system into smaller, more manageable subtasks. The composability paradigm also allows you to reuse models in different parts of your organization. This will become increasingly important as more companies implement comprehensive strategies for building with AI.  

A new standard in AI

In a world obsessed with models, Compound AI sets a new standard. When you build a system with Compound AI, you need to start with the design, not just the models. This encourages strategic thinking at all stages of your project.

The advent of Compound AI may require some realignment in your organization. The AI team is first and foremost a product team: it seeks to create products or tools that make life easier for its users. In this mindset, the AI model itself is just a technology – one that has its skills and limitations and will most likely be replaced at some point. This mindset shift is often the biggest hurdle to designing AI systems efficiently – but it is one that, if taken seriously, can be easily overcome.

To learn more about the product mindset, read our O'Reilly report on "LLM Adoption in the Enterprise."

Compound AI in deepset Cloud

After building countless Compound AI systems for our early customers, we realized that what modern AI teams need is a platform that makes it easy to try different models in different Compound AI setups. Such a platform should also automatically handle versioning and logging in the background. deepset Cloud was born.

In deepset Cloud, you can choose from thousands of different language models to iteratively build the system that best serves your custom tool or product. Once your system is in production, deepset Cloud lets you continuously evaluate and monitor it, so that you can improve or replace models that no longer perform well.

The various workflows we offer in the platform are designed to follow best practices in AI product development, allowing your AI team to collaborate and learn together as they progress through their AI development journey. 

Start building with Compound AI

Curious to try out the platform and build real products in no time? Schedule a demo today.