Milos Rusic

At deepset, our mission has always been to help enterprises adopt and gain value from Natural Language Processing and Generative AI. We've designed our open-source LLM framework, Haystack, to evolve alongside the AI ecosystem, seamlessly integrating the latest models, services, and technologies into a comprehensive framework.

Today, we are thrilled to announce our partnership with NVIDIA on their latest NIM launch. This collaboration brings together the power of Haystack and NVIDIA NIM, enabling developers across all industries to build and deploy production-ready AI applications at scale, in any environment.

NVIDIA NIM is a globally available platform that empowers developers to deploy AI applications with integrations across every layer of the AI ecosystem. With Haystack by deepset integrated with NIM, you can now:

Leverage Proven, Production-Ready AI Frameworks and Architectures: Haystack and NIM offer a robust foundation for your AI applications, ensuring you have the tools you need to succeed from development to deployment.

Ensure a Performant User Experience: Handle high-volume query and indexing workloads efficiently, providing a smooth and responsive experience for your users.

Flexibly Prototype and Deploy: Whether you're working in the cloud, on-premise, or in air-gapped environments, Haystack and NIM offer the flexibility you need to prototype and deploy your AI applications.

Step-by-Step Guide to Building with NVIDIA NIM & Haystack

We are proud to be featured among the leading MLOps partners in NVIDIA's latest announcement and for Haystack to be integrated into the NIM ecosystem. 

To help you get started, we've created a comprehensive guide on building and deploying intelligent, production-ready LLM applications using Haystack and NVIDIA NIM. Check out our detailed blog post here.

Learn how deepset and NVIDIA’s new partnership empowers enterprises to launch and scale powerful AI applications.

Build Powerful AI Applications with Haystack by deepset & NVIDIA NIM

LLMs can significantly improve products and internal tools. However, many organizations are overwhelmed by the rapid growth of generative AI and the operational changes it requires. The biggest challenge is successfully putting together the right team – the AI team – to build and deploy mission-critical applications. In our experience, the recipe for success is

Understanding the nature of an AI team and its unique composition, skill set, and role in the wider organization.

Understanding the importance of a good team lead.

Read this blog post for a rundown of the most important points. For more details, check out our white paper on "Leading a successful AI team."

Building with AI means bridging the gaps between data science, software development, user needs, and business requirements. This challenge is more human than technical. As organizations increasingly adopt AI technologies, the need for cross-functional, product-focused teams is becoming more and more apparent. However, there is often a lack of clarity about the characteristics and leadership styles required for these teams to succeed.

An AI team is a diverse group collaborating to create AI-powered products or tools. This dynamic team typically includes developers, data scientists, AI/ML engineers, and non-technical domain experts. The team leader plays a crucial role in the project's success, combining vision with practical skills.

The AI team leader is critical to managing the team and the process, driving rapid iteration and user centricity. They need to:

Communicate effectively with technical and non-technical stakeholders.

Facilitate communication between team roles.

Coming together to build products that serve end users well requires universal skills, such as a strong ability to communicate with people from other departments and different levels of technical expertise. Not everyone has these skills, so it is the responsibility of the team leader to facilitate communication across roles. They are also responsible for maintaining the product vision and making critical decisions. Here are the other top 7 skills that are critical to the success of an AI project:

Your product must solve real user problems. You need to understand user behaviors, needs, and what generative AI can realistically do. For instance, an AI team developing a customer service chatbot needs to know common customer inquiries, pain points, and how to make conversations feel natural to create a truly helpful tool.

Regularly testing and refining prototypes based on feedback ensures that the product meets user needs and expectations.

Teams must understand how LLMs work, their differences, and how to combine them effectively. They must also monitor and evaluate models continuously.

Involving domain experts throughout the product development process ensures that the AI tools meet the specific needs of professionals like lawyers, doctors, or accountants.

Considering how the AI product fits into user workflows and ensuring it is easy to use and integrate with existing systems is critical.

This involves understanding database management, cloud computing, and cybersecurity to handle data efficiently, securely, and ensure the robustness of AI applications.

The focus should be on skills rather than specific roles. Product-focused, learning-oriented individuals who work well in teams are more valuable than specific job titles. This makes the team more flexible and innovative. For example, a team member with a background in psychology might bring invaluable insights into user behavior, even without traditional tech credentials. By prioritizing skills, you create a more adaptable team that can tackle diverse challenges in the rapidly evolving AI landscape.

To achieve success, AI teams must concentrate on developing products that set them apart from competitors. They need to stay focused on their core mission, avoiding distractions such as model training, system deployment, or production monitoring – tasks now categorized as the 'undifferentiated heavy lifting' of AI.

Because AI teams combine diverse talents from individuals used to different tools and terminologies, it's important to help them communicate and share information effectively.

Effective AI teams need tools that streamline repetitive tasks and that allow them to collaborate on an AI product with different levels of technical load. Enterprise AI production platforms help design and test AI pipelines, optimize models, build continuous delivery infrastructure, and monitor applications in production.

deepset Cloud: The AI Team's Development Platform

Our LLM platform deepset Cloud offers a comprehensive approach to AI development, from prototype to production. It is an AI development platform designed and built with the AI team's unique requirements in mind. The advantages of deepset Cloud at a glance:

Iterate on your prototype in rapid development cycles and get a production-ready AI application to market faster.

Keep your team lean: significantly reduce costs associated with full-time AI consultants and LLM Ops (data integration/system build). While exact savings can vary, organizations have reported cost reductions of up to 30-40% by streamlining these processes with deepset Cloud.

Be confident in the quality of your product thanks to market-leading gen AI observability tools and transparent tracking of LLM groundedness, hallucinations, and many other metrics.

Building a successful AI team requires a diverse skill set, effective collaboration, and the right tools. To get started:

Assess your current talent and identify skill gaps.

Invest in training programs to upskill existing team members.

Recruit strategically to fill critical roles, focusing on skills over traditional job titles.

Implement collaborative tools and platforms like deepset Cloud to streamline your AI development process.

Promote a culture of continuous learning and adaptation to keep pace with rapid AI advancements.

By understanding and implementing these strategies, your organization can build AI applications that are not only innovative but also aligned with user needs and business goals. The journey to AI excellence starts with the right team – are you ready to assemble yours?

To learn more about what an AI team actually looks like, download our free white paper, Leading a Successful AI Team.

A blueprint for assembling and leading AI talent

Building Product-Focused AI Teams

With Haystack, you can set up a fully functional question answering system (QA) in under 30 minutes. But did you know that you could add nodes to your pipeline to leverage all kinds of Transformer-based language models? For example, you could include a module for sentiment analysis, or one that summarizes the pipeline output.

In this post, we show you how to benefit from a wide range of pre-trained Transformer models. As always, the goal is to boost both your pipeline efficiency and the quality of its output. Specifically, we’ll be looking into three different nodes: the Summarizer, the Classification node, and the Generator. Let’s jump right in!

Creating Metadata with the Classification Node

The Classification node is relatively new and we’re particularly excited about it. It accepts a machine learning model that’s been pre-trained on any kind of classification task: Be it sentiment analysis, named entity recognition or hate speech detection.

It complements our well-established Query classifier. However, rather than classifying queries, this node classifies answer candidates selected by the Retriever.

Recall that the Query classifier has two outgoing edges and routes queries to one of the two, depending on the assigned label. By contrast, the Classification node has only one outgoing edge. It predicts labels for incoming documents and adds them to the documents’ metadata.

We’ve previously shown how you can use metadata to add a filter to your pipeline and improve its speed and accuracy. But annotating large amounts of data is expensive, and we may not have the exact metadata for our filter of choice. In such cases, the Classification node allows us to integrate a machine learning model to classify our documents and generate our own metadata — like a label that tells us whether a document is positive, negative or neutral in tone.

To use the Classifier, we need to install the newest version of Haystack from master:

pip install git+https://github.com/deepset-ai/haystack.git

In this example, we’ll be working with a Classifier for sentiment analysis — the task of labeling a given text’s sentiment or attitude. Is it positive or negative, or somewhere in between? We choose a pre-trained, binary sentiment classifier from the Hugging Face Model Hub and pass its name to FARMClassifier:

from haystack.classifier import FARMClassifier

classifier_model = 'textattack/bert-base-uncased-imdb'
classifier = FARMClassifier(model_name_or_path=classifier_model)

Next, we initialize the pipeline as usual, this time adding the Classifier node right after the Retriever node:

pipeline = Pipeline()
pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"])
pipeline.add_node(component=classifier, name='Classifier', inputs=['Retriever'])

For the practical examples in this post, we’ll be working with the “Toys and Games” slice from the large Amazon Reviews dataset. We’ve read the reviews into the document store, keeping the item IDs as metadata. For the next search, we only want to look at reviews of this slurpee maker, so we pass a filter to our pipeline:

filter = {'item_id': ['B007X9DRWO']}
result = pipeline.run(query='How is the quality of this product?', filters=filter, top_k_retriever=30)

The pipeline returns the result to our query as a nested dictionary. Under the ‘documents’ key, we find the answer documents, each having received a label from the classification node: ‘LABEL_1’ for a positive sentiment, or ‘LABEL_0’ for negative. Let’s look at the first positively classified review:

documents = result['documents']
[doc.text for doc in documents if doc.meta['classification']['label'] == 'LABEL_1'][0]

>>> "Children will love this toy. As we all know, kids (and a good number of adults as well, I might add) love Slurpees. And this product, which enables the children to create their own drink, is a sure winner. Cleanup isn't much, either, so this Slurpee Maker comes recommended - would definitely make a nice gift."

[doc.text for doc in documents if doc.meta['classification']['label'] == 'LABEL_0'][0]

>>> 'I wish I could rate this higher because I love Slurpee brand. The plastic this product is made from is so cheap, it breaks easily and since it\'s for kids - described as a "toy" it should be more durable. They made it completely disassemble, which is great for cleaning, but makes it a mess to wash, dry, and store, besides the fact that when you are moving it or trying to use it it keeps falling apart. You must have salt to make the contraption work and kids get tired of turning the pump to make it work. The'

For the sake of this example, we’ve ignored the fact that the Amazon Reviews data actually comes with a sentiment label attached. If you’re working with data that has no such information, adding a classification node to your pipeline can provide valuable insights into the nature of your results.

Trim Down Your Reading Time with the Summarizer

In some cases, we won’t find the best answer to a query in a single passage, but rather by aggregating several documents. This reality motivated us to create the Summarizer node. It uses a pre-trained model to create a summary of the documents returned by the Retriever.

To build a summarization system, we’ll combine a Retriever and a Summarizer in a pipeline. When initializing the Summarizer, we specify a minimum and maximum number of words per summary. Setting the generate_single_summary parameter to “True” gives us one summary of all the returned documents, while setting it to “False” creates one summary per document:

from haystack.summarizer import TransformersSummarizer

summarizer = TransformersSummarizer(model_name_or_path='t5-large', min_length=10, max_length=300, generate_single_summary=True)

The ready-made SearchSummarizationPipeline class allows us to set up the entire pipeline in one line:

pipeline = SearchSummarizationPipeline(retriever=retriever, summarizer=summarizer)

Let’s create a summary for this card game for kids. When working with the Summarizer, we’ll typically want to use keyword queries rather than natural language questions:

filter = {'item_id': ['B0039S7NO6']}
result = pipeline.run(query='preschooler suitability', filters=filter, top_k_retriever=5)

This time, the ‘result’ dictionary is a list with one document. Let’s look at the summary we obtained for our query which we find under the ‘text’ key:

result['documents'][0].text

>>> "'this game is surprisingly fun, and entertains for quite a while. even my three-year-old plays almost as well as the rest of the family. the game is for all genders and ages 4+."

To confirm that this result is indeed a newly generated summary of all of the documents retrieved in response to our query, let’s have a look at the answer candidates. They are stored in concatenated format as part of the metadata, under the ‘context’ key:

result['documents'][0].meta['context']

>>> "The best thing about this game is that even a 5 year old can play it and explain it to friends without adult supervision.  Fun, quick and easy to pack. This is a fun game for a group of friends or just our kids (ages 6-10) and us! Easy to play and understand. I bought this game for my daughters (ages 3 & 5) for Christmas after reading several reviews.  It's a simple game, you simply have to figure out which figures match on two game cards, but it's surprisingly fun, and entertains for quite a while.  What I like about it is that even my three-year-old plays almost as well as the rest of the family. This is a great game (with multiple playing options), good for a quick five minutes, or playing variation after variation.  My 4 year old enjoys the game, as do my husband and I.  This game is for all genders and ages 4+.  Great birthday gift to have waiting in the closet. Fun, simple, and inexpensive game for children of all ages. The younger kids (3-5) can play cooperatively and older kids 6+ can play more competitively. "

The Summarizer is sensitive to the order of the documents it receives. That’s why adding a Ranker node to the pipeline can lead to better summaries. It sorts the documents returned by the Retriever according to their relevance to the query. Visit our documentation page to learn more about the Ranker.

Get Answers Written Just for You with the Generator

Transformer-based language models are already impressing us with their ability to interpret natural language. But it’s something else to see them create statements from scratch. With the AnswerGenerator node, you can add this capability to your question answering pipeline.

Unlike the other models mentioned in this article, the Generator only works in tandem with the dense passage retrieval (DPR) method. Make sure to initialize the DPR Retriever first, as it needs to be passed to both the Generator and the pipeline object:

from haystack.generator.transformers import RAGenerator
from haystack.pipeline import GenerativeQAPipeline
generator = RAGenerator(
model_name_or_path="facebook/rag-sequence-nq",
  retriever=dpr_retriever,
  top_k=1,
  min_length=2)
pipeline = GenerativeQAPipeline(generator=generator, retriever=dpr_retriever)

Because the review dataset is inherently opinion-based, we’ll ask our system a subjective question:

result = pipelines.run(query='What are the best party games for adults?', top_k_retriever=20)
result['answers'][0]['answer']

>>> "wit's & wagers & darts"

Sounds like a fun evening! A closer look at the retrieved documents (which are too long to display here) reveals that all the reviews spelled “Wits & Wagers” correctly. The misspelling in the answer then confirms that, rather than simply extracting the game’s name from one of the documents, the node generated it from scratch.

If you’d like to read more on this topic, check out the following article on long-form question answering (LFQA) by Vladimir Blagojevic.

As we’ve seen, Haystack goes far beyond simple question answering systems. But there’s more: did you know that you could also use the Question Generator to generate questions from scratch? That comes in handy when creating new QA datasets! We’re preparing a full-length tutorial on this node, so stay tuned.

You could also add a multilingual aspect to your QA system. What if your database were in English, but you wanted your users to be able to ask questions in Arabic, French, or Hindi? No problem! Simply add a TranslationWrapper to your pipeline.

Explore Question Answering and More in Haystack

Want to learn more about what Haystack’s natural language processing tools can do for you?

Head over to our GitHub repository. If you like what you see, give us a star :)

Sentiment classification, summarization and even natural language generation can all be part of your question answering system

With Haystack, you can set up a fully functional question answering system (QA) in under ten minutes. Did you know you could add more nodes to build something more powerful?

Build Powerful AI Applications with Haystack by deepset & NVIDIA NIM

What is NVIDIA NIM?

Step-by-Step Guide to Building with NVIDIA NIM & Haystack

More to read

Building Product-Focused AI Teams

Beyond ‘Vanilla’ Question Answering