Talk to Your Docs, RAG Style

Technical documentation is critical to software delivery. But even in a well-structured and designed documentation site, it can be difficult to find the right information, especially when relying solely on keyword searches. It can also be confusing, especially for new users, to know where to find something in the navigation structure.

The solution? Connect your technical documentation to an LLM-powered pipeline that acts as a technical assistant for your users. All they have to do is ask their questions in natural language and get a helpful answer, including pointers to the relevant documentation pages if they want to learn more.

We recently added a pipeline like this to our platform. It accepts questions within the platform UI itself and provides an answer within seconds. It is factual, helpful and does not hallucinate.

In a rather meta setup, we used our LLM platform, deepset Cloud, to design and deploy the technical assistant for deepset Cloud's documentation. We were able to build a prototype in a matter of hours, have it ready for production after another day of teamwork, and release the product shortly thereafter. In this internal case study, we'll walk you through the process of using deepset Cloud to build a product from the experimental stage to production – even with little to no coding experience.

The use case

deepset Cloud is an LLM platform for AI teams. It combines an intuitive user interface for designing and testing LLM pipelines with a powerful, scalable backend. 

While the interface for setting up a prototype is clean and straightforward, the options for customizing the pipeline are so extensive that the documentation for deepset Cloud is quite comprehensive, covering topics from choosing the right language model to running experiments and evaluating user feedback.

Thanks to the success of large language models and the desire across industries to build LLM-powered products, we've had a number of new AI teams join deepset Cloud in recent months. While our solution engineers – a team of LLM experts with extensive experience in building AI products – are there to help our customers every step of the way, we also wanted to make it easier for all of our users to get answers immediately, even when a solution engineer is not available.

So Agnieszka, our lead technical writer, set out to build an "Ask Your Docs" prototype during one of our Hacky Fridays – a monthly one-day hackathon that allows us to work on projects outside of our normal workload.

Setting up a prototype

Thanks to the guided workflow in deepset Cloud, building the prototype was a breeze. In the guided workflow interface, users can choose between different types of LLM tasks. deepset Cloud then creates a template with the pipeline design that fits the task – in the case of our documentation AI, a retrieval augmented generation (RAG) pipeline with hybrid retrieval.

This initial pipeline design can be easily customized by adding or removing components, but Agnieszka found that the default setup worked well for her use case. It answered questions based on the documents quickly and reliably – with a few hallucinations here and there – and looked promising overall.

The most time-consuming challenges were related to the data pre-processing itself: Agnieszka had to figure out how to export the data from the documentation platform, break it up into suitable chunks, and process the links it contained so that they would be clickable in the AI’s answer. To do this, she asked internally for help and brought a solution engineer on board.

Smoothing the edges

As a solution engineer here at deepset, Rob has worked on many customer-facing projects in the past. When he joined Agnieszka's project after the hackathon, a fully functional pipeline was already in place. That's not surprising, he says: “deepset Cloud actually shifts the attention from complicated technical issues like scaling infrastructure to other matters.”

Rob implemented a solution for the links by adding them to the documents' metadata. In this new setup, the pipeline extracted and added relevant links from the documentation to the generated response automatically.

He also used his experience as an expert LLM engineer to refine the prompt and remove any remaining hallucinations. "Prompt engineering is really doable for anyone," says Rob. "There are just a few tips and tricks to keep in mind – like repeating things several times to make sure the model gets it, and just being hyper-literal. LLMs are not good at reading subtext."

Exploring the prompt explorer

One of deepset Cloud’s unique value propositions is the prompt explorer: a sandbox environment that lets you design the perfect prompt. In this environment, users can see how different prompts perform side-by-side in up to three different pipelines. 

Agnieszka found it useful to compare all sorts of different setups: from chunk length to the actual LLM used in the final project (she eventually settled on OpenAI’s GPT-3.5-turbo model – GPT-4 was much too slow).

From prototype to production

One aspect of AI-powered tools that is often underestimated is the importance of user interface design. For this project, front-end lead Eren designed and developed a pop-up component and connected it to the deepset Cloud API. It sits directly within the platform's user interface and can be expanded whenever the user gets stuck and needs help from the documentation. 

This seamless integration means that the AI solution is a win all the way around:

• For our technical writers, who can be confident that their readers will find the information they need, when they need it, without the hassle of keyword searches or worrying about the navigational structure of the documentation site.
• For our users, who can get exactly the information they need through naturally phrased questions, without having to leave the deepset Cloud platform and browse the documentation site.
• For our solution engineers, who have their hands free to help more customers with more difficult problems.

Since going live, the new interface has already proven to be a major time saver, helping customers navigate the documentation in a more intuitive, straightforward way and rely less on us to answer their questions.

It's especially helpful for questions where the answer requires a lot of context, because it summarizes the necessary technical information based on the relevant documentation pages – and links to those pages in case a user wants to learn more. This solution is elegant, intuitive, and saves a lot of time.

deepset Cloud: Start building today, have a proof of concept tomorrow

As the lead technical writer, Agnieszka knows the platform inside and out. But this was the first time that she had implemented an entire product from start to finish in deepset Cloud. She loved the ease of use: "It worked so well right out of the box," she says. "I was able to create a decent prototype on my own in no time.”

Want to learn more about how deepset Cloud makes life easier for AI teams and the AI adoption process more feasible? Read our detailed blog post on deepset Cloud: an LLM platform for AI teams.