Announcing deepset Cloud

Natural language processing (NLP) is more powerful than ever. The astonishing abilities of Transformer-powered systems have inspired visions of language as the primary interface to data. As a result, businesses are increasingly looking to integrate NLP into their own products.

But while pre-trained models like BERT are freely available, setting up a working NLP application is not so simple. It involves choosing the right models for your use case, implementing a database and backend infrastructure, and setting up a demo version with a user interface.

Finally, you need to bring the system to production, where it can benefit the end user. That’s why we’re excited to announce deepset Cloud — a unified platform that takes care of the entire NLP life cycle. With deepset Cloud, you can:

• Unite your data and NLP pipelines in a central location.
• Experiment with different pipeline configurations and track their performance, to identify the best setup for achieving your goals.
• Build fully functional NLP applications within minutes, and use them to build proof-of-concept or production systems.
• Collect user feedback and use it to improve your system.
• Integrate your NLP pipeline seamlessly into your final product. 
• Automatically scale GPU resources depending on the traffic to your application.

Before we cover in detail what deepset Cloud is, who it is for and exactly how to use it, let us briefly explain what motivated us to create deepset Cloud. 

deepset — Who We Are

In June 2018, we founded deepset to bring NLP to any application. Working with our first enterprise customers, we soon identified semantic search and information retrieval as central technologies that helped those companies make sense of their data. Consequently, we started building the open source Haystack framework for modular NLP in Python — to address those use cases, and ultimately enable more and more NLP-powered business applications.

The Haystack framework focuses on two things: modularity and usability. With Haystack, you can plug different Transformer models from the Hugging Face model hub into a question answering or semantic search pipeline, and connect it to your own database to start querying your documents. What’s more, it’s easy to add different nodes to your pipeline — like summarizers, classifiers, or translators.

But most central to our mission is to make NLP available to everyone. That’s why we place such a high value on ease of use. For example, it’s what motivated us to build a free annotation tool to assist teams in creating new labeled data and training their own models.

Lately, we’ve seen that many teams struggle with different aspects of implementing an NLP system — understandably, given the complexity of the task. To help teams address these challenges, we embarked on building deepset Cloud — a unified NLP platform that takes the ease of use and modularity of Haystack and extends it to cover the full cycle of NLP application development.

The Challenges of Adopting NLP in Enterprises

Thanks to the open source practice of model sharing, as well as compression techniques like model distillation, more and more teams can benefit from the advances in NLP. For example, you could incorporate a semantic search interface on your website or within your app, a question answering API or an NLP-enhanced chatbot.

But there are major obstacles to implementing an NLP project in practice. While Haystack makes it easy to set up a working QA pipeline on top of your own documents, bringing an NLP system to production can quickly become complicated.

We’ve often seen teams build a promising prototype system, only to be overwhelmed by the steps required to get the system into a production environment and into the hands of customers while meeting scalability and compliance requirements. Common problems are:

• With all the models out there, how do you find the most suitable combination for your own use case? Keeping track of different system configurations and their performance can quickly get daunting.
• The early version is up and running in a Jupyter notebook, but how do you get it into a presentable format where people can properly interact with it via a UI? 
• Your demo is a success with your teammates, but how do you bring it to production? How does one deploy an NLP pipeline to the cloud? What will the operational costs be?
• How do you incorporate end-user feedback into the pipeline building cycle?
• How can you evaluate a configuration meaningfully and manage the process of creating evaluation data and experimenting with it?

Implementing an NLP-powered system can quickly snowball into a larger project that consumes many of your team’s resources.

deepset Cloud offers production-ready tools and best practices for NLP-based product development, to help teams succeed fast in building NLP applications.

Introducing: deepset Cloud

deepset Cloud is here to make your team’s life much easier at every step of the NLP implementation cycle. In addition to hosting your files and evaluation data, deepset Cloud’s intuitive online interface lets you design, test, scale and monitor your semantic search or question-answering system — all on one platform. Let’s take a closer look at how that looks in practice.

Set up different pipeline configurations

Currently, you can build semantic document search and question-answering pipelines in deepset Cloud. With so many Transformer models out there, you will probably want to experiment with different setups. In deepset Cloud, under “Pipelines,” you can create as many different configurations as you like. These will be connected automatically to your files stored under “Data.”

If you already know which pipeline design works for you, you can use the YAML editor. Otherwise, you might want to create different pipeline designs in a notebook. In this setting, you can directly evaluate the performance of your pipelines to test and compare them against each other. A low-code option for creating pipelines with a drag-and-drop interface will be available soon, in the form of the “Visualizer.”

Evaluate and track your pipeline performance

Keeping track of different system configurations can get confusing quickly. In deepset Cloud, you’ll be able to track your experiments and keep score of how well a given configuration performed on your evaluation dataset.

Performance metrics provide an overall evaluation of your systems’ accuracy. For a more detailed account of your configurations’ strengths and weaknesses, you will soon be able to compare your systems’ answers to the evaluation set on a case-by-case basis. This way, you can get a better idea of which setup best fits your use case.

Set up a demo app and collect feedback from your team

When you’re working with machine learning models, it can be difficult at times to present your preliminary results in a way that makes your team members want to engage with what you’ve built.

In deepset Cloud, you can make any pipeline design directly available to your test users. After setting up your configurations, just send them a link to the “Search” interface, where they can use your setup to query the underlying data, and receive answers in an instant.

The interface facilitates instant feedback: the thumbs-up/thumbs-down icons on the lower right allow your users to share their opinions about the quality of a given answer. This way, you can collect fine-grained performance results from real-world questions, which in turn will let you tune your system even further.

Embed your pipeline in your application

Once you and your team have found the best setup for your search and question answering pipelines, it’s time to move the system to production — that is, embed it in a frontend where your clients can benefit from it. Whether as a search bar on a website, within a mobile app or as an additional functionality for your chatbot — with deepset Cloud, it’s easier than ever to deploy your NLP-powered service.

deepset Cloud exposes a powerful REST API that allows you to easily query and control your pipelines from any other application. In fact, anything that you can do in the UI, you can also do programmatically via the REST API. Just generate a new API key in deepset Cloud under “Connections” and start calling the API from your target application.

It’s common for an NLP application to experience different amounts of traffic. Whether you’re receiving hundreds or millions of requests — deepset Cloud’s big GPU cluster automatically scales to the amount of traffic that you receive.

Can I use Haystack and deepset Cloud in combination?

Yes! If you’re already a user of Haystack, we offer an SDK that lets you switch between the library and the browser interface seamlessly. With the SDK, you can design your model locally and export it to the cloud, or vice versa.

Backed by Top VC Firms

With the launch of deepset Cloud, we also announce our Series A round of $14 Mn led by GV, with the participation of Harpoon Ventures and a team of experienced COSS and ML founders, such as Alex Ratner (Snorkel AI), Mustafa Suleyman (Deepmind), Spencer Kimball (Cockroach Labs), Jeff Hammerbacher (Cloudera) and Emil Eifrem (Neo4j). The funding will be used to continue building our open source Haystack NLP framework, and to make deepset Cloud a leading enterprise platform for NLP application development. We are very excited to have such great and mission-driven partners on board and are always looking for people who want to join us in our journey!

Ready to Build an NLP product?

Why not trying deepset Cloud? Sign up for an early access account today! If you’re curious and want to learn more, check out the deepset Cloud documentation.