Build Semantic Search with Haystack
Haystack NLP framework enables developers to implement production-ready semantic search systems.
For decades, semantic similarity was a concept foreign to computers. If you wanted to teach a computer that two words were synonyms, you had to convey that information via hard-coded rules. Modern search systems, however, don’t need rules. Without being explicitly taught how to do it, they understand the semantic relationships between the following sentences almost as well as humans:
- The cat seems happy.
- This kitten looks entertained.
- Antoine likes ice cream.
Nobody has to tell you that sentence one is similar to sentence two, even though they don’t share any words—and that both, in turn, are very different from the third sentence. That’s because when we evaluate the content of a sentence, we try to grasp its meaning as a whole. Semantic search emulates that capacity by comparing documents on the basis of semantic similarity—and it excels at that thanks to a relatively recent neural network architecture.
Transformer-based language models take a semantic approach to interpreting language. These neural architectures try to mimic the way a human nerve transmits information by wiring together millions of interconnected “neurons.” Neural nets require a lot of computational power and large amounts of data to learn from. Through a training algorithm, the transformer network learns language by being exposed to the data. This is a bit similar to how small children first learn languages, although it happens much faster.
However, building production applications with language models is complex. Deep-learning frameworks like Tensorflow and Pytorch are hard to use for non-specialists as they require writing advanced Python code to get even the simplest examples to work. These major frameworks also aren’t designed to be used with web systems through an HTTP API and therefore don’t fit with the full-stack developer mindset.
Haystack is a framework for building scalable semantic search systems
Haystack is an end-to-end framework for building semantic search solutions.
By combining the power of transformer-based language models, pipeline-oriented structure, and developer-friendly tooling, Haystack is the NLP framework of choice for a range of organizations. Companies like Bank of America, Airbus, BMW, and Sooth.ai all rely on Haystack to set up their semantic search systems.
To implement your own semantic search pipeline on top of your knowledge base, you just need to follow a few simple steps. Index your documents in your database of choice — Elasticsearch, SQL, FAISS, and Milvus are some popular options, and start querying!
With Haystack as your semantic search layer, you may query the database through an HTTP REST API or connect it to web-based systems like technical support forms, search UI elements, and data pipelines.
You can expect your Haystack-powered semantic search systems to return the documents that are most relevant to your query — based on compatibility in meaning rather than lexical overlap. If you choose to implement a question answering pipeline, it will return shorter passages from your documents as answers to your questions. And these are just the basic systems. Do you want to add translation capabilities or a sentiment classifier to your system? No problem, Haystack offers all of that — and much more.
Frequently asked questions
One of the first Transformer language models was BERT (short for Bidirectional Encoder Representations for Transformers) by Google. When a word passes through the trained BERT model, it is transformed into a high-dimensional feature . This high-dimensionality is the key to the model’s deeper understanding of natural language. To determine the similarity between two text passages, we need only compute the distance between their feature vectors in the embedding space. To that end, we can use a metric like cosine distance. The smaller the distance between two vectors, the greater their similarity.
A pre-trained language model can be easily fine-tuned to different NLP tasks through "transfer learning." Fine-tuning, or domain adaptation, takes considerably less time and data than training a model from scratch. For example, we use the annotated SQuAD dataset to finetune a language model like or Facebook’s RoBERTa to the question answering task.