Hybrid search based on OpenSearch and Qdrant / Egor Prokhorenko (Category Architecture)

Interesting. speech My colleague, Yegor Prokhorenko, head of the search technology department, is talking about hybrid search in T. Hybrid because Egor and his guys mixed the standard full-text search OpenSearch and the Qdrant vector base for semantic search by embeddings. And how they did it and what can be learned from the report, which contained approximately the following points:

1. The problem of searching in the bank Search is needed as a customer n (mobile applications, products, articles)and employee n (internal databases, messenger, GitLab). The company has an extensive infrastructure - more 100 Data sources and relatively high load (1 million DAUs, 150 RPS), a lot of indexes. 2. The concept of hybrid search Hybrid search combines classic full-text search and vector search (embedding). This allows you to find relevant documents, even if they do not directly match the keywords. 3. Vector databases and their features Vector OBD (e.g. Qdrant) solve locating ANN (Approximate Nearest Neighbor) They often use a graph algorithm. HNSW (Hierarchical navigable small world). Store vectors and metadata, support filtering and scaling. 4. Hybrid Search Implementation Architecture Architecture is divided into layers L1 is the base index of OpenSearch

• L2 - ML ranking (Catboost)
• L3 - business logic Vector search is implemented as a separate service, which allows you to quickly experiment and not interfere with the main search. 5. Evolution of implementation: from monolith to microservice n The first attempt is to integrate the ANN module into a monolith on C. + + (Elastic)There was no increase in metrics, there were problems with productivity. The second attempt is to transfer vector search to a separate Python backend. (FastAPI, Qdrant), then rewritten to Rust for acceleration. 6. Facet Filtering and Vector Search Limitations In full-text search, filtering is easy, in vector search it is difficult. Post-filtering and combined approaches to select relevant documents had to be implemented. 7. Caching embeddings To accelerate the search and reduce the load on the embedder implemented aggressive caching (before 98% cache-hit, 20 million vectors in the cache). 8. Advanced modeling and use of user data To increase relevance, embedders are trained on click data and relevance from search logs, using ContrastiveLoss. 9. Combining full-text and vector search results The problem was different scales: BM25 for full text and cosine proximity for embeddings. As a solution, different schemes were tried: BM25 normalization, stencil schemes, Catboost-ranger training on both factors. (It showed better results.). 10. Scaling and constraints Hybrid search works well for short documents and long queries, reducing the number of empty results. For large documents, division into chunks is required, the search for the perfect solution is still in the process. The hybrid approach has improved metrics, but requires sophisticated architecture and constant experimentation.

As a conclusion, it can be noted that hybrid search based on OpenSearch and Qdrant allows you to improve the completeness and quality of search, but requires a deep understanding of the architecture, constant optimization and refinement of ranking models.

P.S. I already am. repent about an interesting overview of the capabilities of vector databases, which was in the report at the goto conference 2024 yearA Fun & Absurd Introduction to Vector Databases - Alexander Chatzizacharias" This story complements Egor’s story well.

#Architecture #Software #Engineering #ML #AI #DistributedSystems #SystemDesign