Pre-trained models (transformers) for direct use "in" Qdrant

[do-me]

Idea

Many people probably have the same needs, that is:

1. Qdrant as vector database
2. Some transformer or other pre-trained model for embedding generation
3. Any app connected to Qdrant API

I was wondering about your thoughts considering some kind of "smooth" integration of e.g. sentence-transformers (GitHub here). The same could obviously also be done for image encodings.

It would be charming if Qdrant somehow could integrate these model directly in its API, so internalizing step 2 also in Qdrant.
E.g. when initializing a new collection, one could optionally add any public transformer model and use it as a default in the API.

Advantages

1. Performance
   Instead of having e.g. a Python microservice somewhere (or in whatever language the wrapper is defined) that:

• takes a user input, e.g. converting some text to vectors
• firing a POST request to the Qdrant API
• Qdrant processing and sending the response

this could be done all from the Qdrant API and would very likely be faster than the above "standard" way.

2. Convenience
   It would be so convenient if you think about it!

Writing to DB

When writing to the DB it would be very nice to simply send an array of documents and Qdrant takes care of all the rest.

Reading from DB

Reading is the biggest advantage from my point of view. You could create a fully-fledged app with just Qdrant and a static frontend!
In the frontend, a user would only input a search term and fire a request to the API.

Qdrant would calculate the embedding for the search term, search the DB and respond with the results, all in one go!

Challenges

I am aware that it might be way out of scope for Qdrant but I think it also has the potential to attract a very large user base if it's that beginner-friendly.

A challenge is probably that a model must first be downloaded, initialized and running somewhere which might be tricky to set up in one Docker image.

A more feasible solution would maybe be some kind of link between the Qdrant Docker image and another Docker image serving the model that could be set up in advance if needed.

What do you think about this?

[do-me]

One of my colleagues just told me about Nvidia Triton which is thought for handling different models at the same time.
It also offers a C API - do you think Qdrant could in some way be linked up like described above?

[generall]

Hi @do-me, thanks for such a details proposal!

Your proposal does look very attractive and promising to users. In fact, the topic of embedded encoders is discussed quite often in our internal chats, so I can add a few additional factors to consider.

• First, there is an architectural consideration - models and storage have different pattern of scaling, it is only true for very small installations, that you need one inference service per storage node. Even if you make encoder module optional, the combination of state-full and state-less functions, which requires different scalability patterns under different use-cases is a problem.

• Secondly, there is the problem of generalization - there are numerous ways of how people want to encode embeddings. Even more ways to do pre- and post-processing. Even for the same model, there are use-case for online-vs-offline inference, batching vs minimal latency, different text processing approaches, e.t.c. Of course, you could take just one, the most popular method, but then there would be constant requests to add new encoders, which would shift the focus from search performance to dealing with integrations. There are projects that do this kind of thing, but we don't want to go their way.

• Thirdly, there is the problem of maintenance - models change rapidly and the introduction of even one model will require constant monitoring to ensure that it is not outdated.

• Finally, there is philosophical on - Qdrant tries to stick to the Unix-way in its development. This means that one tool should solve exactly one task, but do it as good as possible.

Having mentioned all this, I must admit that the use-case you described can indeed be very useful and handy.
But as an implementation approach, I would prefer the alternative you describe:

A more feasible solution would maybe be some kind of link between the Qdrant Docker image and another Docker image serving the model that could be set up in advance if needed.

I think that would be a very good satellite project, you can avoid REST and python in communication between those containers, so the data transfer overhead will be negligible compared to model inference time.