Best practises for ID generation

[thomasf1]

Sorry for this noob question, but I couldn´t find anything in the documentation.

Porting some code over from another DB, currently the IDs on an new Item (DB entries / Points) are randomly generated and in the rare case a ID already exists the code inserting it throws an exception and a new ID is assigned.

There seems to not be a insert/create function with qdrant, giving me the feeling that this not encouraged. What´s the best practise on qdrant (while keeping ID assignment in the Code / not shifting it to the DB)?

[generall]

Hey @thomasf1, there are several possible options:

• Use random UUID and keep original ID in payload. UUID is 128 bit and collision probability is zero by all practical measurements.
• Use hash of the original ID

[thomasf1]

So, basically the encouraged strategy is to make the ID as long as possible to then (hopefully) never run into the issues of it existing already as it would overwrite the existing object by default?

[abhiksark]

@generall Do we get a capability of auto-generated id?

[generall]

what do you mean exactly?

[thomasf1]

If I understand the docs correctly, the ID field is either an integer or an UUID. I did not see an option to auto-generate an ID on insert.

It´s an interesting design decision, it seems to put some unnecessary limitations on the product - especially when migrating from another DB and wanting to keep IDs or having an existing strategy for generating IDs. Pretty basic stuff in traditional DB Products...

[timvisee]

You're right, there is no option to automatically generate an ID. This is hard to do in a good way in a distributed environment not routing to a single leader, and we want to be deterministic. It's also very likely that you want to have specific IDs linked to source data anyway.

You can use an auto incrementing number yourself to generate IDs. Using random UUIDs to make sure IDs are unique is also perfectly fine.

[thomasf1]

That makes sense, that can be tricky/costly to implement. The choice of either Integers or UUIDs seems a bit limiting though, but I guess it is the two extremes that are easiest (either fully app/user managed sequential IDs or long ones with a low chance of collision).

Would you guys consider opening up the ID field to Strings (which UUIDs basically are, too), it would be nice for a migration to use existing IDs - also for URLs UUIDs are a bit long.

The problem of inserts would still need some solution though, not sure what happens currently when a numeric or UUID id already exists?

The alternative of managing two sets of IDs (one Record ID and one in the payload) seems like messy though, especially when using features like recommendations which are Record ID based and would need mapping tables or something.

[timvisee]

Would you guys consider opening up the ID field to Strings (which UUIDs basically are, too), it would be nice for a migration to use existing IDs - also for URLs UUIDs are a bit long.

Probably not, unless there's a very good use case for it.

Both integers and UUIDs have some good properties we like, such as being fixed size. Note that UUIDs are not strings, and we don't handle them internally as strings, it's just how they are exposed in our REST API.

not sure what happens currently when a numeric or UUID id already exists?

If it already exists, you update/overwrite. That is desired behavior.

[generall]

Strings (which UUIDs basically are, too)

We do not store UUIDs as Strings

[almostimplemented]

When a collection uses integer IDs, is there an efficient way to query the largest existing ID in the collection?

[timvisee]

No, there is not.

[CharlieGreenman]

UPDATE:
After all is said and done the approach I take is the following. On the creation of an item in the db, I add an accompanying uuid. I am then able to reference that uuid to update/upsert the respective qdrant item.

All things considered, this is much easier than looking up the item first, get the id/uuid, and then upserting.

I wrote an article explaining this approach: https://charliegreenman.medium.com/how-to-update-an-existing-vector-in-qdrant-using-an-external-database-for-uuid-management-ec99cf5a50b1

---

Edit: I tried going the numerical route. It doesn't make sense as 4 billion will be the max anyways. We are working on datasets and might get there anyways. Uuid is the only approach.

Therefore, in order to update, you will have search a vector a specific fields i.e. a string id, then use that to get the id to upsert. Gosh this is really tough, but the best approach at the moment

Can disregard this won'' work. unint32 not large enough

Nnice I just found a cool method for converting strings to numerical ids

orgId abc is 10203
workspaceName xyz is 24025026
itemNumber 123 (no conversion needed so numbers stay as is )

string as id === 10203002402502600123

0 is space between letters
00 is space between distinct words

I'm going to write this article soon, and there should be an open source repo users can use to convert strings to ids for writes and retrievals. Takes away effort from user having to use uuid, in addition can query against id directly if need be.

I am writing an open source library for this soon and report back

Update: Open source repo here and I am working on: https://github.com/razroo/string-to-id