4 ms·

Last time this discussion came up, someone shared this nice article[0] about all the different ID options. It mentions Stripe IDs, which have pretty prefixes li

Last time this discussion came up, someone shared this nice article[0] about all the different ID options. It mentions Stripe IDs, which have pretty prefixes like "prod_foo" and "user_bar". These are great for users of their API (no wondering what an ID refers to) and for internal tooling and debugging, but have to be stored as inefficient varchars, which is not great for indexing and whatnot. So I spent the weekend creating "UPID"[1], which is like ULID but prefixable! Up to four characters for the prefix, 40 bits of time (256ms precision, but could change in a revision) and still 64 bits of randomness. So you can do pretty Stripe-style IDs without losing the benefits of a neat 128-bit UUID column. Have implemented it for Python, Rust and Postgres, maybe worth a kick around for someone! [0] https://brandur.org/nanoglyphs/026-ids https://brandur.org/nanoglyphs/026-ids [1] https://github.com/carderne/upid https://github.com/carderne/upid

shockleyje 2y ago
You are awesome! I work with a former engineering manager of Stripe and one initiative they had me on was migrating our UUID's to the prefixed strategy you mentioned in Stripe. Most of the effort, honestly, went into the compatibility layer when bridging which version of the ID was accepted on a given API, and ensuring that this major change was implemented between services universally (i.e., is service B going to fail if I send a prefixed ID?). All of this to say, I am hesitant to generally recommend migrating from UUID -> prefixed UUID, but I will be considering starting future projects of my own with UPID
- carderne 2y ago
  Thanks for the comment! Yeah I really love Stripe IDs and if I was building an API-first product would definitely make that a priority, but can imagine a bit of headache if there isn't global buy-in from day one... Would be great to hear about it if you do give it a spin. On projects with simpler domain models I can't imagine a huge benefit, but as soon as you have piles of tables it's quite nice that every row tells you up-front exactly what it is.
amelius 2y ago
But why would you need to let the library prefix the ID if you can just prefix it yourself right after the UUID call?
- carderne 2y ago
  Main reason would be that every dev must then remember to add/strip a prefix at some boundary around the DB, which I think would be pretty error prone. A more minor benefit maybe is that they're natively presented with the prefix in the DB, so eg communicating with non-technical team while running SQL queries doesn't involve constantly adding/removing prefixes...
turtles3 2y ago
My understanding of these prefixed IDs is that translation happens at the API boundary - what you store in the database is the unprefixed ULID/UUID in an efficient binary column. Then, whenever the API includes an id in a response, it adds the appropriate prefix. When receiving an id in a request, validate that the prefix is correct and then strip the prefix. That gets you all the advantages of prefixed IDs but still keep all 128bits (or however many bits, you don't have to stick to UUIDs) for the actual id. Or, to put it another way, there's no need to store the prefix in the column because it will be identical for all rows. EDIT: this is not to knock your work - quite the opposite. If you do have a use case where you need rows in the same table to have a dynamic prefix, or the client takes the IDs and needs to put them in their own database, then your solution has a lot of advantages. I think what I'm getting at is that if you're using prefixes then it's a worthwhile discussion to be had about where you apply the prefix.
- carderne 2y ago
  My understanding from Stripe was that they're stored with the prefix as text in the DB, but I might be wrong. As for doing the translation at the API boundary, my only gripe is that it's likely to be error-prone: every dev needs to remember to add/strip the correct prefix in every route. Of course you can add some middleware that is context-aware, but still there will be cases (eg live querying while talking to non-tech team) where not having to translate back-and-forth would be great! Anyway, appreciate the comment and definitely agree that for most teams just using a UUID and adding a bit of code is a more obvious route than using a new ID format someone just made up!
  - tdb7893 2y ago
    As someone who has done that exact sort of stripping at a previous job it's not that bad. If you mess it up your tests immediately fail but also it's just so easy to have a library that does it for you
cryptos 2y ago
I don't think that prefixing IDs with some semantic "type" is not such a great idea, because it makes IDs unnecessary long (and inefficient) and because it is basically a mixture of the actual data and a type, but we have typing systems for that. In general it is also not really an advantage. In case of URLs like http://example.com/items/0190b12d-5ed1-72f5-964c-2252afd2da7d http://example.com/items/0190b12d-5ed1-72f5-964c-2252afd2da7... "items" is already the "type". To avoid confusion on the server side the value extracted from the path could be put into some object like an ItemId (basically a typed wrapper around a UUID).
llimllib 2y ago
I went to add your UUID variant to my notes and found https://github.com/jetify-com/typeid https://github.com/jetify-com/typeid The main differences: - typeid allows up to a 63 byte prefix - typeid follows it with a full 128-bit UUID I prefer your implementation generally I think - I like the fixed size - just wanted to link something similar.
- carderne 2y ago
  Oh thanks for sharing, never found that somehow. It seems that's mostly formalising the general idea of adding a prefix to a UUID, but you still end up storing them as variable length strings. I left 4 bits out in UPID as a version specifier, so could theoretically add a version with longer prefixes or more/less timestamp precision if that seemed useful to anyone.
  - dnsbty 2y ago
    I’m a little late to the conversation, but I’m using the Elixir version of TypeID[0] in my project. While the repo linked to above doesn’t provide any guidance on storing the TypeIDs, the Elixir version does this really well. You store the prefix in the your schema (model) code, but the underlying database column is simply a UUIDv7. So when you call the new function, it will generate a UUIDv7, base32 encode it, and then prepend the prefix. Then at the database layer it will translate that to a UUIDv7 for storing and translate back to the string version when loading. [0] https://github.com/sloanelybutsurely/typeid-elixir https://github.com/sloanelybutsurely/typeid-elixir
    - carderne 2y ago
      Oh yeah that looks ideal. My initial idea was down those lines, but realised this wouldn't be possible with many (most?) languages/ORMs/adapters. Eg for Prisma (TypeScript ORM) it would need to be built in to Prisma itself. Another small benefit of UPID is that it works even for raw SQL sessions, but this obviously requires that it's installed into Postgres, which unfortunately for most is much harder than installing an Elixir/whatever lib. But even if you just store them as u128/UUID, a nice thing is the IDs always know what their prefix is, so eg if you dump data into a warehouse, the IDs don't lose their context and an analytics person can still find what they're looking for.

shockleyje 2y ago

You are awesome! I work with a former engineering manager of Stripe and one initiative they had me on was migrating our UUID's to the prefixed strategy you mentioned in Stripe. Most of the effort, honestly, went into the compatibility layer when bridging which version of the ID was accepted on a given API, and ensuring that this major change was implemented between services universally (i.e., is service B going to fail if I send a prefixed ID?). All of this to say, I am hesitant to generally recommend migrating from UUID -> prefixed UUID, but I will be considering starting future projects of my own with UPID

carderne 2y ago
Thanks for the comment! Yeah I really love Stripe IDs and if I was building an API-first product would definitely make that a priority, but can imagine a bit of headache if there isn't global buy-in from day one... Would be great to hear about it if you do give it a spin. On projects with simpler domain models I can't imagine a huge benefit, but as soon as you have piles of tables it's quite nice that every row tells you up-front exactly what it is.

amelius 2y ago

But why would you need to let the library prefix the ID if you can just prefix it yourself right after the UUID call?

carderne 2y ago
Main reason would be that every dev must then remember to add/strip a prefix at some boundary around the DB, which I think would be pretty error prone. A more minor benefit maybe is that they're natively presented with the prefix in the DB, so eg communicating with non-technical team while running SQL queries doesn't involve constantly adding/removing prefixes...

turtles3 2y ago

My understanding of these prefixed IDs is that translation happens at the API boundary - what you store in the database is the unprefixed ULID/UUID in an efficient binary column. Then, whenever the API includes an id in a response, it adds the appropriate prefix. When receiving an id in a request, validate that the prefix is correct and then strip the prefix. That gets you all the advantages of prefixed IDs but still keep all 128bits (or however many bits, you don't have to stick to UUIDs) for the actual id. Or, to put it another way, there's no need to store the prefix in the column because it will be identical for all rows. EDIT: this is not to knock your work - quite the opposite. If you do have a use case where you need rows in the same table to have a dynamic prefix, or the client takes the IDs and needs to put them in their own database, then your solution has a lot of advantages. I think what I'm getting at is that if you're using prefixes then it's a worthwhile discussion to be had about where you apply the prefix.

carderne 2y ago
My understanding from Stripe was that they're stored with the prefix as text in the DB, but I might be wrong. As for doing the translation at the API boundary, my only gripe is that it's likely to be error-prone: every dev needs to remember to add/strip the correct prefix in every route. Of course you can add some middleware that is context-aware, but still there will be cases (eg live querying while talking to non-tech team) where not having to translate back-and-forth would be great! Anyway, appreciate the comment and definitely agree that for most teams just using a UUID and adding a bit of code is a more obvious route than using a new ID format someone just made up!
- tdb7893 2y ago
  As someone who has done that exact sort of stripping at a previous job it's not that bad. If you mess it up your tests immediately fail but also it's just so easy to have a library that does it for you

cryptos 2y ago

I don't think that prefixing IDs with some semantic "type" is not such a great idea, because it makes IDs unnecessary long (and inefficient) and because it is basically a mixture of the actual data and a type, but we have typing systems for that. In general it is also not really an advantage. In case of URLs like http://example.com/items/0190b12d-5ed1-72f5-964c-2252afd2da7d http://example.com/items/0190b12d-5ed1-72f5-964c-2252afd2da7... "items" is already the "type". To avoid confusion on the server side the value extracted from the path could be put into some object like an ItemId (basically a typed wrapper around a UUID).

llimllib 2y ago

I went to add your UUID variant to my notes and found https://github.com/jetify-com/typeid https://github.com/jetify-com/typeid The main differences: - typeid allows up to a 63 byte prefix - typeid follows it with a full 128-bit UUID I prefer your implementation generally I think - I like the fixed size - just wanted to link something similar.

carderne 2y ago
Oh thanks for sharing, never found that somehow. It seems that's mostly formalising the general idea of adding a prefix to a UUID, but you still end up storing them as variable length strings. I left 4 bits out in UPID as a version specifier, so could theoretically add a version with longer prefixes or more/less timestamp precision if that seemed useful to anyone.
- dnsbty 2y ago
  I’m a little late to the conversation, but I’m using the Elixir version of TypeID[0] in my project. While the repo linked to above doesn’t provide any guidance on storing the TypeIDs, the Elixir version does this really well. You store the prefix in the your schema (model) code, but the underlying database column is simply a UUIDv7. So when you call the new function, it will generate a UUIDv7, base32 encode it, and then prepend the prefix. Then at the database layer it will translate that to a UUIDv7 for storing and translate back to the string version when loading. [0] https://github.com/sloanelybutsurely/typeid-elixir https://github.com/sloanelybutsurely/typeid-elixir
  - carderne 2y ago
    Oh yeah that looks ideal. My initial idea was down those lines, but realised this wouldn't be possible with many (most?) languages/ORMs/adapters. Eg for Prisma (TypeScript ORM) it would need to be built in to Prisma itself. Another small benefit of UPID is that it works even for raw SQL sessions, but this obviously requires that it's installed into Postgres, which unfortunately for most is much harder than installing an Elixir/whatever lib. But even if you just store them as u128/UUID, a nice thing is the IDs always know what their prefix is, so eg if you dump data into a warehouse, the IDs don't lose their context and an analytics person can still find what they're looking for.