7 ms·

Honestly that's a poor blog post. Randomly concludes "the best time-based ID seems to be xid" without saying why or comparing to others e.g. ksuid, UUIDv7 etc

Honestly that's a poor blog post. Randomly concludes "the best time-based ID seems to be xid" without saying why or comparing to others e.g. ksuid, UUIDv7 etc ("xid" is only mentioned twice in the entire blog, first in the above statement and second a link to the reference implementation). Equally unfortunate that they picked "xid" as their supposed "best" because Postgres has an internal identifier that is also called "xid" and is very much NOT to be used as a primary key ! Downplays the many issues with "serial", including somehow thinking the word might "might" has a place next to the words "not want to expose them to the world though" .... you DON'T, full stop. Exposing predictable identifiers to the world is never a good thing. I'm not really sure what that blog post is supposed to be achieving really. I didn't learn anything.

ldjb 4y ago
Collaborative databases (Wikidata, TheMovieDB, VNDB, etc.) all use serial identifiers. What is the problem with this? These websites don't want to hide how many entries they have (they tend to promote them), and it doesn't really matter if you iterate through all the numbers – the data is available through open licences anyway. I think there are many situations where you don't want to expose predictable identifiers, but there are also examples where predictable identifiers may actually be beneficial.
- indymike 4y ago
  > What is the problem with this? It causes meetings with people who think all predictable identifiers are a problem.
- aniforprez 4y ago
  If your data is public and can be scraped anyway, obviously a serial identifier doesn't matter. If your data is sequestered between accounts or has tenants all sharing the same database or API, that's just one accidental permission error away from being able to scrape every single record. If your customer's data is meant to be private, simply easier to generate unique IDs for each record. Plus it makes it incredibly easy for competitors to see the size of your business simply by signing up for an account and looking at the IDs.
vlovich123 4y ago
But if you’re handing out a random ID, you still need to have an index over that ID to have efficient lookups of random ID -> internal ID right? One advantage is that you only need to resolve it once at the edge and then internally you use the external facing value. Are there any others I’m missing? I agree, I would have loved a deeper dive in xid since it seemed to clearly outperform everyone else.
- lbriner 4y ago
  Not necessarily. The idea here is that the id can be exposed publically because it is random. The problem it solves is someone sees a page /accountdetails?id=123 and can easily look for /accountdetails?id=124 and assume it is likely to be valid. If you use a random id, you cannot quickly know what other ids exist which makes looking for unauthorized access to objectids much harder.
  - michaelmior 4y ago
    This of course assumes that all the records backed by these IDs are not public information. In a lot of cases, this may not be true.
  - barrkel 4y ago
    If there needs to be an auth check for other IDs, then it shouldn't matter whether the IDs are random or not. Depending on how highly aesthetic URLs are valued, it's not unlikely that after being in business for a while, the density of your keyspace will mean that even random IDs are found. A better rationale for disconnecting public and private IDs is to make certain types of database migration a little bit easier. I don't think it's a huge win though, it's a chunk of work to maintain multiple IDs, do the required inderictions all the time, ensure public IDs are always sent to front end, and so on, while adding a translation layer as part of a database migration delays this kind of work until it's actually needed, if ever.
    - nevir 4y ago
      https://en.m.wikipedia.org/wiki/German_tank_problem https://en.m.wikipedia.org/wiki/German_tank_problem Preventing competitors from estimating the size of your business (or of your customer's businesses, if generating sequential IDs on their behalf) is one big reason for having unguessable public IDs.
hovering_nox 4y ago
[flagged]
CPLX 4y ago
You really care about exposing a serial number scheme for a list of books your company publishes, or the identifier for each of the various hotels you own, or the cities you have an office in or something?
- michaelmior 4y ago
  Probably not, but you definitely should care about not issuing sequential credit card numbers. Probably not the best example, but I don't think it's hard to imagine some scenario in between the two that still presents a concern.
  - seanhunter 4y ago
    You can't just use sequential numbers for credit cards because they have a builtin checksum validation called "Luhn's algorithm" https://www.creditcardvalidator.org/articles/luhn-algorithm https://www.creditcardvalidator.org/articles/luhn-algorithm
    - michaelmior 4y ago
      Sure, that wasn't really my point though. My point was that there are some cases where sequential identifiers would cause security concerns and some cases where they would not.
- quesera 4y ago
  It's not only about predictability. Exposing a serial number can also be competitive intelligence. Number of users, transactions, etc. Sometimes this matters.
  - CPLX 4y ago
    Yes of course. Which is why I picked a few examples where it doesn’t matter. Stuff you’d just always have a public catalog of.
- nevir 4y ago
osigurdson 4y ago
>> Exposing predictable identifiers to the world is never a good thing. Sometimes it doesn't matter. Example below: https://news.ycombinator.com/item?id=34451344 https://news.ycombinator.com/item?id=34451344
- traceroute66 4y ago
  > Sometimes it doesn't matter. Example below: There is a saying for the examples you and others are posting ...."The exception rather than the rule" Posting contrived examples in order to attempt to prove a point. For the majority of cases, a random ID remains the better option. But unfortunately developers still treat security as an afterthought. They continue to use "serial" because of what can only be described as sheer ignorance, i.e. demonstrably false misunderstandings about database technology. Case in point, I placed an order on an e-commerce site a couple of weeks ago. I was not best pleased to be given a tracking URL that read: "https://example.com/order/WEB-nnnnn https://example.com/order/WEB-nnnnn", where nnnn was clearly an incrementing number. Such a thing is inexcusable in 2023 ! It took a lot of strength to resist the temptation !
  - philliphaydon 4y ago
    Changing the nnnnn should not magically give you an invoice. If your argument That a guid is better used here then you’re wrong. If the endpoint is not secure it doesn’t matter if your used a sequential id or a random id/guid. You could brute force a discovery. You should always validate the input and verify the accessed invoice belongs to the person requesting it.
    - osigurdson 4y ago
      For sensitive information it should not be trivially guessable. Naturally, we need encryption and auth, but using not guessable identifiers multiplies the (already small) probability of successful attack by 1/2^128 so it is a good idea.
      - philliphaydon 4y ago
jeremyjh 4y ago
Yes to me xid has some pretty significant downsides. It is composed of a timestamp, a machine identifier, a process id, and then a sequence that starts from a random value. It can be a bit faster because it doesn't have to generate any random values to make a new ID, but I don't see how it would ever work client-side. I much prefer the simplicity of UUIDv6 and ULID, and they both perform nearly as well. UUIDv6 basically is ULID but without the standard base32 representation, which I think is unfortunate.
onphonenow 4y ago
I really hate this trend away from basic IDs. I feel like it's driven by folks who've never actually worked in the real world. I got account paperwork recently where the company ID account ID and invoice ID were all uuids. 100% this company if I call them will not use this BS to lookup my account and will instead use something easier try to guess like a company phone number. I also had to do some support tickets recently and because of the issues copying uuids from the screen they insisted on screenshots that included the url bar showing uuid. 100% trying to tell them the ID over try the phone or even rekey would run into issues. Give me back my sequential invoice ids!
- brightball 4y ago
  This is a very worthwhile point. You definitely need to prepare for readability if these ids will be customer facing.
  - craigkerstiens 4y ago
    While not a perfect solution to it, we use UUIDs throughout our application as IDs. But when we expose them in URLs or other places provide them as an encoded id (https://docs.crunchybridge.com/api-concepts/eid/ https://docs.crunchybridge.com/api-concepts/eid/). It at least makes them a little more compact, easier to copy/paste than UUIDs, and generally a cleaner look.
- mytailorisrich 4y ago
  As it happens in at least some jurisdictions invoice numbers have to be sequential by law.
  - Scarbutt 4y ago
    That's orthogonal to having a programatic ID for them
- victor106 4y ago
  Are sequential id’s a security risk? In one of our systems we’ve seen customer guess at other accounts by just incrementing the sequence. The rule of thumb I used to use is if an Id is going to be used for lookups or being exposed externally use uuid otherwise us sequential. The hard thing about the above rule is that it’s hard to tell when you are designing the db if the id will be used externally/for lookups or not. Requirements change later.

ldjb 4y ago

Collaborative databases (Wikidata, TheMovieDB, VNDB, etc.) all use serial identifiers. What is the problem with this? These websites don't want to hide how many entries they have (they tend to promote them), and it doesn't really matter if you iterate through all the numbers – the data is available through open licences anyway. I think there are many situations where you don't want to expose predictable identifiers, but there are also examples where predictable identifiers may actually be beneficial.

indymike 4y ago
> What is the problem with this? It causes meetings with people who think all predictable identifiers are a problem.
aniforprez 4y ago
If your data is public and can be scraped anyway, obviously a serial identifier doesn't matter. If your data is sequestered between accounts or has tenants all sharing the same database or API, that's just one accidental permission error away from being able to scrape every single record. If your customer's data is meant to be private, simply easier to generate unique IDs for each record. Plus it makes it incredibly easy for competitors to see the size of your business simply by signing up for an account and looking at the IDs.

vlovich123 4y ago

But if you’re handing out a random ID, you still need to have an index over that ID to have efficient lookups of random ID -> internal ID right? One advantage is that you only need to resolve it once at the edge and then internally you use the external facing value. Are there any others I’m missing? I agree, I would have loved a deeper dive in xid since it seemed to clearly outperform everyone else.

lbriner 4y ago
Not necessarily. The idea here is that the id can be exposed publically because it is random. The problem it solves is someone sees a page /accountdetails?id=123 and can easily look for /accountdetails?id=124 and assume it is likely to be valid. If you use a random id, you cannot quickly know what other ids exist which makes looking for unauthorized access to objectids much harder.
- michaelmior 4y ago
  This of course assumes that all the records backed by these IDs are not public information. In a lot of cases, this may not be true.
- barrkel 4y ago
  If there needs to be an auth check for other IDs, then it shouldn't matter whether the IDs are random or not. Depending on how highly aesthetic URLs are valued, it's not unlikely that after being in business for a while, the density of your keyspace will mean that even random IDs are found. A better rationale for disconnecting public and private IDs is to make certain types of database migration a little bit easier. I don't think it's a huge win though, it's a chunk of work to maintain multiple IDs, do the required inderictions all the time, ensure public IDs are always sent to front end, and so on, while adding a translation layer as part of a database migration delays this kind of work until it's actually needed, if ever.
  - nevir 4y ago
    https://en.m.wikipedia.org/wiki/German_tank_problem https://en.m.wikipedia.org/wiki/German_tank_problem Preventing competitors from estimating the size of your business (or of your customer's businesses, if generating sequential IDs on their behalf) is one big reason for having unguessable public IDs.

hovering_nox 4y ago

[flagged]

CPLX 4y ago

You really care about exposing a serial number scheme for a list of books your company publishes, or the identifier for each of the various hotels you own, or the cities you have an office in or something?

michaelmior 4y ago
Probably not, but you definitely should care about not issuing sequential credit card numbers. Probably not the best example, but I don't think it's hard to imagine some scenario in between the two that still presents a concern.
- seanhunter 4y ago
  You can't just use sequential numbers for credit cards because they have a builtin checksum validation called "Luhn's algorithm" https://www.creditcardvalidator.org/articles/luhn-algorithm https://www.creditcardvalidator.org/articles/luhn-algorithm
  - michaelmior 4y ago
    Sure, that wasn't really my point though. My point was that there are some cases where sequential identifiers would cause security concerns and some cases where they would not.
quesera 4y ago
It's not only about predictability. Exposing a serial number can also be competitive intelligence. Number of users, transactions, etc. Sometimes this matters.
- CPLX 4y ago
  Yes of course. Which is why I picked a few examples where it doesn’t matter. Stuff you’d just always have a public catalog of.
nevir 4y ago

osigurdson 4y ago

>> Exposing predictable identifiers to the world is never a good thing. Sometimes it doesn't matter. Example below: https://news.ycombinator.com/item?id=34451344 https://news.ycombinator.com/item?id=34451344

traceroute66 4y ago
> Sometimes it doesn't matter. Example below: There is a saying for the examples you and others are posting ...."The exception rather than the rule" Posting contrived examples in order to attempt to prove a point. For the majority of cases, a random ID remains the better option. But unfortunately developers still treat security as an afterthought. They continue to use "serial" because of what can only be described as sheer ignorance, i.e. demonstrably false misunderstandings about database technology. Case in point, I placed an order on an e-commerce site a couple of weeks ago. I was not best pleased to be given a tracking URL that read: "https://example.com/order/WEB-nnnnn https://example.com/order/WEB-nnnnn", where nnnn was clearly an incrementing number. Such a thing is inexcusable in 2023 ! It took a lot of strength to resist the temptation !
- philliphaydon 4y ago
  Changing the nnnnn should not magically give you an invoice. If your argument That a guid is better used here then you’re wrong. If the endpoint is not secure it doesn’t matter if your used a sequential id or a random id/guid. You could brute force a discovery. You should always validate the input and verify the accessed invoice belongs to the person requesting it.
  - osigurdson 4y ago
    For sensitive information it should not be trivially guessable. Naturally, we need encryption and auth, but using not guessable identifiers multiplies the (already small) probability of successful attack by 1/2^128 so it is a good idea.
    - philliphaydon 4y ago

jeremyjh 4y ago

Yes to me xid has some pretty significant downsides. It is composed of a timestamp, a machine identifier, a process id, and then a sequence that starts from a random value. It can be a bit faster because it doesn't have to generate any random values to make a new ID, but I don't see how it would ever work client-side. I much prefer the simplicity of UUIDv6 and ULID, and they both perform nearly as well. UUIDv6 basically is ULID but without the standard base32 representation, which I think is unfortunate.

onphonenow 4y ago

I really hate this trend away from basic IDs. I feel like it's driven by folks who've never actually worked in the real world. I got account paperwork recently where the company ID account ID and invoice ID were all uuids. 100% this company if I call them will not use this BS to lookup my account and will instead use something easier try to guess like a company phone number. I also had to do some support tickets recently and because of the issues copying uuids from the screen they insisted on screenshots that included the url bar showing uuid. 100% trying to tell them the ID over try the phone or even rekey would run into issues. Give me back my sequential invoice ids!

brightball 4y ago
This is a very worthwhile point. You definitely need to prepare for readability if these ids will be customer facing.
- craigkerstiens 4y ago
  While not a perfect solution to it, we use UUIDs throughout our application as IDs. But when we expose them in URLs or other places provide them as an encoded id (https://docs.crunchybridge.com/api-concepts/eid/ https://docs.crunchybridge.com/api-concepts/eid/). It at least makes them a little more compact, easier to copy/paste than UUIDs, and generally a cleaner look.
mytailorisrich 4y ago
As it happens in at least some jurisdictions invoice numbers have to be sequential by law.
- Scarbutt 4y ago
  That's orthogonal to having a programatic ID for them
victor106 4y ago
Are sequential id’s a security risk? In one of our systems we’ve seen customer guess at other accounts by just incrementing the sequence. The rule of thumb I used to use is if an Id is going to be used for lookups or being exposed externally use uuid otherwise us sequential. The hard thing about the above rule is that it’s hard to tell when you are designing the db if the id will be used externally/for lookups or not. Requirements change later.