6 ms·

UUID is also used to avoid leaking information about the underlying system. This includes temporal information that could be used to infer the size of the datas

UUID is also used to avoid leaking information about the underlying system. This includes temporal information that could be used to infer the size of the dataset for all users. If this isn’t a concern, then using a timestamp based approach as recommended in this article is a good approach. That is the default in MongoDB. If it is a concern, one approach is to use random UUIDs to give to end users but then internally to have something like auto increment ids.

jandrewrogers 3y ago
Organizations that care about leaking information encrypt their identifiers as UUIDs. UUIDs conveniently have the same size as a single AES block, for which there is dedicated silicon on all modern CPUs. There is almost no overhead. Every other issue with random UUIDs etc, which are ignored here, are solved by encrypting your identifier. Random UUIDs (i.e. UUIDv4) are banned in many places for good reasons.
- Fire-Dragon-DoL 3y ago
  Won't encrypting defeat the ordering though, reintroducing the performance penalties?
WeAddValue 3y ago
Or use hashids.org. Use it to obfuscate your integer auto increment ids anywhere a user might see it.
- VWWHFSfQ 3y ago
  It's a good option purely for cosmetics but don't rely on it for any kind of serious obscurity since it's trivially reversed. I've used it to great effect in the past to encode multiple integer values like start/end ID, sort order (0/1), etc. for cursor-based pagination. But that's only because there's nothing secret in those numbers. Just purely for convenience.
  - code_biologist 3y ago
    Salting your hash should work? You could also use a 32 or 64 bit block cipher like skip32 if you want to prevent reversal. Or at least, it makes reversing non-trivial.
    - VWWHFSfQ 3y ago
      This is a common misconception. hashids is _not_ an encryption algorithm. It's just an encoding with a tiny bit of obscurity layered in. hashids.org: > Do not encode sensitive data. This includes sensitive integers, like numeric passwords or PIN numbers. This is not a true encryption algorithm. There are people that dedicate their lives to cryptography and there are plenty of more appropriate algorithms: bcrypt, md5, aes, sha1, blowfish.
  - willvarfar 3y ago
    I once heard of some fancy data science that inferred that the company was the supplier's major customer from the gaps in the sequential ids they saw when using that supplier's API. That was useful in negotiations. And so the company that did that data science realised they too were susceptible to exactly the same 'attack'. So they created a system to obscure the ids they were themselves exposing to their customers, using some cheap cut-down tea64 encryption iirc. My memory is it never went live, though.
    - theluketaylor 3y ago
      Not that fancy (or at least not that novel). In WWII the allies used sequential serial numbers on destroyed and captured tanks to have a frighteningly accurate picture of German industrial output. https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem
CGamesPlay 3y ago
How could temporal information be used to infer the size of the dataset for all users? Specifically, more accurately than “I now know that this database was created in 2022”.
- iwwr 3y ago
  Sequential serial numbers reveal information. See: https://en.m.wikipedia.org/wiki/German_tank_problem https://en.m.wikipedia.org/wiki/German_tank_problem
  - CGamesPlay 3y ago
    Yes. How could temporal information be used to infer the size of the database? To clarify this a bit, UUID v7 (or the existing ULID) are timestamps with random bytes at the end. You can learn when a UUID v7 was created. How do you infer the total size of the database from this information?
- tgv 3y ago
  I use uuids in links. Using sequential ids gives people access to information they shouldn't see. Unfortunately, the link has to be publicly accessible. Then UUIDs become a very practical key for your records. That said, the number of times those records are accessed is low, so there are no performance considerations.
babbledabbler 3y ago
I actually have been using a combination of a numeric integer ID as the PK and a UUID as a lookup field to do routing lookups etc. for this purpose in Postgres backed app I'm working on. I found this approach to be more trouble than it's worth and plan on switching to a UUID PK key and doing away with the integer sequence. Here are the complications I ran into: The libraries I'm using for the ORM and API are designed to work with a primary key for single record get access. For example they want you to do `resource.get(ID)` where ID is the primary key, however, I now have to do `resource.find({ where: { uuid: 'myuuid' }})`. This is for all resources on all pages. In Postgres, the integer PK sequence has a state that keeps track of what number it's at. In certain circumstances it can get out of sequence and this can trip up migrations. We already have created_at and updated_at fields and these are probably better for ordering than the sequencing. Had I to do it again, I would just use UUIDv4 until it runs into issues and either date fields or a sequence where necessary. If anyone has better ideas I would be most grateful as this is something I go back and forth on.
- bavell 3y ago
  As someone who has been on the fence about this for awhile now, thank you for your valuable anecdote!
- damethos 3y ago
  > Had I to do it again, I would just use UUIDv4 until it runs into issues and either date fields or a sequence where necessary How soon is soon? And how would you handle it if you already had all PKs as UUIDs
  - babbledabbler 3y ago
    Probably when there is some downtime or as the opportunity presents. I would use a date field to do ordering or a sequenced field when needed.
- 3pt14159 3y ago
  Well, I'll give another take. If you do not have two separate forms of identifier AND you have a "public" API (including basically any client apps, frontend JS, or anything found in query params) then you are making compliance with European regulators a massive headache when it comes to erasure of PII, since shared identifiers must be destroyed one way or another. Trying to merely delete the records is complicated by the fact that you need legal holds to comply with various federal laws. Between the work involved in using two identifiers, one for joins and one for external lookups, versus the work involved in manually coding up all sorts of erasure work arounds, something I was in charge of in the past, I would strongly consider just using two IDs. If your ORM gets in the way, just modify the ORM. This is easier than you'd think. For example, in Django just make a helpers module with something like this: class OurModel(Model): def get(...): And have some sort of programatic way (lint, etc) of ensuring that your models.py doesn't use the stock class. It's simpler. It ruffles some feathers at first, but if your framework is getting in the way of a real use case just change the framework and don't worry about it.
jmull 3y ago
In my experience internal auto-increment ids and external random UUID works well.
gmhafiz 3y ago
Expanding on temporal information leak 1. Size: If a client receives a record with id=10004578, they can guess that 4578 orders have been made. 2. Rate of growth: Receiving two different orders means they can track the growth rate of record insertion. And also Iteration attack: If your API endpoints do not have authorization, an attacker can try to access with GET /api/users/1, GET /api/users/2, GET /api/users/3, etc. UUID makes this next to impossible.
hbrn 3y ago
Key type is not the right layer for solving information leak (and incorrect joins, mentioned above). Use hashids to avoid the leak, and as a bonus your client-facing keys will be short and easily copy-pastable.

jandrewrogers 3y ago

Organizations that care about leaking information encrypt their identifiers as UUIDs. UUIDs conveniently have the same size as a single AES block, for which there is dedicated silicon on all modern CPUs. There is almost no overhead. Every other issue with random UUIDs etc, which are ignored here, are solved by encrypting your identifier. Random UUIDs (i.e. UUIDv4) are banned in many places for good reasons.

Fire-Dragon-DoL 3y ago
Won't encrypting defeat the ordering though, reintroducing the performance penalties?

WeAddValue 3y ago

Or use hashids.org. Use it to obfuscate your integer auto increment ids anywhere a user might see it.

VWWHFSfQ 3y ago
It's a good option purely for cosmetics but don't rely on it for any kind of serious obscurity since it's trivially reversed. I've used it to great effect in the past to encode multiple integer values like start/end ID, sort order (0/1), etc. for cursor-based pagination. But that's only because there's nothing secret in those numbers. Just purely for convenience.
- code_biologist 3y ago
  Salting your hash should work? You could also use a 32 or 64 bit block cipher like skip32 if you want to prevent reversal. Or at least, it makes reversing non-trivial.
  - VWWHFSfQ 3y ago
    This is a common misconception. hashids is _not_ an encryption algorithm. It's just an encoding with a tiny bit of obscurity layered in. hashids.org: > Do not encode sensitive data. This includes sensitive integers, like numeric passwords or PIN numbers. This is not a true encryption algorithm. There are people that dedicate their lives to cryptography and there are plenty of more appropriate algorithms: bcrypt, md5, aes, sha1, blowfish.
- willvarfar 3y ago
  I once heard of some fancy data science that inferred that the company was the supplier's major customer from the gaps in the sequential ids they saw when using that supplier's API. That was useful in negotiations. And so the company that did that data science realised they too were susceptible to exactly the same 'attack'. So they created a system to obscure the ids they were themselves exposing to their customers, using some cheap cut-down tea64 encryption iirc. My memory is it never went live, though.
  - theluketaylor 3y ago
    Not that fancy (or at least not that novel). In WWII the allies used sequential serial numbers on destroyed and captured tanks to have a frighteningly accurate picture of German industrial output. https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem

CGamesPlay 3y ago

How could temporal information be used to infer the size of the dataset for all users? Specifically, more accurately than “I now know that this database was created in 2022”.

iwwr 3y ago
Sequential serial numbers reveal information. See: https://en.m.wikipedia.org/wiki/German_tank_problem https://en.m.wikipedia.org/wiki/German_tank_problem
- CGamesPlay 3y ago
  Yes. How could temporal information be used to infer the size of the database? To clarify this a bit, UUID v7 (or the existing ULID) are timestamps with random bytes at the end. You can learn when a UUID v7 was created. How do you infer the total size of the database from this information?
tgv 3y ago
I use uuids in links. Using sequential ids gives people access to information they shouldn't see. Unfortunately, the link has to be publicly accessible. Then UUIDs become a very practical key for your records. That said, the number of times those records are accessed is low, so there are no performance considerations.

babbledabbler 3y ago

I actually have been using a combination of a numeric integer ID as the PK and a UUID as a lookup field to do routing lookups etc. for this purpose in Postgres backed app I'm working on. I found this approach to be more trouble than it's worth and plan on switching to a UUID PK key and doing away with the integer sequence. Here are the complications I ran into: The libraries I'm using for the ORM and API are designed to work with a primary key for single record get access. For example they want you to do `resource.get(ID)` where ID is the primary key, however, I now have to do `resource.find({ where: { uuid: 'myuuid' }})`. This is for all resources on all pages. In Postgres, the integer PK sequence has a state that keeps track of what number it's at. In certain circumstances it can get out of sequence and this can trip up migrations. We already have created_at and updated_at fields and these are probably better for ordering than the sequencing. Had I to do it again, I would just use UUIDv4 until it runs into issues and either date fields or a sequence where necessary. If anyone has better ideas I would be most grateful as this is something I go back and forth on.

bavell 3y ago
As someone who has been on the fence about this for awhile now, thank you for your valuable anecdote!
damethos 3y ago
> Had I to do it again, I would just use UUIDv4 until it runs into issues and either date fields or a sequence where necessary How soon is soon? And how would you handle it if you already had all PKs as UUIDs
- babbledabbler 3y ago
  Probably when there is some downtime or as the opportunity presents. I would use a date field to do ordering or a sequenced field when needed.
3pt14159 3y ago
Well, I'll give another take. If you do not have two separate forms of identifier AND you have a "public" API (including basically any client apps, frontend JS, or anything found in query params) then you are making compliance with European regulators a massive headache when it comes to erasure of PII, since shared identifiers must be destroyed one way or another. Trying to merely delete the records is complicated by the fact that you need legal holds to comply with various federal laws. Between the work involved in using two identifiers, one for joins and one for external lookups, versus the work involved in manually coding up all sorts of erasure work arounds, something I was in charge of in the past, I would strongly consider just using two IDs. If your ORM gets in the way, just modify the ORM. This is easier than you'd think. For example, in Django just make a helpers module with something like this: class OurModel(Model): def get(...): And have some sort of programatic way (lint, etc) of ensuring that your models.py doesn't use the stock class. It's simpler. It ruffles some feathers at first, but if your framework is getting in the way of a real use case just change the framework and don't worry about it.

jmull 3y ago

In my experience internal auto-increment ids and external random UUID works well.

gmhafiz 3y ago

Expanding on temporal information leak 1. Size: If a client receives a record with id=10004578, they can guess that 4578 orders have been made. 2. Rate of growth: Receiving two different orders means they can track the growth rate of record insertion. And also Iteration attack: If your API endpoints do not have authorization, an attacker can try to access with GET /api/users/1, GET /api/users/2, GET /api/users/3, etc. UUID makes this next to impossible.

hbrn 3y ago

Key type is not the right layer for solving information leak (and incorrect joins, mentioned above). Use hashids to avoid the leak, and as a bonus your client-facing keys will be short and easily copy-pastable.