7 ms·
UUID Benchmark War
- ttfkam 3y agoWish they had included UUIDv1. Likely better than UUIDv4, but I'm curious how close it gets to UUIDv7 since it's already available.
- dtech 3y agoUUIDv1 is generally not recommended since it leaks MAC addresses
- nerdponx 3y agoAnd on cloud hosts, you might end up with the same MAC address on what you thought should be 4 separate "nodes".
- Vecr 3y agoDo be clear, people have been pwn'd due to this. It's not a theoretical problem.
- GrayShade 3y agoThat sounds interesting, do you have a reference on hand?
- stephenr 3y ago... ok, I can see how the host MAC segment could be an issue if the UUID is generated "client" side and distributed.. What's the risk (in terms of "pwning") when the UUID is generated on a server?
- michaelt 3y agoStep 1: Generate UUIDs using a highly predictable pattern Step 2: Use the UUID as a security key - like saving a private file at files.example.com/12345678-1234-5678-1234-123456781234/private-file.pdf and assuming nobody will be able to download it without knowing the UUID Step 3: Attacker predicts the UUID and downloads the private file. Obviously the real solution here is to not use UUIDs as security keys - but dumbasses shoot towards their feet all the time, and UUIDv4 makes them hit less often.
- stephenr 3y agoBy that logic serial-type ids are unsafe for use too, and much more vulnerable. I was kind of expecting some claim that actually exploits the inclusion of a hardware identifier, rather than "don't use predictable numbers as secrets".
- bhaak 3y ago> By that logic serial-type ids are unsafe for use too, and much more vulnerable. Yes, they are. Still, it's not good security advice to point towards an other thing and say "this is even worse".
- stephenr 3y agoThe point I was making is that the claimed "exploit" is completely outside the bounds of what a reasonable person would do. It's like complaining that a garden hose can be used to fill a car with carbon dioxide so we should recommend against using hoses in gardens.
- bhaak 3y agoThat's often an issue that people think attackers wouldn't got all the way because it's too cumbersome. "Reasonable" has a very imprecise definition. > On a recent Code Assisted Penetration Test (CAPT) our team identified a vulnerability in a client's web application which allowed the consultant to bypass authentication and take over legitimate user accounts. The issue stemmed from the use of UUID version 1 in password reset tokens instead of the secure version 4 counterpart. https://realizesec.com/blog/sandwich-attacks-exploiting-uuid-v1 https://realizesec.com/blog/sandwich-attacks-exploiting-uuid...
- stephenr 3y agoOk, well for this case specifically let's just assume that "reasonable" means "knows a UUIDv1 isn't random, and shouldn't be used as if it were". You can probably sub in "suitably experienced" for "reasonable" if it reads better.
- kozak 3y agoThere are probably (some other) risks in exposing a precise timestamp as well.
- jbosh 3y agoWhat about when those 6 bytes are randomly generated? RFC 4122 does allow the MAC address in a version-1 (or 2) UUID to be replaced by a random 48-bit node ID, either because the node does not have a MAC address, or because it is not desirable to expose it.
- lazide 3y agoThen it wouldn’t have that issue - as long as it isn’t stable/reused for too long. The big problem - how can you tell for sure that is what is happening, and how would you catch a reversion if the underlying library changes behavior? Assuming you’re making a call into someone else’s stuff anyway. A big challenge I’ve seen with UUID implementations is dependence on some kind of hidden system state that causes issues. like a MAC or nodeid file somewhere in a VM that gets cloned, resulting in duplicates where ‘duplicates should be impossible’.
- ttfkam 3y agouuid_generate_v1mc() from the uuid-ossp extension uses a randomly generated mac, so no leakage. https://www.postgresql.org/docs/current/uuid-ossp.html#UUID-OSSP-FUNCTIONS-SECT https://www.postgresql.org/docs/current/uuid-ossp.html#UUID-...
- preya2k 3y agoWish they included alternatives like nanoid.
- sgarland 3y agoNanoid’s performance is likely to be on-par with UUIDv4, if not worse. While it’s smaller in ASCII, Postgres and MySQL (when coerced) both store UUIDs as 16 binary characters, well below the 21 of Nanoid. And in any case, the primary driver of the poor performance is from k-sortability due to page spread. The only alternative to UUIDv? that would do well is ULID, as it’s lexicographically sortable.
- BSDobelix 3y agoWish they used a real server and not a AWS one.
- lionkor 3y agowhats a little bit of cpu stealing between friends! (yes please benchmark on bare metal)
- ericpauley 3y agoc7a servers have dedicated CPUs. Personally I like seeing benchmarks on AWS instances because anyone can reproduce them without needing to buy the same hardware. The virtualization overhead is basically nil; what’s not to like?
- BSDobelix 3y ago>c7a servers have dedicated CPUs I can't see any proof of that, sources? They talk about vCPU here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/compute-optimized-instances.html#compute-instances-hardware https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/compute-... Do you mean for example c6g.metal? For c7a there are also "metal" instances available, but they don't mention that: https://docs.aws.amazon.com/ec2/latest/instancetypes/co.html#co_summary https://docs.aws.amazon.com/ec2/latest/instancetypes/co.html...
- ndriscoll 3y agoThey're referencing this: > One of the major differences between C7a instances and the previous generations of instances, such as C6a instances, is their vCPU to physical processor core mapping. Every vCPU on a C7a instance is a physical CPU core. This means there is no Simultaneous Multi-Threading (SMT). By contrast, every vCPU on prior generations such as C6a instances is a thread of a CPU core. So your "vCPU" is not an SMT thread sharing a core with some other "vCPU". Of course you're still sharing the rest of the CPU, so things like memory bandwidth and presumably upper level cache, and you might be affected by thermal load from other cores? idk whether that kind of thing applies in server contexts. https://aws.amazon.com/ec2/instance-types/c7a/ https://aws.amazon.com/ec2/instance-types/c7a/
- RedShift1 3y agoIs performance of UUIDs that important? In what practical scenario are you inserting thousands of records continuously all the time?
- lofaszvanitt 3y agoI heard there could be problems generating ample uids, like in banking.
- sumtechguy 3y agoOne project I worked on someone picked UUID for tracking AAA telco records. Then decided to make it the primary key (you can guess how well that worked out). There are lots of uses for thousands per second. Usually in some sort of logging tracking application where you have have lots of processes and users.
- sgarland 3y agoIt’s not just the writes. If you’re using a non-k-sortable item (like UUIDv4) as the PK, you’re throwing away every linear read-ahead that the kernel is doing for you, because the odds of what you need being on the same page is nil. When you’re paying for IOPS, like in a cloud, that’s even worse. You’re also causing huge amounts of WAL bloat (unless you’re running ZFS and can thus safely disable full-page writes) [1]. And on a system with a clustered index (InnoDB, SQL Server), the performance and space bloat is even worse. [0]: https://www.cybertec-postgresql.com/en/unexpected-downsides-of-uuid-keys-in-postgresql/ https://www.cybertec-postgresql.com/en/unexpected-downsides-... [1]: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes/ https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-pa...
- jabart 3y agoI have some vendor webhooks that spike to above this number for 30 minutes at a time, multiple times a day. They use UUIDs for the event id you are supposed to use for event deduplication.
- hobs 3y agoThousands? In many scenarios I have worked with the number is millions or billions, but I do "data engineering". Want to tag all traffic on your website? 1 billion visits = ~375 visits a second. Some of the accounting systems I have worked with were raking in datasets ~300GB at a time that were almost purely compressed transactions. Some of the companies I have worked for in retail have thousands of stores and a pretty constant volume, getting to that rowcount for some of their customer tracking stuff would easily blow that out.
- nickjj 3y agoIt's too bad a number of managed Postgres services don't support UUIDv7 yet through supported extensions. DigitalOcean and AWS RDS both support the https://www.postgresql.org/docs/16/uuid-ossp.html https://www.postgresql.org/docs/16/uuid-ossp.html extension but it mentions v5 is the highest supported UUID version.
- sondr3 3y agoYou can generate the UUID in your application and insert it, UUIDv7 can be used in any UUID column in Postgres. UUIDv7 natively in Postgres are coming in v17: https://commitfest.postgresql.org/43/4388/ https://commitfest.postgresql.org/43/4388/.
- kiwicopple 3y agoThere’s a pure pgsql implementation here which you can use: https://github.com/kiwicopple/pg-extensions/tree/main/pg_idkit https://github.com/kiwicopple/pg-extensions/tree/main/pg_idk... Also since RDS supports trusted language extensions you can install it via dbdev: https://database.dev/kiwicopple/pg_idkit https://database.dev/kiwicopple/pg_idkit
- canadiantim 3y agoThanks for the links
- maxloh 3y agoWhat is the use case for UUIDv4/v7 when bigint is faster and uses less storage space?
- EvertBouw 3y agoyou can generate them anywhere, and not guess others
- sgarland 3y agoPostgres can return the id to you that it generates.
- tremon 3y ago...which means you need a round-trip to the database before doing anything else. That's latency you can avoid with client-side predictable (or determined) UUID's. Whether you should do anything else before the data has been persisted is a totally different discussion.
- sgarland 3y agoUnless you’re doing some wild async YOLO, you’re already waiting for the DB to send back success; adding a max of 8 bytes into that is highly likely to fit into the same TCP packet and thus add little to no overhead. If you are doing async writes, as you alluded to, why bother with RDBMS in the first place? ACID is out the window.
- lazide 3y agoNot so in all cases. - insert/then retrieve ID can easily result in duplicate records in some edge cases, and won’t necessarily be able to be easily fixed either. the inserted record doesn’t have a global ID until it’s inserted. Can this generally be fixed using good transactions semantics? Yes usually. But it’s expensive. And in many cases you’ll have to default to failing writes instead of eventually consistent behavior. - CRDT type behavior works better when things have a known valid unique ID from the get go. insert/update/ignore can happen quickly and easily without two way communication and in bulk, and edge cases have more easily modelable ‘eventual consistency’. - generating unique IDs in the database forces serialization of certain processes in the database, which can cause scaling issues and high latency. For instance, using the DB to create unique request IDs for web or API requests? Asking for problems. Generating UUIDs for them at request time, and then putting those IDs where needed when later correlation/tracking is desirable? Much better. Same can apply for other object ID creation, when there aren’t other natural keys that need to be checked first.
- semiquaver 3y ago> 30% faster and 7% smaller than uuidv4 Why would UUIDv7 be more compact than v4? Doesn’t it take the same number of bits?
- masklinn 3y agoThe random insertion location of UUID4 causes bloat (at least index bloat) as it forces splitting leaves in order to add new entries, so you end up with a lot more partial leaves: with sequential insertion (sequences, uuid7) the leaves get filled and when a leaf is filled you append a new one, you don’t go and revisit leaves.
- sgarland 3y agoI love how the general tech blogosphere / Twitter is slowly coming around to the idea that random UUIDs are a terrible PK. I tried convincing people of this at my last job, to little interest. Maybe now that “thought leaders” are saying it, people will take notice.
- packetlost 3y agoAside from maybe size and generation speed, I don't really see why a UUID is any worse than any other synthetic identifier. I don't really see generating a UUID being a limiting factor in DBs if they're designed for it, so what's your reasoning?
- thomasmg 3y agoThe problem is not "UUID" but "random UUID". The randomness makes it hard to index and query, due to missing locality. UUIDv7 is time-sortable, and so has locality. For details, see for example https://uuid7.com/ https://uuid7.com/ - the main reason (for me) is this part: "Since UUIDv7 is time-sortable, database indexing mechanisms can better optimize the storage and retrieval processes...".
- lucumo 3y agoI don't know. The leaking of information via time-based identifiers always strikes me as an annoying trade-off to work around an implementation detail in certain database engines. Sometimes you have to do it, but it shouldn't be a general rule.
- thomasmg 3y agoUUIDs are globally unique (unique with such a high probability that in practise there are no collisions, if you follow the spec). But they are not cryptographically safe. So the time part is not a "leak" in this sense. If you need something cryptographically safe, then you would need to encrypt the UUID.