12 ms·
New UUID Formats
- gigatexal 4y agoUuidv7 looks really interesting. I like that the article mentions all the other projects that attempted to fix uuid issues for different use cases.
- stevesimmons 4y agoThis had appeared a few times before at earlier stages of the draft process... https://news.ycombinator.com/item?id=28088213 https://news.ycombinator.com/item?id=28088213 [244 comments] Personally, I really like UUIDv7, except I transform the UUID to a 25-character string which has all the same properties except it doesn't look like a UUID. The last thing I want is my index of time-sortable UUIDs getting contaminated with with some UUIDv4 fully random ones. Since UUIDs may be generated and persisted in a distributed manner, it's a simple way to at least spot this.
- ikornaselur 4y agoThe UUID version is in the actual ID, so you would be able to spot the version as it isn't fully random. Although you light mean just visually you'd spot the difference much easier!
- theptip 4y agoCan’t you just check the “version” bits and reject if it’s not 4/7? Or are you worried about someone generating a completely random (non-compliant) set of bits that happens to parse as a v4/7
- kstenerud 4y agoIt's a shame they didn't finally drop big endian. Little endian won out and our newer protocols should reflect that for efficiency's sake.
- zuzun 4y agoIt's necessary to keep them sortable at byte level.
- finnh 4y agoBut network byte order is big endian, so I can see arguments both ways (as it were)
- chrismorgan 4y agoBig-endianness is mandatory for opaque bytewise sorting (or lexicographic ordering in string form), which is a very desirable property of UUIDv6 and UUIDv7.
- wyager 4y agoI guarantee you that byte order swapping is not the limiting factor on the efficiency of any system involving UUIDs on the wire.
- amluto 4y agoNo, they should have tried to find a way to drop little endian. A cycle or two when generating UUIDs is irrelevant, and almost all UUID implementations, and most of the RFC 4122 text, are big-endian.
- bearjaws 4y agoUUIDv6 coming in just about 10 years late. Guess its something.
- deleted 4y ago[deleted]
- kbumsik 4y agoUUIDv7 looks interesting, but how is it different from ULID [1] in practice? I was considering using ULID for a upcoming new project because it is lexicographically sortable but it looks like UUIDv7 just can replace that. [1]: https://cran.r-project.org/web/packages/ulid/vignettes/intro-to-ulid.html https://cran.r-project.org/web/packages/ulid/vignettes/intro...
- rekwah 4y agoAs the author of a popular ULID implementation in python[1], the spec has no stewardship anymore. The specification repo[2] has plenty of open issues and no real guidance or communication beyond language implementation authors discussing corner cases and the gaps in the spec. The monotonic functionality is ambiguous (at best), doesn't consider distributed id generation, and is implemented differently per-language [3]. Functionally, UUIDv7 might be the _same_ but the hope would be for a more rigid specification for interoperability. [1]: https://github.com/ahawker/ulid https://github.com/ahawker/ulid [2]: https://github.com/ulid/spec https://github.com/ulid/spec [3]: https://github.com/ulid/spec/issues/11 https://github.com/ulid/spec/issues/11
- RobertRoberts 4y agoThank you, I've been using ULID for a while now, and it serves my purposes. But I have long term support concerns. UUIDv7 really seems like the sweet spot between pure INT/BIGINT auto incrementing PKs and universally sortable universal ids.
- kortex 4y agoI've bee using ULIDs in python for about a year now and so far have been super happy with them, so a) thank you for maintaining this! b) I always felt a bit uneasy about the way the spec describes the monotonicity component. Personally I just rely on the random aspect as I am fortunate enough to say that two events in the same millisecond are effectively simultaneous. At that point, it's basically just UUID7 with Crockford base32 encoding, more or less. IMHO the in-process monotonically increasing feature of ULID is misguided. As you mention, distributed ids are a pain. The instant you start talking distributed, monotonic counters or orderable events (two threads count as distributed in this case), you need to talk things like Lamport clocks or other hybrid clock strategies. It's better to reach for the right tools in this case, vs half-baked monotonic-only-in-this-process vague guarantee.
- jwilk 4y agoOld-school formatting of the same document: https://datatracker.ietf.org/doc/html/draft-peabody-dispatch-new-uuid-format-03 https://datatracker.ietf.org/doc/html/draft-peabody-dispatch...
- deleted 4y ago[deleted]
- ripe 4y agoSide note: I love the HTML format of these IETF RFCs, as in TFA. Over the decades, I was used to seeing the old text format (which I like), but this one is particularly easy on the eye, especially on my Android phone.
- deleted 4y ago[deleted]
- sureglymop 4y agoI love the design and would like to copy it for my blog! Especially the code blocks with the nice 'figure' description below them.
- hoten 4y agoOnly mobile issue I see is there is a really long link that overflows the intended document width, which introduces an horizontal scrollbar. That makes scrolling a bit finicky on mobile.
- amluto 4y agoFortunately this is a bit less relevant today as Windows loses market share in database and server applications, but: UUIDs have historically massively screwed up endian handling. While this new draft discusses sorting UUIDs as strings of octets (bytes) and the text of RFC4122 is fairly explicit about most significant bytes coming first, the C UUID structure in RFC 4122 appendix A is entirely misguided: typedef struct { unsigned32 time_low; unsigned16 time_mid; unsigned16 time_hi_and_version; unsigned8 clock_seq_hi_and_reserved; unsigned8 clock_seq_low; byte node[6]; } uuid_t; Those aren’t bytes — they’re integers of various sizes. (Hint: do not use integer types in C code for portable data structures. ntohl, etc are a mess. Just use arrays of bytes.) I don’t know the whole history, but MS somehow took this structure at face value and caused problems like this: https://github.com/uuid-rs/uuid/issues/277 https://github.com/uuid-rs/uuid/issues/277 So, if you want to do anything (e.g. sorting) that depends on the representation of a UUID (or even depends on converting between string and binary representations), be aware that UUIDs coming from Windows may be little-endian. In my book, this is a Windows bug, but opinions may differ here.
- blueflow 4y agoThis mishap lives forth in the UUID stored in a machines DMI data, as well in GPT partition tables, which are required when EFI is used. It would be really cool if we had some replacement for EFI that would not harbour these kind of painful legacies.
- dtech 4y agoOn the other hand, this is an easy to implement conversion, while changing such a fundamental thing from EFI sounds pretty hard, making it not worth it.
- blueflow 4y agoYou cannot fix this with an conversion because you do not know if your UUID is correct or needs conversion. DMI data for example has inconsistent endian-ness depending on the vendor. So if you have a UUID sticker on a new server, you still have two options which UUID the machine will send during PXE, either the printed UUID in big-endian encoding or in the microsoft mixed-endian encoding. Use BIOS boot instead of EFI, it has less legacy to implement: PE executables, FAT file system, Win64 ABI
- duxup 4y agoI’m always amazed how much work goes into these. Meanwhile 99% of the time i just call a function when “I just need something unique here”, -calls function-, and it works! Thanks everyone!
- GiorgioG 4y agoI used to be a big proponent of using UUIDs for database PKs but I've found them inherently difficult to work with. It's much easier to remember/recognize an integer based PK when troubleshooting a data problem. This isn't to say you shouldn't use UUIDs at all, but I much prefer to use an "ExternalId" column of UUID type if you don't want to expose your integer based PKs externally.
- staticassertion 4y agoI feel like `serial` has got to be pretty fast for writes and gives you some nice properties like sorted ordering based on insertion time, a smaller key, and a key that can be compressed far better than a uuid. I get the idea of a ULID/UUID7 encoding a timestamp so you get sorted order, but I wonder at what scale that beats just using serial.
- stormbrew 4y agoAt any scale where you have to have multiple writers generating IDs or need to merge results from multiple sources, basically. Synchronizing a serial increment across hosts and across time is a pain. Obviously you can composite a timestamp to an incrementing id, but then the serial part of it is kind of useless for ordering, so you may as well use a random number and avoid needing to synchronize at all. And then you've just reinvented a time ordered uuid (but to be fair, without the endian compatibility bs mentioned elsewhere).
- staticassertion 4y agoI'm wondering what that performance actually looks like though. The closest benchmark I have is a single postgres instance with a uuid pkey and a bigint column, transactionally incrementing the integer in the column. I mean I guess mathing it out, updating an atomic integer is ~2-100ns, depending on contention. If you need to coordinate the writes you have anywhere from ~250μs-10ms. We can basically throw away the increment at that point since the network is hundreds/thousands of times slower. So at 10ms, that's 100 op/s. At 250μs more like 40kops/s. That ignores the fact that your db can perform those writes concurrently and then batch the writes off to the other database, so long as it doesn't pretend that they're all committed at once. psql HOT updates would presumably be a thing here idk. If I had to guess, I'd lean towards the "40kop/s" being closer than the "100op/s" but idk! I wish we had benchmarks but I can't find anything :\
- Zamicol 4y agoWhy isn't there an option with a strong cryptographic hash like SHA-256?
- staticassertion 4y agoIt would be slow and doesn't really serve a purpose since you either have uuids that are totally random or uuids that need to preserve their structure.
- jwilk 4y agoOr UUIDs derived from other identifiers: https://datatracker.ietf.org/doc/html/rfc4122#section-4.3 https://datatracker.ietf.org/doc/html/rfc4122#section-4.3
- jamesrom 4y agoYou could utilize UUIDv8 for this purpose. Of course v5 isn't strong in the cryptographic sense, but probably suits most purposes where you need a unique ID per an object: https://datatracker.ietf.org/doc/html/rfc4122#section-4.3 https://datatracker.ietf.org/doc/html/rfc4122#section-4.3 The idea of using UUIDs for cryptographic purposes is probably misguided anyway.
- operator-name 4y agoWith the introduction of UUIDv8, my fun script to generate vanity uuids[0] can finally be spec comfortant! [0]:https://github.com/operator-name/vanity-uuid https://github.com/operator-name/vanity-uuid
- davidjfelix 4y agoI've been staring at the supposedly readable and memorable uuid for like 4 minutes without any idea what it says.
- operator-name 4y agoYeah, looking back I could have done a bit more than simple substitution. Since the sentences don't gave meaning, interpreting between 1 as i vs l is especially difficult. 5eedbed5-f05e-b055-ada0-d15ab11171e5 seedbeds-fose-boss-adao-disabilities "Memorable" was definitely tongue in cheek, as was the spec bending.
- canadiantim 4y agoHow would one go about trying to use UUID 7 in a Postgres database / python codebase now?
- oittaa 4y agoI made a simple Python test library that extends the standard UUID class with UUIDv6 and UUIDv7. You might want to check it out. https://github.com/oittaa/uuid6-python https://github.com/oittaa/uuid6-python The official UUID Draft repository has also some alternatives if you'd like to check those out. https://github.com/uuid6/prototypes https://github.com/uuid6/prototypes Postgres supports UUIDs with any version number natively so you can then do something like this with it: create table data (id uuid, firstname varchar(100)); insert into data (id, firstname) values ('017f21cf-d130-7cc3-98c4-dc0c0c07398f', 'John'); select * from data;
- Starlevel001 4y agoPostgres UUIDs are treated as an opaque 8-byte array, you can use anything as long as it's in the textual format.
- kjmph 4y agoI created a PL/pgSQL function for this purpose. As Starlevel001 mentioned, the UUID format will accept opaque bytes. Perhaps you could find this useful? https://gist.github.com/kjmph/5bd772b2c2df145aa645b837da7eca74 https://gist.github.com/kjmph/5bd772b2c2df145aa645b837da7eca...
- stevesimmons 4y agoMy Python implementation is here: https://github.com/stevesimmons/uuid7 https://github.com/stevesimmons/uuid7 https://pypi.org/project/uuid7/ https://pypi.org/project/uuid7/
- oittaa 4y agoNice to see another implementation which takes a bit different approach. Just for your information, there's now Draft 03 which changes the format a little bit. I kinda liked the arbitrary precision of Draft 02, but the newer one just requires millisecond precision and then basically leaves it up to the implementation how to handle the generation of multiple UUIDs within the same millisecond. https://github.com/uuid6/prototypes/issues/21 https://github.com/uuid6/prototypes/issues/21
- nixpulvis 4y agoMaybe I'm being slow right now, but can somewhat help me understand why Max UUID is ever specifically useful?
- okr 4y agoMy guess is, that it is just a constant, ready to be used for bitwise operations.
- nixpulvis 4y agoYep... I was just being slow. I thought the specialized variant was a new kind of UUID (being inverse of RFC4122), but it's just the inverse of RFC4122's Nil UUID; a single value. Sorry for the silly comment. This value is just a bunch of binary 1s.
- lewisl9029 4y agoI've been using ULID for a while now, which analogous to UUID v7 but with a different (better IMHO) string representation. They've been awesome for using as sort keys in dynamo for instance, since they're lexicographically sortable as strings. But one thing I'm still wary about is exposing these IDs with millisecond-precision time components to end users, since I've seen multiple discussions here on HN about the potential for timing attacks. How worried should I really be? Do people have useful heuristics on the kinds of data where it's safe/unsafe to expose timing information, or should I just only expose a separate UUID v4 externally across the board just to be safe?
- doliveira 4y agoI came up with a scheme in which the random part of the ULID is a slice of the hash of a UUID, which seems to work fine because I'm generating it all server side, I guess? It works very well for insertion into NoSQL databases, for instance. So I'd also like to know the threat model for these timing attacks.
- djbusby 4y agoULID has like 80 bits of random per millisecond. I found some calculator online that showed the probability of collision on 80bits was very, very low even across time much longer than millisecond. I think knowing when an ID was created is harmless - and if it isn't there is a design problem that is larger than ID choice. There is another lib called hashid which can be used if masking that ID algo is important.
- lewisl9029 4y ago> I think knowing when an ID was created is harmless - and if it isn't there is a design problem that is larger than ID choice. I hope you're right, since I've been cautiously operating under that assumption so far. Cautiously, because of warnings, from people better versed in cryptography than I, in discussions like this one: https://news.ycombinator.com/item?id=29805433 https://news.ycombinator.com/item?id=29805433. I still haven't quite been able to internalize when this should vs shouldn't be something I should be concerned about, hence the comment.
- swyx 4y ago(poster here) this RFC is not new new, but is still pretty new and i was surprised to learn that UUIDv7 and v8 are being worked on. the context is i keep a list of uuid impls and knowledge for my own reference. posted this up today simply because I got a PR from some subscribers https://github.com/sw-yx/brain/pull/36 https://github.com/sw-yx/brain/pull/36
- tqkxzugoaupvwqr 4y agoIs there a way to verify a UUID was created by me? I want to avoid database lookups of UUIDs from malicious/manipulated URLs.
- rossriley 4y agoYou could use the Luhn algorithm or something similar to add a check digit to the end, so for instance, generate a standard uuid, remove the last character, pass the remaining characters through the Luhn function and then use the result as the final replacement character for the UUID. It's obviously not cryptographically secure, as in someone else could use the same algorithm to generate UUIDs that your system recognises, but could be a quick way of doing a verification before you hit the DB.
- fer 4y agoDepends on your workload and how it's distributed, but a bloom filter might help you there. The filter needs to be stored somewhere, so if your workload is write-heavy you'd be replacing lookups with updates, but if it's read-heavy most DB lookups can be replaced with filter lookups.
- borplk 4y agoThe only secure way of doing it is to somehow crypotgraphically sign it. You'd have to trade that DB lookup with CPU cycles for a signature validation. For example you can use HMAC. You send the UUID and HMAC-of-UUID to the client. The client sends back UUID + HMAC-of-UUID. You re-calculate HMAC-of-UUID-provided-by-client (using your secret key). If the calculated HMAC matches the HMAC provided by the client you have confirmation that the UUID was issued by you. Because without the secret key the client can't calculate the correct HMAC for a random or modified UUID.
- billpg 4y agoWarning: The entirety of this comment might be a bad idea. There are 124 bits available. (Actually a little less but let's pretend only 4 bits are needed for the version to keep it simple.) You'll have to decide how many are the ID and how many are the signature. Let's say 32 bits are your ID and the remaining 92 are the signature. (Adjust according to your own needs.) Let's suppose your ID is 1. Now you need to hash that ID together with a private key using an HMAC algorithm. This signature will have more than 92 bits so trim your signature to this length. Now build your UUID by combining the 32 bits of your ID with the 92 bits of the signature and four bits specifying version 8 (proprietary). To test if a claimed UUID is yours, pull out the 32 bits of ID and repeat the process of signature and building a complete UUID. If the bits match, success! If not, its a fake UUID. Now read the responses to this comment to find out why you shouldn't do this.
- joshspankit 4y agoWhen I read the headline I thought “why do we need another UUID format?” but my respect was quickly earned. A lot of excellent thought put in to this, and thank you to the authors for casting aside the abomination called leap seconds.
- Daegalus 4y agoI maintain the Dart UUID library. For anyone using dart, or wants to see one of many implementations of v6,v7 and a custom v8 UUID, feel free to look at my in-progress branch linked below, I plan to merge it in once they add different string representations in a future draft (I've been involved in the conversations). https://github.com/Daegalus/dart-uuid/tree/uuidv6 https://github.com/Daegalus/dart-uuid/tree/uuidv6
- oofbey 4y agoI really don’t understand why we need standards for UUIDs. I get that with cryptography it’s super easy to make subtle mistakes. But UUID4 (the most commonly used variant these days) is just a long random number. Except for a few bits which aren’t random because the standard says so. As long as you’re using a goodsource of randomness (most are these days) it’s pretty hard to screw up. Why do we need a standard to tell us how to combine time stamps and random numbers?
- swyx 4y agothats what the authors of uuid4 thought. but if you read the RFC again you’ll see they listed out the flaws and pain points. eg the impact of true randomness on database writes at scale. having standards encodes lessons from a decade worth of pain. don’t dismiss it so casually.
- oofbey 4y agoSorry, I still don't get it. It's a timestamp plus a random number. As their references show, plenty of people have figured out this is a fine strategy. I could explain to an intern how to build this in 10 seconds with "use a 48 bit timestamp plus 80 bits of random, then encode it as a hex string" and they're done. Finding timestamps and high-quality RNG's is just not hard these days. Replace "hex string" with "base62 encoding" and we have something significantly better than a UUID. I get that the UUID authors feel a need to tell people there's a better way to do it, and I guess adding a new version to their standard is easier than telling people to go use something else like ULID. But I still don't see it as particularly important.
- kdps 4y agoMy main concern with random-based UUIDs always has been running out of entropy and cause the application to remain in a blocking state (e.g. as described here: https://blog.fastthread.io/2022/03/09/java-uuid-generation-performance-impact/ https://blog.fastthread.io/2022/03/09/java-uuid-generation-p...). Not due to any negative experiences that I made myself, but due to a particular colleague of mine who sees it as a dealbreaker. Is this an actual issue? Most people don't seem to care when talking about random UUIDs. The target platform of our applications is mostly Kubernetes on cloud environments, if that makes any difference. Why I'm asking: UUID Version 7 looks quite interesting to me, and the document describes rand_a and rand_b just as "pseudo-random data"... which made me think that in the context of "uniqueness per millisecond", a source of entropy is conceptually not required. However, chapter 6.6 clearly advises the usage of CSPRNGs, so I guess the overall problem remains :(
- astrange 4y ago“Running out of entropy” is not possible; that’s a property of ancient PRNGs written by confused people. Even if your PRNG could run out of entropy, rdrand would give it all it needs.
- ciupicri 4y agoFrom the documentation for java.security.SecureRandom from Java 17 [1]: > Note: Depending on the implementation, the generateSeed, reseed and nextBytes methods may block as entropy is being gathered, for example, if the entropy source is /dev/random on various Unix-like operating systems. [1]: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/security/SecureRandom.html https://docs.oracle.com/en/java/javase/17/docs/api/java.base...
- yencabulator 4y agoOn current day Linux, /dev/random will only block during bootup, never after it has once started.
- astrange 4y ago
- johntony678 4y ago[flagged]
- kagitac 4y agoCan someone ELI5 why Unix timestamp would be stored as left-padded 48 bits for a 32-bit or potentially a 64-bit number? Is it a coincidence that 48 is the middle of 32-64?
- jojohohanon 4y agoIs the author of the spec here? I notice in section 6.1 reliability, the word “must” in clock going backwards mitigation is not MUST. Intentional?