16 ms·
Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs
- wood_spirit 3y agoUUIDv7 has been taking HN by storm for years now! When is it going to become a proper standard, and when are libraries and databases and all the rest going to natively support it?
- kijeda 3y agoIt would appear to be in the final stages of standardization in the IETF: https://datatracker.ietf.org/doc/draft-ietf-uuidrev-rfc4122bis/ https://datatracker.ietf.org/doc/draft-ietf-uuidrev-rfc4122b...
- Daegalus 3y agoIts been going through drafts and improvements. It's very close to being standardized, and many libraries are supporting it already, or new offerings are being added. For example I maintain the Dart UUID library, and my latest beta major release has v6, v7 and a custom v8. There is a list of them somewhere, I know I get pinged on every new draft by the authors because I am listed as a library maintainer on one of their pages.
- Nelkins 3y agoHow much does it change between drafts? Close enough to where I could use it in production?
- Daegalus 3y agoSeeing as how its nearly done, it doesn't change much. It changed more often in the beginning, but its like on its final draft, or near final draft. I think the IETF plans to make final soon.
- vbezhenar 3y agoWhat kind of support do you expect? I'm pretty sure that absolute majority of software does not care about any particular bits in UUID, so you can use it today. If some software cared about any particular bits, just imitate UUIDv4, I mean those bits could be randomly generated as well. If you need generation procedure, write it yourself, it's easy.
- dolmen 3y ago+1 IDs generation is usually private to a company scope and rarely need to be "universally unique".
- ajkjk 3y agoUnrelated, but this links to "Crockford's alphabet", https://www.crockford.com/base32.html https://www.crockford.com/base32.html , which is a base-32 system that includes all alphanumeric characters except I and L (which are confusable with 1), O (which is confusable with 0), and U (????). The page says the reason for excluding U is "accidental obscenity'. What the heck is it talking about?
- jszymborski 3y agoFUCK
- Racing0461 3y agoyep, youtube video ids has/had? same issue where it would have things like fag/f4g etc in it. eg: google "allinurl:fag site:youtube.com"
- stronglikedan 3y agoYou can prevent any obscenity, O and 0 confusion, and I and L confusion, just by excluding vowels. If someone interprets "f4g" in an offensive way, then they have bigger issues than can be dealt with in software.
- taosx 3y agoWhy we care about obscenity in pseudo-random ids and url?
- whimsicalism 3y agoanglo morals
- Racing0461 3y agoi don't, the people that can start a **storm on twitter/tumblr causing your stock to drop 10% does.
- deleted 3y ago[deleted]
- Xeoncross 3y agoAssuming you don't need to use UUIDv7 (or any UUID's) then https://github.com/segmentio/ksuid https://github.com/segmentio/ksuid provides a much bigger keyspace. You could just append a string prefix if you wanted to namespace, but the chance of collisions of a ksuid is many times smaller than a UUID of any version. ksuid is the best general purpose id generator with sort-able timestamps I've found and has libraries in most languages. UUID v1-7 are wasteful.
- tasn 3y agoWe maintain a couple of popular ksuid libraries[1][2] and use it, so we definitely like ksuid. Though one big issue with ksuid is that being 160bit means that it doesn't fit into native uuid types in databases (e.g. postgres), which means that they come with a performance penalty. 1: https://github.com/svix/rust-ksuid https://github.com/svix/rust-ksuid 2: https://github.com/svix/python-ksuid https://github.com/svix/python-ksuid
- Xeoncross 3y agoI'm curious, why do you not store these as binary data or do you and you're saying that the UUID operations are better optimized than sorts on binary data?
- rtheunissen 3y agoI moved to ULID because they are always lowercase and therefore case-insensitive.
- atulvi 3y agoNaive Question: The type safe part is just appending a string at the beginning? What if I do that with UUIDv4? is user_49b9cd12-9964-4b9c-8512-742f0a2c9be4 type safe now?
- dloreto 3y agoThat's how the type is encoded as a string, but type-safety ultimately comes from how the TypeID libraries allow you to validate that the type is correct. For example, the PostgresSQL implementation of TypeID, would let you use a "domain type" to define a typeid subtype. Thus ensuring that the database itself always checks the validity of the type prefix. An example is here: https://github.com/jetpack-io/typeid-sql/blob/main/example/example.sql https://github.com/jetpack-io/typeid-sql/blob/main/example/e... In go, we're considering it making it easy to define a new Go type, that enforces a particular type prefix. If you can do that, then the Go type system would enforce you are passing the correct type of id.
- davidjfelix 3y agoYep. The whole point is that you /never/ assign ids that begin with "user" to types that are not users. Because of that, you can be sure nobody can accidentally copy an id that begins with "user" when meaning to address a different type and get back a result other than "not found". Example: I have userId=4 and userId=2. Suppose a user can have multiple bank accounts and userId=4 has accountId=5 and accountId=6 and a defaultAccound accountId=5. userId=2 has an account, accountId=7; I want to send userId=4 some money so I use the function `sendUserMoneyFromAccount(to: int, from: int)`. This is a bad interface but these things exist in the wild a lot. I could accidentally assume that because I want to send userId=4 the money to their default account that I would call it using `sendUserMoneyFromAccount(4, 7)` and that would work, but if under the hood it wants 2 accountIds, I've just sent accountId=4 money rather than userId=4's defaultAccount, accountId=5. With prefixed ids that indicate type, a function that assumes type differently from the one supplied will not accidentally succeed. In addition, humans who copy ids will be less likely to mistake them. This is just an ergonomic/human centric typing.
- deleted 3y ago[deleted]
- bombela 3y agoI have some complaints about UUIDs. Why not just combining time + random number without the ceremony of UUID versioning. And for when locality doesn't matter, just use a 128bit random number directly. And in my experience most people somehow think a UUID must be stored into the human friendly hex representation, dashes included. Wasting so much space in database, network, memory.
- hot_gril 3y agoThe only worthwhile UUID standard IMO is v4 (simple random), and I still don't get why it needs dashes. The other ones don't really accomplish anything.
- stronglikedan 3y ago> combining time + random number You can't guarantee that this will be globally unique.
- ceejayoz 3y agoNo identifier can guarantee that. We just get close enough to be acceptable. Per Wikipedia, the probability to find a duplicate within 103 trillion version-4 UUIDs is one in a billion. so-youre-saying-theres-a-chance.gif
- duped 3y agoA billion is not that big of a number for UUIDs
- ceejayoz 3y agoRe-read. You'd have to generate 103 trillion to have a one billionth chance of a collision. A billion isn't that big a number, but 103 trillion is.
- jandrewrogers 3y ago
- pphysch 3y agoI implemented something similar recently, but opted to write my own UUIDv7 that uses the last 16 flexible bits for a "type ID". That allows 65K different data models, which should be more than enough. Could even partition that further to store a node ID for a globally distributed setup. So it's got all of the above perks, but it also an actual UUID and fits in a Postgres UUID column. It's very cool to be able to resolve a UUID to a particular database table and record with almost zero performance overhead (cached table lookup + indexed record select).
- Daegalus 3y agoIdeally, you should version that as UUIDv8. Since that is a more custom implementation, but I guess changing the random bits with type info works fine for UUIDv7, they jsut arent random anymore.
- pphysch 3y agoNot every bit of UUID is required to be random. The goals are smallness, uniqueness, monotonicity, resistance to enumeration attacks, etc. Not randomness for randomness sake. My UUIDv7+ can be consumed as a standard UUIDv7. It is not intended to be v8. A program can treat the last 16 bits as random noise if it wants.
- Daegalus 3y agoI am aware, just saying per spec, its supposed to be random bit data, thats all I was saying. I am familiar with a spec since I maintain a UUID library that has 6,7, and a custom 8 implemented. It can have extra monotonicity data instead, per section 6.2 but ideally its random. Again, Not saying you can't do what you are doing, I just know per the conversations while the draft was gathering feedback, your type of change was intended to be done as uuidv8
- pphysch 3y ago> per spec, its supposed to be random bit data > It can have extra monotonicity data instead Well, which is it? These are incompatible requirements. If I give you a standard UUIDv7 sample, it is impossible for you to interpret the last 62 bits. You cannot determine how they were generated. If I give you two samples with the same timestamp, you cannot say which was generated first. These bits are de facto uninterpretable, unlike e.g. the 48 MSB, which have clearly defined semantics.
- vikeri 3y agoAnother, less known, useful thing about these IDs is that you can double click on them and the full id will always be selected
- jrockway 3y agoThis is a setting in your terminal emulator. For me, plain UUIDs are selected just fine when double clicking.
- mojuba 3y agoThere's life outside of the terminal. For example you want to double-click on the part of a URL in your browser.
- avarun 3y agoIt's mentioned in the README. "can be selected for copy-pasting by double-clicking"
- Eduard 3y agoAlso, they are safe to use within filenames and directory names (Filesystem paths) without conversion (at least in today's Filesystem not limited to e.g. 8.3 characters) . Compare that with otherwise nice ISO 8601 datetime format (e.g. 2023-06-28T21:47:59+00:00): it requires conversion for file systems that don't allow colons and plus signs.
- jszymborski 3y agoThis is very similar to how I generate IDs in a project I'm working on. Example: |-A-|-|------------B--------------| NMSPC-9TWN1-HR7SV-MTX00-0H8VP-YCCJZ A = Namespace, padded to 5 chars. Max 5 chars. Uppercase. B = Blake3 hashed microtime with a random key. I like how it folds in a time component but that it also doesn't reveal the time it was generated. Here's the snippet: https://gist.github.com/jszym/d3c7907b7b6e916f68205c99e5e48922 https://gist.github.com/jszym/d3c7907b7b6e916f68205c99e5e489...
- goostavos 3y agoNamespacing identifiers in general is a great idea for handling those class of integration tests which cannot be fully isolated. It makes it easy to write all kinds of garbage from even concurrently running tests all without any of them colliding or accidentally reading each others writes (because they are themselves namespace aware!). It's low effort to get all the pieces of your system to play along (often entirely transparent via DI), but gives a huge power to weight ratio. Basically deletes an entire class of problems which usually plague large, mature test suits
- rowbin 3y agoDo you generate a random key for each id? If so, you loose the time component of your I'd, because two hashes made at a different time can produce the same hash when using a different key (I think). If you use a single key you need to manage it and keep it secret which isn't really ideal for something simple like id's. If you reveal the key you could as well use an unkeyed hash, right? Maybe a salted hash with the same salt for every key to make dictionary attacks on the creation time harder. But then again it's probably easier to just generate random bits for the id, as you don't have sortability in any case.
- AtNightWeCode 3y agoAn important aspect of identifiers is to not leak any information in the identifier. In some scenarios a prefix might be fine but less important things have been blocked by our dpo department.
- jeremyjh 3y agoYeah, this idea makes me uneasy as well for the same reason. The natural conclusion would be to have DB frameworks automatically do this based upon code type names and that definitely feels like a leak, handing the world at large a roadmap to your internal system architecture.
- clintonb 3y agoRequirements depend on the use case. I don’t consider the prefix a “leak” and neither does Stripe.
- AtNightWeCode 3y agoYeah and Twitter IDs used to be 32-bit integers. In this case it is kind of obvious that the IDs leaks information about internal data types. Which is not ok at many companies.
- timf 3y agoI do a similar thing [1]. One of the great advantages to formally namespaced IDs is including a systematic conversion into strong types in your code. It's harder to accidentally mix things up when coding; function parameters and return tuples are more 'self documented' (and enforced by compiler where applicable). [1] - https://www.peakscale.com/strongly-typed-ids/ https://www.peakscale.com/strongly-typed-ids/
- eezing 3y ago“…can be selected for copy-pasting by double-clicking” Details matter.
- koito17 3y agoHow does this compare to a SQUUID for sorting or nano-id for human readability? Both are options I've used in the past when using databases like Datomic or XTDB. SQUUIDs in particular because I have a UUID that can be ordered by timestamp, nano-id when prototyping things and I want meaningful prefixes in my entity IDs rather than a bunch of UUIDs.
- ajanuary 3y agoNeat, I like the type safe prefix idea. Personally, I rarely find I need ids to be sortable, so I just go with pure randomness. I also like to split out a part of the random section into a tag that is easier for me to visually scan and match up ids in logs etc. I call my ID format afids [0] [0] https://github.com/aJanuary/afid https://github.com/aJanuary/afid
- changoplatanero 3y agowhat does k-sortable mean?
- sixtram 3y ago[dead]
- tomnipotent 3y agoThat if a hundred servers are generating (uuid, timestamp) tuples that are subsequently merged on a single machine, and sorted by uuid, it would have almost the same order as if sorted by timestamp. This property is useful for RDBMS writes, when the UUID is used as a primary key and this locality ensures that fewer slotted pages need to be modified to write the same amount of data.
- nzgrover 3y agoIs that what they mean by "used as the primary key in a database while ensuring good locality"/"database locality"? That read/write access will hit fewer disk pages?
- netcraft 3y agoyes, exactly
- nickjj 3y ago> it would have almost the same order as if sorted by timestamp. Is there documentation covering the scenarios on how the order can become out of sync and what the odds are? There's a big difference between "almost" and "always" if we're talking about using this as a database PK.
- tomnipotent 3y ago> There's a big difference between "almost" and "always" Not in the context of an RDBMS, which use b+/b*-tree variants (or LSM sstables). Sequentially generated UUID's will end up near each other when sorted lexicographically, regardless of the fact that the sort order doesn't perfectly match the timestamp order.
- stephen 3y agoNeat! Love the "type-safe" prefix; we'd called them "tagged ids" in our ORM that auto-prefixes the otherwise-ints-in-the-db with similar per-entity tags: https://joist-orm.io/docs/advanced/tagged-ids https://joist-orm.io/docs/advanced/tagged-ids We'd used `:` as our delimiter, but kinda regretting not using `_` because of the "double-click to copy/paste" aspect... In theory it'd be really easy to get Joist to take "uuid columns in the db" and turn them into "typeids in the domain model", but probably not something that could be configured/done via userland atm...that'd be a good idea though.
- wongarsu 3y agoReddit does something similar, but optimized for string length: elements have ids like "t3_15bfi0" where t3_ is a prefix for the type (t3 is a post, t1 a comment, t5 a subreddit, etc) and the remaining is a base36 encoding of the autoincrementing primary key.
- stephen 3y agoNice! The `t<X>` makes sense; we currently guess a tag name of "FooBarZaz" --> "fbz", but allow the user to override it, so you could hand-assign "t1", "t2", etc. as you added entities to the system. Abbreviating/base36-ing even the auto-incremented numeric primary key to optimize lengths is neat; we haven't gotten to the size of ids where that is a concern, but it sounds like a luxurious problem to have! :-)
- hamburglar 3y agoMy company has a typed internal ID system that originally used colons as delimiters but we quickly switched to dots (.) as the delimiter because it’s very annoying to have url-encoded IDs balloon in size because colons need to be %-encoded. Makes your urls ugly and long.
- ukuina 3y agoWhy is it beneficial to sort random IDs?
- davidjfelix 3y agoSo you don't have to create an additional index to scan them in a somewhat sensible order. createdAt just happens to be a naturally decent scan order.
- swyx 3y agofor those researching this topic, I keep a list of these UUID/GUID implementations! https://github.com/swyxio/brain/blob/master/R%20-%20Dev%20Notes/uuid%20list.md https://github.com/swyxio/brain/blob/master/R%20-%20Dev%20No...
- crdrost 3y agoThanks for this! I have one idea which is perhaps nerdy enough to make the list but I've never fully fleshed it out, it's that one can encode the nonnegative integers {0, 1, 2, ...} into the finite bitstrings {0, 1}* in a way which preserves ordering. So if we use hexits for the encoding the idea would be that 0=0, 1=1, ... E=14, then F00 = 15 F01 = 16 ... F0F = 30 F100 = 31 F101 = 32 ... F1FF = 286 F2000 = so the format is F, which is the overflow sigil, followed by a recursive representation of the length of the coming string, followed by a string of hexits that long. What if you need 16 hexits? That's where the recursion comes in, F F00 0123456789ABCDEF \ \ \ \ \ \----- 16 hexits \ \ \ \-- the number 15, "there are 15+1 digits to follow" \ (consisting of overflow, 0+1 digits to follow, and hex 0) \ \--- overflow sigil Kind of goofy but would allow a bunch of things like "timestamp * 1024 + 10-bit machine ID" etc without worrying about the size of the numbers involved
- Ironchefpython 3y agohttps://datatracker.ietf.org/doc/html/rfc2550 https://datatracker.ietf.org/doc/html/rfc2550 Note that this RFC also supports lexicographic sorting for negative numbers using the 10k complement notation.
- CMCDragonkai 3y agoWe have a uuidv7 implementation that we've been using with rocksdb for over a year https://github.com/matrixai/js-id https://github.com/matrixai/js-id
- iillexial 3y agoI didn't get the "type-safe" part. How would it work in Go? Let's say I have structs: type User struct { ID TypeID } type Post struct { ID TypeID } How can I ensure the correct type is used in each of the structs?
- kibwen 3y agoAny time you ever read a string, its type is always just going to be "string" (modulo whatever passes for a "string" in your programming language of choice). To get an actual non-string type, you'd need to parse that string, and presumably your parsing function would read the prefix and reject the string if it was passed an ID whose type doesn't match. So it's dynamically type-safe, if not statically type-safe.
- hfkwer 3y agoThis isn't about object types in any particular language.
- leetbulb 3y agoOne way is to enforce in Marshal[0] and Unmarshal[1] [0] https://pkg.go.dev/encoding/json#Marshaler https://pkg.go.dev/encoding/json#Marshaler [1] https://pkg.go.dev/encoding/json#Unmarshaler https://pkg.go.dev/encoding/json#Unmarshaler
- zeroxfe 3y agoIt's not a language primitive. It's a data format that enables type safety in libraries or APIs (as opposed to a more opaque data format like UUIDv7.)
- avgcorrection 3y agoIt’s stringly-typed type-safety: check if the value has the expected prefix.
- Merad 3y agoI don't know go, but in C# I'd probably do something like the code below. The object really only needs to carry the uuid/guid, let the language type system worry about the difference between a user and post id. We just need a generic mechanism to ensure that UserId object can only be constructed from a valid type id string with type = user. For production use you'd obviously need more methods to construct it from a database tuple (mentioned elsewhere in the comments), etc. interface ITypeIdPrefix { static abstract string Prefix { get; } } abstract class TypeId<T> where T : TypeId<T>, ITypeIdPrefix, new() { public Guid Id { get; private init; } public override string ToString() => $"{T.Prefix}_{Id.ToBase32String()}"; // Override GetHashcode(), Equals(), etc. public static bool TryParse(string s, out T? result) { if (!s.StartsWith(T.Prefix) || !TrySplitStringAndParseBase32ToGuid(s, out var id)) { result = default; return false; } result = new T { Id = id }; return true; } } class UserId : TypeId<UserId>, ITypeIdPrefix { public static string Prefix => "user"; } class PostId : TypeId<PostId>, ITypeIdPrefix { public static string Prefix => "post"; }
- TeeWEE 3y agoGood, but I dont see a big advantage over UUIDv7 Anyone has some good ones?
- dloreto 3y agoIt's based on UUIDv7 (in fact, a TypeID can be decoded into an UUIDv7). The main reasons to use TypeID over "raw" UUIDv7 are: 1) For the type safety, and 2) for the more compact string encoding. If you don't need either of those, then UUIDv7 is the right choice.
- kortex 3y agoDoes the prefix ("user_") get recorded in the DB (so every string in the column starts with the same "user_"), or does are there constraints and other clever chicanery to save those bytes in every record? Or do modern DB engines even care? Is this premature optimization?
- carlsverre 3y agoThe authors have created a specialisation for Postgres that leverages a custom type which is a tuple of type and uuidv7: https://github.com/jetpack-io/typeid-sql/blob/main/sql/typeid.sql https://github.com/jetpack-io/typeid-sql/blob/main/sql/typei... This is more optimal for Postgres while making it slightly more difficult to interop between the db and the language (db driver needs to handle custom types, and you need to inject a custom type converter). And while there are hacks you can do to make storing uuid-alikes as strings less terrible for db engines, if you want the best performance and smallest space consumption (compressed or not) make sure to use native ID types or convert to BINARY/numeric types.
- hot_gril 3y agoIn my experience, using just uuid as a pkey in Postgres already causes noticeable slowdowns vs the typical bigint. I wouldn't jump into anything other than bigint pkeys unless I'm solving an existing problem and have benchmarks to prove it.
- wg0 3y agoCan anyone guide me about the pros and cons of xid, ksuid and this type-safe option?
- aartav 3y agoI've been doing this kind of thing for years with two notable differences: 1. I don't believe people actually hand type-in these values, so I'm not really concerned about the 'l' vs '1' issue. I do base 32 without `eiou` (vowels) to reduce the likelihood of words (profanity) sneaking in. 2. I add two base-32 characters as a checksum (salted of course). This is prevents having to go look at the datastore when the value is bogus either by accident or malice. I'm unsure why other implementations don't do this.
- sokoloff 3y ago> base 32 without `eiou` (vowels) to reduce the likelihood of words (profanity) sneaking in. We had “analrita” as an autogenerated password that resulted in a complaint many years ago. Might consider adding ‘a’ as an excluded letter.
- michaelt 3y agoPresumably base 32 means 26 letters + 10 digits - 4 banned letters So adding an excluded letter is not easy.
- sokoloff 3y agoWhy not use base-31 and (optionally) more characters? (Or go upper and lower or add a symbol if you had to stay with a fixed-size and base-32 for some reason)
- chipsa 3y agoBecause Base32 is just bit shifting and then converting the 5 bits into a char, and vice versa. Doing Base31 requires base conversion.
- manquer 3y agoWouldn’t that be excluded because i is already removed ?
- jtmarmon 3y agoThis looks great! Is there a reason one couldn't use this with v4 UUIDs? A quick test shows that they encode/decode just fine. Wondering if I could use the encoded form as a way to niceify our URLs without having to change how the IDs (currently v4 uuids) are stored
- dloreto 3y agoThe CLI tool will support encoding/decoding any valid UUID, whether v1, v4, or v7. We picked v7 as the definition of the spec, because we need to choose one of them when generating a new random ID, and our opinion is that by default, that should be v7. We might add a warning in the future if you decode/encode something that is not v7, but if it suits your use-case to encode UUIDv4 in this way, go for it. Just keep in mind that you'll lose the locality property.
- lll-o-lll 3y agoK-Sortable is a great concept; having weakly sorted keys solves a bunch of use-cases. I really like the idea of a typed, condensed string representation. However I wonder if an unintended side affect of UUID V7 is going to be a bunch of security problems. People aren’t meant to use uuids as tokens, and they aren’t supposed to use PKs from a DB for this either - but they do. Because UUID v4 is basically crypto random, I think we’ve been getting away with a bunch of security weaknesses that would otherwise be exploited. With UUID v7 we are back to 32bits of actually random data. It’s going to require some good educating to teach devs that uuids are guessable. [edit] Looks like I am off base with the guess-ability of the V7 UUID, as the draft recommends CSPRNG for the random bits, and the amount of entropy is at least 74 bits and it is specifically designed to be “unguessable”. It does say “UUID v4” for anything security related, but perhaps that is simply in regard to the time stamp?
- atonse 3y agoThis brings up an interesting ergonomics problem. By naming them UUIDv4 and UUIDv7, is it going to be this never ending confusion for people to have to remember which one is good for databases and which one good for one time tokens? Not sure what the backwards compatible solution here is either. In elixir the function is UUID.uuid4() to generate a v4 UUID. So we could theoretically scan code for its use I suppose. But all this increases chances of errors.
- hot_gril 3y ago> is it going to be this never ending confusion for people to have to remember which one is good for databases and which one good for one time tokens Yes, because this is what's been happening already with the past versions. It's not just sequential and random, there are also hash-based UUIDs. They shouldn't have sequential (heh) version numbers.
- hot_gril 3y agoI can see some use cases for it, but every time in the past I've encountered other kinds of partially-sequential UUIDs like v1 or v5, they've been misused. Same with the hash-based ones like v3. v4 is simple and not prone to misuse.
- klabb3 3y agoA couple of suggestions: Lock down the prefix string now before it’s too late and document it. I see in Go that it’s lowercase ascii, which seems fine except for compound types (like “article-comment”). May be worth looking at allowing a single separator given that many complex projects (and ORMs) can’t avoid them. The Go implementation has no tests. This is very unit-testable. Add tests goddammit! For Go, I’d align with Googles UUID implementation, with proper parse functions and an internal byte array instead of strings. Strings are for rendering (and in your case, the prefix). Right now, it looks like the parsing is too permissive, and goes into generation mode if the suffix is empty. And the SplitN+index thing will panic if no underscores, no? Anyway, tests will tell. As for the actual design decisions, I tried to poke holes but I fold! I think this strikes the sweet spot between the different tradeoffs. Well done!
- avgcorrection 3y ago> The Go implementation has no tests. This is very unit-testable. Add tests goddammit! Yep. The readme asks people to provide other implementations. Having a test suite would be good for third-party code.
- tomcam 3y ago> Add tests goddammit! Hey, you’re pretty smart. How about you add them?
- klabb3 3y agoI’m by no means a test police. I’m in fact opposed to a lot of mindless testing for the sake of it. But there are places where unit tests shine, and this is one of them. If you mean that criticism is only allowed if you are willing to commit labor, I disagree with that. I always welcome critique myself - it may be something that I’ve missed. The maintainers always has the last word. As long as there are no hidden expectations, it’s all good.
- tomcam 3y agoDo you need to be “test police“ to add tests you literally demanded of them?
- onlypositive 3y ago> Compare to entirely random global ids, like UUIDv4, that generally suffer from poor database locality. What does this mean in more words?
- armchairhacker 3y agoBecause the bytes are all random, UUIDv4 is sorted randomly. So whenever you insert a database entry with a new UUID, it ends up getting put in some random memory location. In practice, you often want to select database entries which were inserted near each other in time. Ex: you would like to select the most recent entries or entries within a time frame. Even when selecting entries by other information, entries inserted closer to each other in time are generally more likely to be related. Fetching entries near each other in memory is faster, so it would be nice to insert entries sequentially in time; we want entries inserted near each other in time to be located near each other in memory. This is what counter-based indexing does: the database has a counter which increments on each insertion, and the current value becomes the inserted entry's id. But the problem with counters is when the database is distributed and insertions are happening in parallel, and you definitely don't want to sync the counter because that's way too slow. UUIDv7 combines a sort of counter (Unix time) with a randomly-generated number. The counter bytes are first, so they determine the sort order; but in case 2 entries get inserted at the same time, the randomly-generated number keeps them distinct and totally ordered.
- hot_gril 3y agoRight, but worth noting that distributed DBs don't necessarily play nice with sequential pkeys. Spanner explicitly tells you not to use a time-based pkey.
- nhumrich 3y agoI feel validated. I wrote about this very problem before and came up with something similar https://dev.to/nhumrich/why-i-dont-like-uuids-5d9n https://dev.to/nhumrich/why-i-dont-like-uuids-5d9n
- beyonddream 3y agoThis looks really interesting! Tangentially related if anyone is interested, I recently wrote[1] a pure C (no external dependency) version of coordination free, k-ordered 128-bit UUID Generator library which is inspired by snowflake but has bigger key space and few nifty features to protect against clock skew etc. The 128 bits are split into {timestamp:64, worker_id:48, seq: 16} where the seq is incremented if the unique id is requested within the same millisecond. [1] - https://github.com/beyonddream/snowid https://github.com/beyonddream/snowid
- hot_gril 3y agoI don't understand putting type names into DB row IDs. You're safest using whatever IDs in your DB make it happiest (usually bigserial in Postgres), and the needs might be pretty specific. Whenever you want to log row IDs, you add whatever context is needed, which will probably include things besides the ID either way. When you want to share an identifier with a customer, you use something entirely different. The UUIDv7 properties are interesting, but it's worth noting that at least one DBMS really doesn't like the K-sortable property: https://cloud.google.com/spanner/docs/schema-design#uuid_primary_key https://cloud.google.com/spanner/docs/schema-design#uuid_pri...
- deleted 3y ago[deleted]
- quelltext 3y agoJust think about logging polymorphic object IDs. If you see an identifier with a type you can immediately know what table it is in and you can even build super simple tools like browser extension that let you look up objects by IDs as is. You can also just share these IDs as is with folks on Slack to debug etc. and they know exactly what kind of entities you are talking about. Super useful and I honestly don't wanna go back to a project that doesn't use this pattern.
- hot_gril 3y agoThose debug IDs (or URLs, etc) are worth having for the reasons you describe, but it doesn't mean you use those as primary keys in the DB. Something only needs to print them, and debug tools need to understand them.
- BugsJustFindMe 3y agoSkip the base32 because it's cargo culting and go with a higher base. The premise of crockford's chosen exclusions is fundamentally flawed. In arbitrary fonts, symbol collisions are arbitrary. There are plenty of fonts where any of 5 and S, 2 and Z, V and U, G and 6, 8 and B, and 9 and g are confusable. Likewise vv and w, nn and m.
- inopinatus 3y agoI'm not wild about the Crockford encoding. In practice I've found it to be a flat-out mistake when you come to provide technical support or analysis for values encoded this way. The Crockford alphabet is based on design goals that are rarely encountered in practice, such as pronouncing identifiers over the phone. It introduces ambiguity, which is a disaster for grepping logs or any other circumstances where you might query or cross-reference based on the encoded string instead of the decoded value, then permits hyphens, a leading source of cut-and-paste and line-break errors. Note that people generally do not type in object identifiers, but they do frequently cut-and-paste them between applications and chat/forum interfaces, forward them by email, search for them in log files. Verbal transmission is rare to non-existent. Under these conditions, pronunciation proves irrelevant, and case-insensitivity becomes an impediment, but consistency and paste/break resilience become necessary. Base 58 offers a bijective encoding that fits these concerns much more effectively and is more compact to boot. Similarly inspired by Stripe, I've been using type-prefixed base58-encoded UUIDs for object identifiers for some years. user_1BzGURpnHGn6oNru84B3Ri etc. Edit to add: to be fair to Douglas Crockford, his encoding of base 32 was designed two decades ago, when the usage landscape looked quite different.
- Lazare 3y agoI agree; base58 or base62 (which KSUIDs use) have a lot to recommend them. Crockford's base32 works, but I don't love it. My first choice would be to just use type-prefixed KSUIDs, which gives you 160-bit K-sortable IDs with base62 encoding, which works great unless you need 128-bit IDs for compatability reasons.
- dloreto 3y agoI hear you ... and I debated using either base58 or base64url. I do like the more compact encoding they provide. Ultimately I ended up leaning towards a base32 encoding, because I didn't want to pre-suppose case sensitivity. For example, you might want to use the id as a filename, and you might be in an environment where you're stuck with a case insensitive filesystem. Note that TypeID is using the Crockford alphabet and always in lowercase – *not* the full rules of Crockford's encoding. There's no hyphens allowed in TypeIDs, nor multiple encodings of the same ID with different variations of the ambiguous characters.
- VanillaCafe 3y agoA pet nit, and the standards probably don't permit this, but for encoding 128-bit numbers, I prefer base-57 in my own implementations. 22 characters for a 128-bit encoding, same as base-64. You can split it into two 11-character 64-bit encodings. You can avoid the two non-alphanumeric characters in base-64 as well as the similar-looking characters like l1 and oO0. And it takes less visible space, so a bit easier for debugging and tabular output with otherwise no loss of generality.
- darajava 3y agoIn the readme, Crockford’s alphabet is referenced [0]. In this specification, “U” is excluded because it is an “accidental obscenity”. Does anyone have any idea what that means? Is it a joke? 0. https://www.crockford.com/base32.html https://www.crockford.com/base32.html
- deleted 3y ago[deleted]
- autoexec 3y agoI'm guessing that it's because U just happens to appear in some popular four letter words, which might make things awkward when you're reading letters out to another person over the phone. It might also come before a lot of other rude letter combinations making them seem more personal.
- matthewfcarlson 3y agoI actually implemented something similar for a personal project with a different syntax. It was two characters for the type, a colon, and then a UUID. Made it really easy to tell what primary key it corresponded to.
- dap 3y agoIt's weird to me that people talk about UUIDv4 as bad for databases because of locality. This behavior can be good or bad, depending on what you're doing. Sure, UUIDv4 is worse if you're using it to locate a bunch of related objects that you want to be on the same database node. (If you're doing that, though, you can often attach some kind of group identifier -- e.g., "user" -- and index that.) I generally prefer UUIDv4 in a lot of distributed database applications because as a sharding key it's likely to distribute data well across all available nodes. I debugged a system that used the equivalent of time-ordered UUIDs where even though the database was horizontally scaled, the performance was limited by the capacity of one server. It was exactly because the uuids being generated started with similar prefixes, so at any given time, all the current data was going to whichever database node was responsible for that range of the keyspace.
- hot_gril 3y agoYep. Spanner docs even tell you not to use time-ordered keys.
- ongteckwux 3y agoDid a Typescript port. Thanks for creating this! Very useful. https://github.com/ongteckwu/typeid-ts https://github.com/ongteckwu/typeid-ts
- tlrobinson 3y agoI really like prefixed identifiers and strongly recommend using them from the beginning. One question: I’ve seen people use both integer IDs (as the primary key) and prefixed GUIDs (for APIs) on the same table. This seems confusing and wasteful. Is there a valid reason to do that, and is it common? Performance when doing joins or something? EDIT: it sounds like that’s the reason for them to be “K-Sortable”: > TypeIDs are K-sortable and can be used as the primary key in a database while ensuring good locality. Compare to entirely random global ids, like UUIDv4, that generally suffer from poor database locality.
- jhoechtl 3y agoI am no big fan of the semantic web but the prefix should be an URI to be able to segment the domain. A single prefix is IMHO not enough.
- sophiabits 3y agoSomething really nice about type prefixes in IDs is it makes “Global Object Identification” [1] in GraphQL super straightforward. The node query can simply inspect the ID and immediately know where to fetch the object from, whereas with “regular” IDs you either need to perform a bunch of database queries or otherwise maintain some index that maps IDs->types. Not implementing this isn’t the end of the world, but it makes refetching individual bits of data in your cache far easier. It’s really nice being able to implement it basically for free. [1] https://graphql.org/learn/global-object-identification/ https://graphql.org/learn/global-object-identification/
- BiteCode_dev 3y agoNice. I should use a similar id with ULID. Although, if stored in a DB, I would probably split the tag and the id in 2 separate columns, because DBMS often have a dedicated efficient UUID field type. A little client code wrapper can make that transparent.
- quelltext 3y agoWouldn't it make sense to allow encoding additional data in the suffix? For instance a sharding key (or whatever you wanna call it)?