4 ms·
We should consider the following properties when evaluating ID formats and generation algorithms: 1. Private: you shouldn’t be able to gain information about t
by vlmutolo 4y ago
We should consider the following properties when evaluating ID formats and generation algorithms:
1. Private: you shouldn’t be able to gain information about the system using the IDs from an ID alone. E.g. document enumeration attacks like what happened with Parler (https://www.wired.com/story/parler-hack-data-public-posts-images-video/ https://www.wired.com/story/parler-hack-data-public-posts-im...)
2. B-tree/cache friendly: newly created IDs should all exist in a narrow range of values. This is helpful for databases.
3. Stateless: ideally you shouldn’t need to know the current state of the system to create a new ID.
4. Human-friendly: IDs should be easily dictated, copied, pasted, etc. This means they should be encodable as text that is short and does not include ambiguous characters. Bonus points for error detection like with credit cards.
Some of the these properties are in conflict. Statelessness is achieved by randomly generating long IDs, but people don’t like reading or typing long IDs.
Different use cases will need these properties in varying amounts. If you don’t intend to expose the IDs to users, (4) doesn’t matter. Just use long, randomly generated byte strings prepended with the date. Most databases have a UUID type that fits the bill.
If users are going to be working with IDs, that’s more complicated. If not every document has a user-facing ID, just go with the non-user-facing ID like before, and generate a shorter, random, stateful ID as needed.
I don’t think NanoID prepends the date, which means it won’t be efficient when inserting large numbers of IDs into a large index. They also default to using ambiguous characters like 1 and I and l. Also no error code. But they are shorter than UUIDs. So it doesn’t meet property (2), and it only kind of meets property (4). NanoIDs are random, so you’re probably safe from enumeration attacks (1). NanoIDs mostly leave statelessness as a decision for the user. They have a nice tool that helps estimate how long the IDs should be (https://zelark.github.io/nano-id-cc/ https://zelark.github.io/nano-id-cc/) for a given collision resistance.
I think we can do better overall. Bitcoin uses a good encoding scheme called base58check (https://en.bitcoin.it/wiki/Base58Check_encoding https://en.bitcoin.it/wiki/Base58Check_encoding). It generates fairly short strings and uses a checksum at the end. I think it could be refined for non-bitcoin purposes, but it’s already pretty good.
A 128-bit value like the ASCII string “hackernewstestid” is encoded as “Dtajqjz5pptWcmGrNcwBx7”. It’s about 2/3 the size of the equivalent UUID, even with the (unnecessarily long for this use case) checksum. It also has no punctuation.
I’d like to see a small ID standard that meets the above requirements and has a choice for either stateless and long or stateful and short. Maybe another choice for secure random or insecure. But all options would have binary form and a text form. The text form would use something similar to base58check, but probably with a smaller (or user-determined) length for the checksum.
- throwaway2016a 4y agoIt took me a bit to realize that what you're referring to as "error code" in many other contexts is "checksum"... this may just be the domain I work in but for me, "error code" usually means an "id that signifies a specific error happened" Another thought is that UUID V4 (which is the most common UUID implementation I've seen in the wild) also does not have a checksum. Though I think in general a checksum is probably a good idea if you want to help prevent people from mistyping the code or an optical reader from making an error.
- vlmutolo 4y agoGood point. “Checksum” is probably the more common word here. I updated my comment.
- irq-1 4y agoI want a set of symbols which are commonly used in most native languages and easily accessible from the keyboard. Numbers and math/common symbols should be mostly universal. @ from email. Plus, minus, multiply. Less and greater and HTML brackets. Hex without the english characters, and works in URLs. 0123456789@+-*<>
- keredson 4y agolooks like there's an impl already: https://github.com/cbschuld/uuid-base58 https://github.com/cbschuld/uuid-base58
- vlmutolo 4y agoThat’s pretty cool. I still think it would be nice to have a checksum at the end. That way, applications could return nicer errors to the user than 404.
- Rafert 4y agoULID hits most of these, and can be converted to UUID for use with databases supporting this datatype (not a strong column): https://github.com/ulid/spec https://github.com/ulid/spec