9 ms·
I've become a fan of unique, relatively short and "human-readable" IDs, such at the ones used by Stripe, e.g. `cus_MJA953cFzEuO1z` for an ID of a customer. Here
by strnisa 2y ago
I've become a fan of unique, relatively short and "human-readable" IDs, such at the ones used by Stripe, e.g. `cus_MJA953cFzEuO1z` for an ID of a customer. Here's a Stripe dev article on the topic: https://dev.to/stripe/designing-apis-for-humans-object-ids-3o5a https://dev.to/stripe/designing-apis-for-humans-object-ids-3...
If you use JavaScript/TypeScript, you can make them like this:
function makeSlug(length: number): string {
const validChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
const randomBytes = crypto.randomBytes(length);
let result = "";
for (let i = 0; i < length; i++) {
result += validChars[randomBytes[i] % validChars.length];
}
return result;
}
function makeId(tableName: string): string {
const idSpec = TABLE_NAME_TO_ID_SPEC[tableName];
const prefix = idSpec.prefix;
const slugLength = idSpec.length - prefix.length - 1;
return `${prefix}_${makeSlug(slugLength)}`;
}
- metafunctor 2y agoType safe and unguessable IDs are what I've been using in my projects for the past, oh, 10 years maybe? Inspired by Stripe! In my databases, I often prefer integer primary keys for performance reasons. On the other hand, I don't want to expose my primary keys because they are easy to guess. Recently I've been playing with Rust, and ended up publishing a library to encrypt IDs in they way I like: https://crates.io/crates/cryptid-rs https://crates.io/crates/cryptid-rs
- vanderZwan 2y agoNice! For the record: the valid chars string is 62 characters, so naively using a modulo on a random byte will technically introduce a bias (since dividing 256 values by 62 leaves a remainder). I don't expect it to really matter here, but since you're putting in the effort of using crypto.randomBytes I figured you might appreciate the nitpick ;). Melissa E. O'Neill has a nice article explaining what the problem is, and includes a large number of ways to remove the bias as well: https://www.pcg-random.org/posts/bounded-rands.html https://www.pcg-random.org/posts/bounded-rands.html (in this case adding two more characters to the validChars string would be the easiest and most efficient fix, but I'm not sure if that is a possibility here)
- naasking 2y ago> For the record: the valid chars string is 62 characters, so naively using a modulo on a random byte will technically introduce a bias Indeed, there's no reason you couldn't just add "_" and "-" or "." as well to complete the set. Your identifier will still be URL-safe. I've been using this type of encoding for years [1] for these kinds of ids to use in URLs, and encoding/decoding is super-fast with some bit shifts. And unlike Base64, you don't need padding characters. [1] https://sourceforge.net/p/sasa/code/ci/default/tree/Sasa.Web/Url64.cs https://sourceforge.net/p/sasa/code/ci/default/tree/Sasa.Web...
- notfed 2y ago- and _ tend to break text selection.
- naasking 2y agoI'm not sure what you mean by "break". If you mean that touching or double-clicking on a block of text only extends up to the nearest symbols, that's true. But if your text selection UX is not terrible then it should be simple to extend that further. That said, iOS and Android text selection have gotten worse recently, IMO.
- Too 2y agoUsually one is not in control over every place where text can be selected. As a developer I will be exposed to ids being displayed in code, terminals, browsers of various sorts, database editors, json dumps, text editors, api responses, chat messages, you name it.
- naasking 2y agoSure, I agree. I'm not sure what point you're trying to make though.
- 2y ago
- whyever 2y agoPlease don't use % to generate integers from a range, it's not uniform, which can be disastrous if you rely on your numbers not being predictable. You can use crypto.randomInt instead.
- CodesInChaos 2y agoThat is definitely an improvement, but I find that concern a bit exaggerated. If all you need is unpredictability, a minor bias is sub-optimal but not disastrous. The 256 % 62 bias used here should reduce the min-entropy per character by 5%. And you can easily minimize the bias by using larger integer than a single 8-bit byte. There few algorithms where minor biases cause a disaster, like DSA.
- vincentdm 2y agoI don't like the prefix idea: besides the duplication of information, it also becomes a liability if you ever rename things. Imagine you prefix all customer IDs with `cus_`, but at some point decide to rename Customer to Organization in your codebase (e.g. because it turns out some of the entities you are storing are not actually customers). Now you have some legacy prefix that cannot be changed, is permanently out of sync with the code and will confuse every new developer.
- codeulike 2y agoI wouldn't worry about that - I still think its worth it. I've had systems which during development we thought the Contact page was going to be called 'Contact' in the UI but at the end it got re-labelled to 'Individual' but in all the code it was still called Contact and the IDs all started with a C - but you know what? It was still useful to look at an ID, see the C and know that it was an Individual.
- shawabawa3 2y ago> Now you have some legacy prefix that cannot be changed Yes you can. You can support the old cus_<ID> prefix as well as the new org_<ID> prefix, but always return org_<ID> from now on
- wongarsu 2y agoReddit prefixes their IDs with t1_ for comments, t2_ for accounts, etc. That sidesteps the renaming issue. Though I believe they mostly do it because their IDs are sequential, so without prefix you wouldn't easily notice if you use the wrong kind of id. They also only apply prefixes at the api boundary when they base36 encode the IDs, the database stores integers
- chuckadams 2y agoI have that exact issue with a couple different identifiers, and it's not a big deal. Usually it goes along with some data model change you already have to write compatibility code for, the new and old names tend to be related, and the old name tends to stick around other parts of the code anyway. Opaque IDs don't reduce the confusion there, documentation in appropriate places does.
- codeulike 2y agoExcept the Stripe ones are case sensitive which can be annoying with some databases
- datavirtue 2y agoYou know what's annoying? People storing GUIDs in a case-insensitive database.
- codeulike 2y agoClassic RFC9562 GUIDs were often represented as hex like {550e8400-e29b-41d4-a716-446655440000} and so were case insensitive
- garblegarble 2y agoThis approach will randomly generate profanity. If the ID is visible to users it can cause some to get upset (and best-case simply looks unprofessional). On a purely technical level if visible in URLs it can cause links to be blocked/altered by e-mail filters / filtering proxies. It's generally a good idea to drop vowels for this reason.
- account42 2y agofvck cvnt b1tch
- Knufferlbert 2y agoWe once had a rather angry Irish customer calling our support complaining that we called him a pikey (slur for gypsy). After some back and forth it turns out we just gave him an apikey. We never had a similar issue with our random numbers/letters/reset passwords or anything like that which don't have any kind of "dont return profanity" protections. Though I agree, someone getting a randomly generated customer portal url or something containing fuck or similar would look bad. Our cloudfront or something (or was it main public facing s3 bucket? can't remember) starts with "gay" and was never picked up on.
- biztos 2y agoTotally off topic, but "gypsy" is itself a slur for Romani people in much of the world. https://en.wikipedia.org/wiki/Romani_people https://en.wikipedia.org/wiki/Romani_people
- isaacremuant 2y agoI'd say those articles show Wikipedia's political bias and a tendency to overly politically correct instead of portraying reality. Many in real life do refer to themselves as gitano or Gypsy and would ask for others to refer to them as such. Of course it's very easy to find an article saying otherwise and then using that as the end of discussion for Wikipedia editors.
- theophaniel 2y agoI feel like https://github.com/jetify-com/typeid https://github.com/jetify-com/typeid is the solution to this
- mettamage 2y agoWhy is such a thing called a slug?
- inopinatus 2y agoIt's an old typesetting term that found its way into content management systems. https://archive.nytimes.com/www.nytimes.com/times-insider/2014/11/24/whats-in-a-slug/ https://archive.nytimes.com/www.nytimes.com/times-insider/20...
- corobo 2y agoOoh I had this wonder a while back and jotted it down just in case anyone else ever wondered about it: Comes from the ye olde paper-based blogs they call newspapers. When an article is being put together it’s given a short name, sort of like a project name. This name would remain the same throughout the article’s life - from reporter through to editor - it left its trail through the process. Like a slug.
- meindnoch 2y agoThat's not what Wikipedia says about its etymology though. >The origin of the term slug derives from the days of hot-metal printing, when printers set type by hand in a small form called a stick. Later huge Linotype machines turned molten lead into casts of letters, lines, sentences and paragraphs. A line of lead in both eras was known as a slug.
- corobo 2y agoOoh interesting. It looks like I have either misinterpreted or found a source that misinterpreted (it was a few years back, unsure if I came to the conclusion or found it). I'll have to update my notes, cheers! Apologies for the wetbrain hallucination, HN!
- 8372049 2y agoI'm not sure you're hallucinating. The dictionary I checked lists the printing and journalism terms separately. It's quite possible they have diverging etymologies, meaning both can be correct: 5. Print. a. a thick strip of type metal less than type-high. b. such a strip containing a type-high number or other character for temporary use. c. a line of type in one piece, as produced by a Linotype. 8. Journalism. a. a short phrase or title used to indicate the story content of a piece of copy. b. the line of type carrying this information.
- bambax 2y agoNice, but better to remove ambiguous / hard to read letters: ijlo, IJLO, and 01 (and maybe 7 as well?)
- inopinatus 2y agoI'm not convinced that all Stripe IDs are wholly random strings. I just decoded the base62 part of four genuine "acct_" objects, which at 16 characters are just shy of representing a 12 byte value (log2 62^16 =~ 95.3), and they all have a leading byte of "00000011" and two of them even have a leading 32 bits that is very suspiciously close to an epoch timestamp of a couple of years ago. There is a similarly suspicious pattern in some of their longer identifiers, invoices for example at 24 characters (ca.143 bits) all seem to have a first byte of "00000010". Even in the article you've linked to, look closely at the IDs: pi_3LKQhvGUcADgqoEM3bh6pslE pm_1LaXpKGUcADgqoEMl0Cx0Ygg cus_1KrJdMGUcADgqoEM card_1LaRQ7GUcADgqoEMV11wEUxU Notice the consistently leading low-integer values (a 3, then three 1s)? and how it's almost always followed by a K or an L? That isn't random. In typical base62 encoding of an octet string, that means the first five or six bits are zero and the next few bits have integer adjacency as well. It also looks like part of the customer ID (substring here, "GUcADgqoEM", which is close to a 64-bit value) is embedded inside all of the other IDs and then followed by 8 base62 characters, which might correspond to 48 bits of actual randomness (this is still plenty, of course). Based on these values it seems there's a metadata preamble in the upper bits of the supposedly "random" value, and it's quite possible that some have an embedded timestamp, possibly timeshifted, and a customer reference, as well as a random part, and who knows maybe there's a check digit as well. It's possible - albeit this is not analytical but more of a guess - that the customer ID includes an epoch-ish timestamp followed by randomness or (worst case, left field) is actually a sequence ID that's been encrypted with a 64-bit block cipher and 32 bits of timestamp as the salt, or something similar (pro tip: don't try that at home). My view is that either Stripe's engineering blog is being disingenuous with the truth of their ID format, or they're using a really broken random value generator. If the latter, I hope it's only in scope of their test/example data.
- dividuum 2y agoI took a look at a bunch of stripe customer ids I have stored and at least mine look very random on first glance. I assume their blog post uses demo keys or something similar.
- wslwsl 2y agoIn the comments it's mentioned, that the IDs contain a shard key for faster lookups. https://dev.to/stripe/designing-apis-for-humans-object-ids-3o5a#comment-2823m https://dev.to/stripe/designing-apis-for-humans-object-ids-3...
- strnisa 2y agoThanks for all the improvement suggestions! Taking them into account, the `makeSlug` function becomes: function makeSlug(length: number): string { const alphabet = "0123456789abcdefghjkmnpqrstvwxyz"; let result = ""; for (let i = 0; i < length; i++) { result += alphabet[crypto.randomInt(alphabet.length)]; } return result; }
- biztos 2y agoI think that 0 and 1 are likely to cause problems when customers end up reading their "user ID" back to your employees in Customer Support Country over the phone. "It's one-three-oh-dee-ee-el. Yes, I'm sure, EL as in elephant."
- jagged-chisel 2y agoOne can convert those to the correct characters. Since “oh” and “el” are excluded from the alphabetic range, they become “zero” and “one” - deciding whether that is done in software for the help desk, or in the brain of the help desk staff is left as an exercise to management.
- smaudet 2y agoI wonder if Unicode could be used to alter the characters such that these mistakes would be less possible, e.g. using ⓪.
- fhars 2y agoThat is "AT", isn't it? (No, it isn't, if you look closely enough.)
- smaudet 2y agoYeah...what I'd really like to do would be to give a character a "natural" background color, e.g. Then its simple for support to say "red is one, green is ell". But you can't just add a color to a character, because copy paste/rich formatting don't work everywhere, or even transfer well... Alternatively, if you use ⓪ and ⒈ it matters less if the user says "at" or "zero", and more that they didn't say "oh" or "one".
- scrollaway 2y agoI don’t recommend using random bytes for this. What I have done in my previous projects is take a uuid6, reserve a couple of bytes inside of it to replace with an object ID, and I convert that to obj_XXXXXXX. This means you can store them in the db not as a string but as a uuid, which is a lot more performant. You also get time stamping for free.
- cogman10 2y agoA problem with this approach is it's not monotonical. Especially if you want to use this thing as an index in a database, you'll run into problems where you try doing middle insertions frequently, which causes fragmentation. The solution to this problem is making the higher order characters time sorted [1]. You don't need to go all out like uuid, you can have a pretty low resolution. It's more important that new insertions tend to be on the same page. If you have a low frequency insertions then minute resolution is probably good enough. (Minutes since 2000 is an easy calculation). To implement that here, I'd suggest looking at how base64 or 85 encoders work and use that instead of repeated mods. You can then dedicate the upper bits to a time component and the lower bits can remain random. [2] [1] https://vladmihalcea.com/uuid-database-primary-key/ https://vladmihalcea.com/uuid-database-primary-key/ [2] https://github.com/mklemm/base-n-codec-java/blob/master/src/main/java/net/codesup/utilities/basen/AnyBaseEncoder.java https://github.com/mklemm/base-n-codec-java/blob/master/src/...
- reissbaker 2y agoIMO it's nice to have two keys: 1. An auto-incremented 64-bit (unless you have a good reason, in which case 32-bit is fine) primary key, used internally for foreign key relations. This will generally result in less index bloat on associated tables, and fast initial inserts. 2. A public-facing random string ID. Don't use this internally (other than in an index on the table it's defined for), since it's large. But this should be the only key you expose to end-users, to prevent leaking data via the German Tank Problem: https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem Only create the second key if this is data you're exposing to users, of course — for data that's only used internally, just use the 64-bit auto-incremented PK and skip the added index bloat entirely.
- cogman10 2y agoFor the number 2, I think one issue is that you are going to be semi-frequently whacking the db to do a mapping of that random string id back to the real id. OK for smaller entities but might be a pain if there's a lot of those ids to wrangle. You can throw a secondary index on it, but that will still have some minor fragmentation issues. One benefit of a random id is if you are working with more complex data models it can make creating those easier/faster. Instead of having a centralized location to get new ids from (the DB) you can create ids on the fly from the application which can turn the write into a single action from the application rather than a dance of inserting the main table, getting the new id, then inserting to the normalized tables.
- giancarlostoro 2y agoIf I'm going to do that I think I'd use Bitcoins BASE58 which avoids letters that could be confused for each other. The number of times I see an O and 0 and wonder which is which, because the font does not make it clear really annoys me. Edit: Other honorable mentions: ObjectID's as used by MongoDB which contain the creation timestamp. Also Discord's snowflakes (inspired by Twitter's iirc), which also contain the creation timestamp.
- kibwen 2y agoIf you want random IDs to be human-readable (and human-communicatable), I'd just recommend base 32 or even base16. You don't actually save that many bytes from base58 or base64 when it comes to short IDs. Case in point, the parent poster's base64 ID is 14 characters long. When encoded as base32 that's still only 17 characters (or 19 in base16), and now you have completely gotten rid of all notion of casing, which is annoying to communicate verbally.
- giancarlostoro 2y agoI think the more I think of it, the more I favor the Discord snowflake ID's because they're just integers, and they can be generated on the fly. I think new messages generated them on the client if I'm not mistaken.
- cimnine 2y agoYes, and I like to combine two established concepts instead of rolling my own: URI and UUIDv7. So my IDs become `uri:customer_shortname:product_or_project_name:entity_type:uuid`. An example ID could be `uri:cust:super_duper_erp:invoice:018fe87b-b1fc-7b6f-a09c-74b9ef7f4196`. It's even possible to cascade such IDs, for example: `uri:cust:super_duper_erp:invoice:018fe87b-b1fc-7b6f-a09c-74b9ef7f4196:line_item:018fe882-43b2-77bb-8050-a1139303bb65`. It's immediately clear, when I see an ID in a log somewhere or when a customer sends me an ID to debug something, to which customer, system and entity such an ID belongs. UUIDv7 is monotonic, so it's nice for the database. Those IDs are not as 'human-readable' for the average Joe, but for me as an engineer it's a bliss. Often I also encode ID's I retrieve from external systems this way: `uri:3rd_party_vendor:system_name:entity_type:external_id` (e.g. `uri:ycombinator:hackernews:item:40580549:comment:40582365` might refer to this comment).
- phkahler 2y agoSomething like that should have a built in check "digit" if people are going to see it and possibly type it in. For numeric values, making them all a multiple of 11 is a simple way to catch all single digit errors or single transpositions.
- chuckadams 2y agoThat's why VINs make a decent natural key, because they do have a check digit. Plus they're not opaque: if you look up the VIN and the make/model/year is completely different than the car in front of you, you know you either have the wrong VIN or the wrong car.
- psychoslave 2y agoTo my mind, it always felt so saddening that adoption of a truly straightforwardly readable notation for numbers never took of. I mean it’s so easy to do. You can start for example with a single syllable per digit, and for example only target CV syllables. From this there is many possibilities, but for example, let’s consider only a base ten. Starting with vowels order o, i, e, a, u with mnemonic o, i graphically close to 0, 1 and then cyclically continue the reverse order in alphabet (<-a, <-e, <-i, <-o*, |-u). We now only need two consonants for the two series of 5 cardinals in our base ten, let’s say k and n. So in a quick and dirty ruby implementation that could be something like: $digits = %w{k n}.product(%w{o i e a u}).map{it.join('')} def euphonize(number) = number.to_s.split('').map{$digits[it.to_i]}.join('-') euphonize(1234567890) # => "ki-ke-ka-ku-no-ni-ne-na-nu-ko" That’s just one simple example of course, there are plenty of other options in the same vein. It’s easy to create "syllabo-digit" sets for larger bases just adding more consonants, go with some CVC or even up to C₀C₁VC₀C₁ if sets for C₀ and C₁ are carefully picked.
- psychoslave 2y agoAnd of course a naive implementation of the reverse is also trivial: def numerize(euphonism) = euphonism.split(?-).map{$digits.find_index(it)}.map{it.to_s}.join.to_i
- jmcphers 2y agoThere is a old system for making numbers pronounceable as words using a mapping from each number to a consonant value. It's typically used to help memorize numbers: https://en.wikipedia.org/wiki/Mnemonic_major_system https://en.wikipedia.org/wiki/Mnemonic_major_system
- samatman 2y agoThis is how the Urbit address scheme works, btw: https://urbit.org/blog/the-urbit-address-space https://urbit.org/blog/the-urbit-address-space
- ssahoo 2y agoUUIDv7's premise to solve this, isn't it? it's 32 chars without dashes.
- Merad 2y agoI'd recommend using Crockford base 32 [0] to encode the bytes. It makes the text more human friendly by eliminating case sensitivity, removing similar/ambiguous letters, and preventing accidental profanity. And in most cases I think you're also better off just using a uuid and encoding its bytes as base 32, in which case you're basically doing type id's [1]. If the "slug" portion of the id actually encodes a uuid, then it gives you the option to store it in your database using an appropriate uuid type. This will make your database much happier than using long string PK's. 0: https://www.crockford.com/base32.html https://www.crockford.com/base32.html 1: https://github.com/jetify-com/typeid https://github.com/jetify-com/typeid
- longerd2 2y agoHow is this readable?