9 ms·
Things you should never do: use integers as ID's. This is literally a solved problem, and the solution is UUIDs, which were invented for exactly this job.
by marcus_holmes 6y ago
Things you should never do: use integers as ID's. This is literally a solved problem, and the solution is UUIDs, which were invented for exactly this job.
- mherrmann 6y agoWhat's wrong with integer IDs?
- dkarp 6y agoThe main issue in my experience are so called "user enumeration attacks", especially with sequential ids that are normally used. This is where an attacker is able to leak information from your system just by guessing ids. If you used a sequential id then you can cycle from 1 to X and probably easily find which are valid users. You can then likely see how many valid users/ids there are and potentially pull their data if the authentication has been implemented incorrectly. By using a UUID, the key space is so large that you cannot reasonably guess user ids. So you can't use those same brute force techniques and it makes extricating data much harder.
- marcus_holmes 6y agothis, also there's no possibility of collision. You can't have two processes generate the same ID (which means you can generate your IDs in code and send them to the database, which is really useful in some situations). Also, and this is the main one for me - if I mess up my code and accidentally use a document_id instead of a user_id, I'll just get "not found" instead of someone else's data.
- mherrmann 6y agoActually, afaik uuids are pseudo-random and can thus absolutely collide in theory. Collision is solved by every db backend out there.
- derwiki 6y agoMany shops (I presume, since I work at non-FAANG) have their own version of UUID generation that includes info about the machine/instance it was generated from. I don’t think we’ve formally proved it but I think ours is guaranteed unique in our fleet.
- marcus_holmes 6y agoUUID collision is not possible in practice. You can absolutely code in the absolute knowledge that your application will not generate 2 identical UUID's ever. The odds are astronomical. You have bigger concerns. And yes, every DB backend has solved collision. But as I said, it's useful to be able to generate ID's in code sometimes. This is possible with UUID's and not possible with integers.
- marcosdumay 6y ago> UUID collision is not possible in practice. UUID is a loosely defined concept. There are many implementations. Some of them have no chance of collision at all, some have an astronomical chance, and some have very real odds that you'll receive a call at 3AM during the new year's celebration.
- deckard1 6y agojust make sure you have everything configured correctly on your system if using UUIDs: https://github.com/ramsey/uuid/issues/80 https://github.com/ramsey/uuid/issues/80 In the types of systems that need UUIDs there is probably no easy way to check for collisions. The prospect of mystery data corruption with no ability to trace it down frightens the hell out of me. The only reason that issue was reported is because someone was actually doing the collision checking. That's not going to be the norm in UUID systems. Think about it.
- vsareto 6y agoThis is really an ancillary benefit and not a real solution to enumeration vulnerabilities. The concern is usually over sharding the data, unique ID generation guarantees, and performance. You shouldn't really be deciding UUID vs. Int based on the possibility for ID enumeration. It's almost certainly easier to come up with a solution to slow/prevent enumeration than move from Ints to UUID.
- mewpmewp2 6y agoThere's also leaking financial, business data. E.g you create an order and you see your order id being 3000 the first day and 4000 the next, so then you'll be able to guess how many orders this company has etc.
- tjalfi 6y agoThis is called the German Tank Problem[0]. [0] https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem
- meepmorp 6y agoThe primary key serves a database function, and its data type should be optimized to suit that function. If you need an enumeration resistant id for public use, a uuid is a great idea, but it's a mistake to make the primary key in a database do double duty as a public identifier and impose inappropriate constraints on it as a consequence.
- closeparen 6y agoObscure user IDs could be a defense in depth measure but really you need to be authorizing the data you release against the authenticated session cookie. A view meant for the user's own consumption shouldn't take a user ID at all, just pull it from the session.
- dkarp 6y agointeger ids are still often used internally for database primary keys with UUIDs being the done thing for external interfaces. Personally I've never experienced the "whole class of bugs" that starting with a big integer is supposed to solve. I'm not using PHP so maybe that's why?
- blacktriangle 6y agoI've run into one, but it was pretty dumb. Multitenant application using the account ID as the first element in the URL on a Rails app. when it got to account 404, my address.com/404 went to the static 404 page rather than account 404. That user was pretty confused for awhile.
- corytheboyd 6y agoOuch, using the top level path for something dynamic like that is rough. Hope someone learned the right lesson from that :p
- edmundsauto 6y agoThis is discussed in the article - they suggest to start your INT ids at a really large number to avoid this kind of thing.
- Can_Not 6y agoI don't know rails, but that is just terrible design. address.com/accounts/404 is what the devs should have used, and "404" is a response code and an error page, not a URL you redirect/rewrite URLs to.
- chromatin 6y agoThe reason (other than "don't expose integer PK externally") people say you should use integer as PK and UUID as a secondary/external facing id is that conventional B-tree indexing of UUID is not as efficient as B-tree indexing of autoinc integers in most databases. However if you want any sort of efficient lookup on the external key (UUID), your database still needs an index on the UUID, and you are back at square one. I choose to forego the integer PK and just use UUID since I have to create index over it anyway.
- LordHeini 6y agoNo you should not. UUIDs are not always a good solution. If you order things by id (often useful for pagination, since incrementing ids are usually ordered by creation date) uuid are of no help. You would need to sort by created_at and this would require an additional index. No to speak of pagination by id ranges or whatever is used sometimes. UUIDs are mpossible to remember and thus somewhat cumbersome to use too. The weird id mixed up with index problem from the article is something which never happened to me ever in over 10 years of web development. I stay away from PHP as much as possible so that may be the reason ;)
- marcus_holmes 6y agoInteger ID's are always a bad thing. Yes, you should be sorting things by created_at. Having another index is not a bad thing. > The weird id mixed up with index problem from the article is something which never happened to me ever in over 10 years of web development. How do you know? It's a remarkably difficult bug to detect.
- urban_alien 6y ago> Integer ID's are always a bad thing. You are very wrong and wildly misunderstanding the problem that UUIDs solve
- jeremyjh 6y agoWhen what you know just isn’t so.
- mumblemumble 6y agoSo, for example, one previous database I worked on had an access pattern where we always wanted to get results sorted by the order in which they were inserted, and we needed this to be a completely stable ordering. Even two rows that get inserted during the same microsecond, as part of the same transaction, needed to have some official (even if arbitrary) designation of which one is first and which one is second. And both of those requirements also meant that the ordering key needed to be unique. And we also needed table access to be as fast as possible, and disk-friendly, and tend to offer the fastest access for the most recent data. All of this points to using serial numeric IDs as both a unique key and the clustering key. UUIDs do nothing to help with any of those requirements, and generally hurt all of them. They do that because of certain characteristics they have that are specifically there in order to solve one of the few problems we didn't have to worry about. Tangentially, in software systems design, I've long since realized that the word "always", when left to roam around freely, unchaperoned by any qualifiers to limit its universality, is an indicator of limited breath of experience. So, when you encounter it, it's useful to mentally insert "in my experience" as a stand-in qualifier. Having done so, the next conundrum is that "in my experience" advice is only actionable to the extent that you know what experience the advice giver has to draw on.
- k1ll3r 6y agoagreed, this is a terrible idea.
- edoceo 6y agoULID. 128bit, human "readable", sorted (like int!), nearly impossible to guess/collide and has timestamp embedded. I love this format.
- andrewem 6y agoHere’s a link to a description of ULID, for those like me who had never heard of it: https://github.com/ulid/spec https://github.com/ulid/spec
- reader_1000 6y agoHow does it compare to cuid [1]? If you know the differences, can you please explain them for those like me who do not know both? [1] https://github.com/ericelliott/cuid https://github.com/ericelliott/cuid
- kweinber 6y agoTell that to your database which is highly optimized for using integers as IDs. Using a UUID everywhere when you could be using an integer is an invitation to low-performance-city.
- T-hawk 6y agoThis sounds like a case of "premature optimization is the root of all evil." Do you have any performance numbers that show UUIDs being any noticeably slower? A UUID is only the size of two 64-bit integers anyway. And either identifier likely will be a tiny fraction of the data in a particular row. So I'd doubt this is any real performance problem in the vast majority of applications. UUIDs look big and scary in hex notation, but underneath it's a compact and fast binary format, just a 128-bit integer.
- cobaltoxide 6y agoUUIDs are (very large) integers.
- bob1029 6y agoIntegers are by far the best primary key material possible. Only for lack of discipline do they seem inferior to alternatives. See a prior book I wrote on HN regarding this: https://news.ycombinator.com/item?id=25309248 https://news.ycombinator.com/item?id=25309248 The current domain model I am working with utilizes a global integer sequence to key all entities. This implicitly eliminates the class of bugs where the same keys of different types overlap and would otherwise mask exceptions. It also enables powerful domain modeling techniques in which the identities of things are themselves to be thought of as first class entities and referred to as a common class of thing. This is a little mind-bending at first, but it enables some really powerful abstractions that would otherwise be infeasible if we had to switch over all possible types of keys. The benefits of an integer key vs a guid key are quite profound when you get into the academics of information theory. They provide implicit creation order of things, whereas GUIDs cannot. They are deterministic in that there will never be a collision. Their range can be made to be infinite. Integers are perfectly efficient, even if the computer representation isn't necessarily so - BigInteger types scale gracefully.
- baix777 6y agoThe creation of integers doesn't scale for large data volumes. Plus you need a place to create these integers, and a failover location, which adds complexity. Multiple machines can each be creating their own guids in a very simple manner. And the odds of a guid collision is extremely low, and for most applications is acceptable. Having worked with petabytes of data guid performance isn't really an issue as there are more important factors to worry about.
- bob1029 6y ago> The creation of integers doesn't scale for large data volumes. At which specific integer does the scaling start to slow down?
- closeparen 6y agoOne concurrent insertion, or one network partition. UUIDs can be generated on many machines with no awareness of each other and merged later.
- nostrademons 6y agoOne of the best pieces of programming advice I got at Google was consider IDs carefully. This is not a solved problem, and there isn't a one-sized-fits-all solution. Rather, there are a bunch of guide rules, and then you need to understand your problem domain very well to choose good identifiers. An ID should be: 1.) Actually an identifier, i.e. it needs to identify objects uniquely and must be immutable throughout the lifetime of the object. 2.) Assigned at object-creation time; you can't have an object without an ID (in persistent storage), lest you'll have no way to refer to that object later. This presents particular problems for a lot of workflows where you want to introduce the concept of a user early, before soliciting personal data. Think of storing browsing histories for guest users, or persisting shopping carts before checkout. 3.) Integral to the storage system. IDs will usually be the ways that you lookup and join objects; the performance characteristics of your database can influence the type of data you choose for an ID. 4.) Because of #2 & #3, oftentimes a significant concurrency bottleneck for object creation. This is the downfall of many auto-increment integer schemes. 5.) The foreign key for other objects. This has space implications that you often have to trade-off against latency implications. If you store useful information within the ID, you can use that info without needing to make a separate query or join against the DB. However, you need to carefully consider whether that'll run afoul with #1, and the more information you put in the ID, the bigger the size bloat for other objects that reference that ID. 6.) Oftentimes a security & PII risk. Because IDs are the primary means for lookup and are guaranteed to be unique, it's awfully tempting to put them in your public APIs (like HTTP URLs). But then anyone who has a URL has all the information included in the ID. If you use sequential IDs, they also have the ability to scan your entire database; this was the downfall of Parler. Integer keys do really well for #1, #2, #3, and #5, but are very problematic for #4 and #6. But then, there are fairly easy workarounds for those: #4 is often solved by hashing a unique natural key of the object (also solving #6), while #6 can be solved by never exposing internal IDs to the outside world and instead using a lookup table on some friendly URL scheme, which is better for UX anyway (at the cost of #3). GUIDs do well for #1, #2, #4, and #6, but perform worse for #3 and #5. Natural keys (where you make the identifier some combination of the actual data) are also frequently underrated. The most obvious use for these are relation tables, where each row just indicates that two entities have the relationship that the table describes; you wouldn't normally put an auto-increment ID on that. But think also of something like a search refinement: the most natural key for that is [query, language] => [list of suggestions], and that uniquely identifies each set of refinements, and it has other useful properties where the rest of your search engine doesn't need to know that refinements exist (it already has the query), and you don't need a lookup call from query to some search_refinement_id or vice versa, and you can easily enumerate the set of languages that a given query has data for, and you can follow a chain of refinements without any intermediate lookups. If you were to suggest either integers or UUIDs for this problem I'd say that you're overengineering. And if the problem domain changes such that the key-set changes (for example, you want to include past search context, or you want to personalize refinements to each user) then I would recommend you start a different system from scratch and eventually replace or merge in the current one, because those problems have sufficiently different requirements that your whole data pipelines are going to be different. (In particular, personalized search requires PII handling, a lot of care in logging, encryption of the data when at rest, knowledge of a lot more entities in the system, etc.)