3 ms·
I’ve been working on a robust scheme for encrypted sequential IDs, which is done, including library implementations in Rust, JavaScript and Python, pending just
by chrismorgan 4y ago
I’ve been working on a robust scheme for encrypted sequential IDs, which is done, including library implementations in Rust, JavaScript and Python, pending just a smidgeon more writing about it and reviewing a decision on naming. You store an integer in the database, then encrypt it with a real block cipher, and stringify with Base58. I have three modes: one for 32-bit IDs, using Speck32/64 and producing 4–6 character IDs; one for 64-bit IDs, using Speck64/128 and producing 8–11 character IDs; and one hybrid, using the 32-bit mode for IDs below 2³² and the 64-bit mode above that, providing both a forwards-compatibility measure and a way of producing short IDs as long as possible. Contact me (see my profile) if you’re interested, or I’ll probably publish it in another day or two. Trouble is that I’ve been getting distracted with other related concepts, like optimally-short encoding by using encryption domains [0, 58¹), [58¹, 58²), …, [58¹⁰, 2⁶⁴) (this is format-preserving encryption; the main reputable and practical choices I’ve found are Hasty Pudding, which I’ve just about finished implementing but would like test vectors for but they’re on a dead FTP site, and NIST’s FF1 and FF3, which are patent-encumbered), and ways of avoiding undesirable patterns (curse words and such) by skipping integers from the database’s ID sequence if they encode to what you don’t want, and check characters with the Damm algorithm. If I didn’t keep getting distracted with these things, I’d have published a couple of weeks ago.
(I am not aware of any open-source library embodying a scheme like what I propose—all that I’ve found have either reduced scope or badly broken encryption; https://github.com/yi-jiayu/presents https://github.com/yi-jiayu/presents encrypts soundly, but doesn’t stringify; Hashids is broken almost beyond belief and should not be considered encryption; Optimus uses an extremely weak encryption.)
UUIDs are crazy overkill in any situation where you can have centralised ID allocation. Fully decentralised? Sure, 128 bits of randomness or mixed clock and randomness or similar, knock yourself out. But got a master database? Nah, you’re just generating unreasonably long values that take up unnecessary space and make for messy URLs and such.
- mappu 4y agoI've done something similar to obfuscate private DB IDs in a large existing application - just ensure they're all Skip32-encoded in all query parameters with an app-wide secret. It works well but you have to be very disciplined to catch every case individually. Using GUID PKs from the start just removes this entire category of problem.
- chrismorgan 4y agoIf it requires discipline, you’re doing it wrong. For best results, it should be threaded through your ORM or equivalent so that you can’t do it wrong, that at your app layer you never get raw IDs.
- iroddis 4y ago> UUIDs are crazy overkill in any situation where you can have centralised ID allocation. Except that’s specifically the use case of UUIDs: to have a decentralized method to generate unique IDs with minimal chance of collisions. If you have centralized control, of course there will be options with more attractive properties: they aren’t dealing with the same constraints.
- chrismorgan 4y agoSure, decentralised allocation is the intent of UUIDs, but in practice UUIDv4 is used extremely widely where it’s completely unnecessary, because it’s a convenient way of generating non-sequential IDs with good tooling support (UUID libraries, UUID data types in some databases, &c.).
- yencabulator 4y agoI dabbled with doing that. XTEA is a 64-bit block cipher, which suits 64-bit IDs pretty well, and well you can't really ask for much more without using larger IDs. I use z-base-32 to stringify -- I considered base58, but I think case-sensitive URLs are not nice. Looks like user=1 doc=1 -> /users/a7e34gz71r4ig/documents/xs69f1c878rzq user=42 doc=13 -> /users/am8hng8rnoopg/documents/9othzs4tgujrw I have the code in Go (it's near-trivial), but ended up doing something else for that project. Also, I hate that Postgres doesn't actually have unsigned integers.