4 ms·

I struggle to think of use cases where involving encryption in creating, parsing, evaluating, or otherwise using primary keys is anything but a mistake. But I'm

I struggle to think of use cases where involving encryption in creating, parsing, evaluating, or otherwise using primary keys is anything but a mistake. But I'm sure I'm insufficiently creative.

chrismorgan 4y ago
Encrypting sequential identifiers is an excellent way of avoiding leaking the ID sequence, which is regularly at least mildly sensitive data, generally more sensitive than you expect, and preventing ID enumeration (either completely, if the ID space is sparsely occupied, or at least sequentially if it’s fullish).
- preseinger 4y ago
  In what circumstances does the value of that encryption outweigh its costs? Particularly in high-RPS (e.g. 1M+ RPS) applications? My intuition is that UUIDs, ULIDs, etc. would be orders of magnitude faster to {generate, parse} than anything involving an {encryption, decryption} step; am I off base?
  - wongarsu 4y ago
    Internally you can work completely with sequential int64 IDs, and at the API edge you have to spend ~30 clock cycles (about the cost of an L3 cache hit) on each id to encrypt/decrypt it. For any workload that can benefit from better data locality from sequential IDs this could well be a performance win over UUIDs, especially at scale. Of course ULIDs offer something very similar. But they leak more information, which makes them less desirable for some use cases.
    - preseinger 4y ago
      The benefit here, I guess, that you get the advantages of sequential IDs internally? I can't see how this could possibly outweigh the costs of user-visible IDs being different than internal IDs, and transformable only thru a {de,en}cryption process -- requiring all the standard maintenance, key rotation, etc.
  - chrismorgan 4y ago
    Encryption is much, much faster than you think. In my carefully-optimised Rust implementation of my scheme (see another comment in this thread) that uses a variant of Speck128 for encryption, and simple base-conversion into a 20-character array (no allocations!), the base conversion takes takes 100× as long as the encryption, which, in large batches, takes two cycles. Now AES (the topic here) is slower than Speck (which is one of the very fastest “real encryption” schemes out there), so the base-conversion wouldn’t be as much slowerer (certainly nowhere near 100×), but it still goes to show something. I would not expect you to be able to even measure the difference on general workloads. On massive batch workloads, I think there’s a good chance that the encryption approach with my scheme at least (AES would be a little more iffy) and a 64-bit ID would be just a teensy bit faster than a UUID primary key because of dealing with less data in the database, network and app before the encryption is done—in fact, assuming equivalent stringification I have decided to confidently assert my scheme would be faster, since the only thing that can slow it down is the added encryption code and round keys, which at scale are negligible, and my scheme will save around 60 cycles on stringification due to needing only 20 characters rather than 26. (And you could reduce it another six characters and sixty cycles if you limited it further to 70-bit IDs.) But even there I think you’d be straining to measure it.
    - preseinger 4y ago
      And how do you manage maintenance like key rotation, etc?
svnpenn 4y ago
came here for this comment. Why in the hell are people combining encryption with UUID? If you are putting any encryption with a primary key, you have failed in some other aspect of your security design. This whole article just screams red flag to me.

I struggle to think of use cases where involving encryption in creating, parsing, evaluating, or otherwise using primary keys is anything but a mistake. But I'm sure I'm insufficiently creative.

chrismorgan 4y ago

Encrypting sequential identifiers is an excellent way of avoiding leaking the ID sequence, which is regularly at least mildly sensitive data, generally more sensitive than you expect, and preventing ID enumeration (either completely, if the ID space is sparsely occupied, or at least sequentially if it’s fullish).

preseinger 4y ago
In what circumstances does the value of that encryption outweigh its costs? Particularly in high-RPS (e.g. 1M+ RPS) applications? My intuition is that UUIDs, ULIDs, etc. would be orders of magnitude faster to {generate, parse} than anything involving an {encryption, decryption} step; am I off base?
- wongarsu 4y ago
  Internally you can work completely with sequential int64 IDs, and at the API edge you have to spend ~30 clock cycles (about the cost of an L3 cache hit) on each id to encrypt/decrypt it. For any workload that can benefit from better data locality from sequential IDs this could well be a performance win over UUIDs, especially at scale. Of course ULIDs offer something very similar. But they leak more information, which makes them less desirable for some use cases.
  - preseinger 4y ago
    The benefit here, I guess, that you get the advantages of sequential IDs internally? I can't see how this could possibly outweigh the costs of user-visible IDs being different than internal IDs, and transformable only thru a {de,en}cryption process -- requiring all the standard maintenance, key rotation, etc.
- chrismorgan 4y ago
  Encryption is much, much faster than you think. In my carefully-optimised Rust implementation of my scheme (see another comment in this thread) that uses a variant of Speck128 for encryption, and simple base-conversion into a 20-character array (no allocations!), the base conversion takes takes 100× as long as the encryption, which, in large batches, takes two cycles. Now AES (the topic here) is slower than Speck (which is one of the very fastest “real encryption” schemes out there), so the base-conversion wouldn’t be as much slowerer (certainly nowhere near 100×), but it still goes to show something. I would not expect you to be able to even measure the difference on general workloads. On massive batch workloads, I think there’s a good chance that the encryption approach with my scheme at least (AES would be a little more iffy) and a 64-bit ID would be just a teensy bit faster than a UUID primary key because of dealing with less data in the database, network and app before the encryption is done—in fact, assuming equivalent stringification I have decided to confidently assert my scheme would be faster, since the only thing that can slow it down is the added encryption code and round keys, which at scale are negligible, and my scheme will save around 60 cycles on stringification due to needing only 20 characters rather than 26. (And you could reduce it another six characters and sixty cycles if you limited it further to 70-bit IDs.) But even there I think you’d be straining to measure it.
  - preseinger 4y ago
    And how do you manage maintenance like key rotation, etc?

svnpenn 4y ago

came here for this comment. Why in the hell are people combining encryption with UUID? If you are putting any encryption with a primary key, you have failed in some other aspect of your security design. This whole article just screams red flag to me.