5 ms·

> The key hadn't changed in months - at least 6 months by the time I found it. I was told this same key would be used to encrypt web session data in a cookie. T

> The key hadn't changed in months - at least 6 months by the time I found it. I was told this same key would be used to encrypt web session data in a cookie. There were more than a dozen people who I knew had access to the key, and another 5 had come and gone (and had had access to the same key) in the previous 6 months. What's so wrong with any of this? Software requires operators and developers. If you can't trust them, you can't trust your service, period. It's normal for operators and developers to occasionally have access to sensitive data (e.g. when your service crashes, somebody has to look at the crash dump). Such access should be restricted (requiring approval) and logged, of course, but it's difficult to eliminate entirely at scale. It's good to rotate keys on a regular basis - but an annual key rotation doesn't seem negligent to me for a key used to encrypt session data, which is necessarily long-lived.

bdhess 8y ago
> Such access should be restricted (requiring approval) and logged, of course, but it's difficult to eliminate entirely at scale. It’s not clear how you could practically enforce this requirement if devs just have the raw key on their workstations.
- hanniabu 8y ago
  Would be nice to use a multi-sig so the dev would need their key which they always have access to plus a key from an approver.
antonvs 8y ago
> What's so wrong with any of this? A very basic principle of security is the principle of least authority. To implement that, you don't use one key for many different purposes over a long period of time and give it to anyone who needs it. You use different keys for different purposes, and replace the keys periodically to help ensure that they're only available to a limited group. Another way to answer your question is "because that's how all these breaches happen."
- charlesdm 8y ago
  The short (and sad) answer is that no one really cares about security; they only care about it when something hits the fan. Security isn't a revenue driver, only a cost. It's commonly known that most companies (small or large) vastly prefer investing in things that increase revenue and profit.
- lowercased 8y ago
  There was no ability to tell who'd even seen the key. A colleague of mine 'got it', and when it was shown he had it, there was a "WTF? You're not supposed to have that? Where'd you get it?". It was checked in to version control. It was just in a project that only a few people were assigned to, but there was no restriction on who could view those projects.
- figers 8y ago
  I understand no one having access to the key, but if data is already encrypted using this key and you want to rotate to a new key, how do you decrypt the old data? re-encrypt all old data with the new key?
PeterisP 8y ago
For keys that matter, good practice is that no one has direct access to the key. E.g. you have a process where there are ways to sign stuff with that key, and there are ways to gain access to that key if extraordinary circumstances arise, but in normal operation you should be able to make sure that you can revoke the ability to sign stuff from anyone, which requires you to be certain that they never ever had a chance to see, copy or write down private key material.
- arethuza 8y ago
  Isn't that the point of HSMs where nobody can get access to retrieve the private key - all you can do is sign stuff with the key?
  - PeterisP 8y ago
    Yup, but even in cases where the expense of a proper HSM isn't warranted, you can set up a software solution/process that works in a similar manner to a HSM, just with less tamper-resistance.
    - arethuza 8y ago
      Of course even if you have HSMs there is still the issue of controlling access to the HSMs!
lowercased 8y ago
What was being encrypted was a sequential DB ID that identified your user ID in the system. Decrypting the data (with the key that multiple people had access to and there was no way of telling who'd taken it) would allow you to change your ID and re-encrypt and become someone else. (also, AFAICT, the same key was used for some internal data encryption too - same key used in multiple areas across different teams) This was initially just a marketing site, but I was being told to use the same system for authentication around a financial service, and I objected. I suggested the ID be random, then mapped to something else. I had multiple 'senior' developers try to explain why "random isn't really random" and had a lot of timing attacks associated with it. I kept pushing back that timing attacks weren't as big a problem as "someone having the key can impersonate anyone, and we can't prove that people haven't taken the key out of the company". I "only" had 9 years experience at this point, but was the 'new guy' in the company, having only been there a few months, and apparently hadn't "paid my dues" in their ecosystem for long enough to be taken seriously. It took weeks of me objecting before we had a big meeting, and their architecture was mapped out on a board, and defended. I then asked "how can you prove the key hasn't left the building with previous people?" Then my manager sort of "got it" and realized this was all kinda ... not something we should rely on for guarding financial info. Maddening that it took weeks of "deliberation" on this issue when it was patently insecure. I've not ever whiteboarded it out to anyone with more than a few years of experience who didn't stop me partway through to tell me that it was insecure. Dejected through all of this, I asked to be put on another project, but I was seen as a troublemaker at that point, then left.
closeparen 8y ago
There are mature and well known techniques for mitigating insider threats. Key rotation capability is table stakes. Keys that are too expensive to rotate need to be in HSMs, and probably protected by operator quorum. There may be some short-lived, limited purpose keys that are more vulnerable, but the whole company’s master keys to everything should absolutely not be held by any one person.

bdhess 8y ago

> Such access should be restricted (requiring approval) and logged, of course, but it's difficult to eliminate entirely at scale. It’s not clear how you could practically enforce this requirement if devs just have the raw key on their workstations.

hanniabu 8y ago
Would be nice to use a multi-sig so the dev would need their key which they always have access to plus a key from an approver.

antonvs 8y ago

> What's so wrong with any of this? A very basic principle of security is the principle of least authority. To implement that, you don't use one key for many different purposes over a long period of time and give it to anyone who needs it. You use different keys for different purposes, and replace the keys periodically to help ensure that they're only available to a limited group. Another way to answer your question is "because that's how all these breaches happen."

charlesdm 8y ago
The short (and sad) answer is that no one really cares about security; they only care about it when something hits the fan. Security isn't a revenue driver, only a cost. It's commonly known that most companies (small or large) vastly prefer investing in things that increase revenue and profit.
lowercased 8y ago
There was no ability to tell who'd even seen the key. A colleague of mine 'got it', and when it was shown he had it, there was a "WTF? You're not supposed to have that? Where'd you get it?". It was checked in to version control. It was just in a project that only a few people were assigned to, but there was no restriction on who could view those projects.
figers 8y ago
I understand no one having access to the key, but if data is already encrypted using this key and you want to rotate to a new key, how do you decrypt the old data? re-encrypt all old data with the new key?

PeterisP 8y ago

For keys that matter, good practice is that no one has direct access to the key. E.g. you have a process where there are ways to sign stuff with that key, and there are ways to gain access to that key if extraordinary circumstances arise, but in normal operation you should be able to make sure that you can revoke the ability to sign stuff from anyone, which requires you to be certain that they never ever had a chance to see, copy or write down private key material.

arethuza 8y ago
Isn't that the point of HSMs where nobody can get access to retrieve the private key - all you can do is sign stuff with the key?
- PeterisP 8y ago
  Yup, but even in cases where the expense of a proper HSM isn't warranted, you can set up a software solution/process that works in a similar manner to a HSM, just with less tamper-resistance.
  - arethuza 8y ago
    Of course even if you have HSMs there is still the issue of controlling access to the HSMs!

lowercased 8y ago

What was being encrypted was a sequential DB ID that identified your user ID in the system. Decrypting the data (with the key that multiple people had access to and there was no way of telling who'd taken it) would allow you to change your ID and re-encrypt and become someone else. (also, AFAICT, the same key was used for some internal data encryption too - same key used in multiple areas across different teams) This was initially just a marketing site, but I was being told to use the same system for authentication around a financial service, and I objected. I suggested the ID be random, then mapped to something else. I had multiple 'senior' developers try to explain why "random isn't really random" and had a lot of timing attacks associated with it. I kept pushing back that timing attacks weren't as big a problem as "someone having the key can impersonate anyone, and we can't prove that people haven't taken the key out of the company". I "only" had 9 years experience at this point, but was the 'new guy' in the company, having only been there a few months, and apparently hadn't "paid my dues" in their ecosystem for long enough to be taken seriously. It took weeks of me objecting before we had a big meeting, and their architecture was mapped out on a board, and defended. I then asked "how can you prove the key hasn't left the building with previous people?" Then my manager sort of "got it" and realized this was all kinda ... not something we should rely on for guarding financial info. Maddening that it took weeks of "deliberation" on this issue when it was patently insecure. I've not ever whiteboarded it out to anyone with more than a few years of experience who didn't stop me partway through to tell me that it was insecure. Dejected through all of this, I asked to be put on another project, but I was seen as a troublemaker at that point, then left.

closeparen 8y ago

There are mature and well known techniques for mitigating insider threats. Key rotation capability is table stakes. Keys that are too expensive to rotate need to be in HSMs, and probably protected by operator quorum. There may be some short-lived, limited purpose keys that are more vulnerable, but the whole company’s master keys to everything should absolutely not be held by any one person.