6 ms·

Probably a backup. It makes sense to have an offline backup in cleartext (for DR), as long as you have the appropriate storage and security controls in place to

Probably a backup. It makes sense to have an offline backup in cleartext (for DR), as long as you have the appropriate storage and security controls in place to protect it.

brohee 6y ago
It makes absolutely no sense. Such highly valuable secrets are usually saved using Shamir's secret sharing with parts of the split secret held by people unlikely to collude. Key ceremonies are done in a way that at no point a human being is in position to single-handedly extract the secret from its HSM. This is a huge failure.
- m4rtink 6y ago
  What surprises me is that you can extract the private key at all - I would expect it to be firmly inside a HSM, that only accepts signing requests and the key never leaves the module & the HSM wipes the key if it detects tampering (there is usually a battery inside to power the tampering detection even if the device is not plugged in). So just exporting the private key so easily without some pretty involved hight-tech HSM key extraction sounds insane.
  - closeparen 6y ago
    I worked on a new internal PKI, nothing as high stakes as banking, but we did have a security consultant who had been around the block. We did, of course, purchase such HSMs and design a signing ceremony. But in his opinion it was also normal and expected to keep a decryptable copy of the private key, in case of e.g. changing HSM vendors. It would be even less accessible, but it would exist.
    - m4rtink 6y ago
      That's a good point - I guess if you know what you are doing then I guess it's fine.
  - brohee 6y ago
    One article said they key was left in clear on a laptop, so maybe a HSM migration was involved.
- briffle 6y ago
  Interesting.. What Open source tools do you use for this? I would love to read further how I can not have a printed copy of a master key in a safety deposit box.
  - stefan_ 6y ago
    Here is an implementation of this: https://www.cloudflare.com/dns/dnssec/root-signing-ceremony/ https://www.cloudflare.com/dns/dnssec/root-signing-ceremony/ They have a page for each signing ceremony: https://www.iana.org/dnssec/ceremonies/41 https://www.iana.org/dnssec/ceremonies/41 With a script of everything done: https://data.iana.org/ksk-ceremony/41/KC41_Script.pdf https://data.iana.org/ksk-ceremony/41/KC41_Script.pdf
  - brohee 6y ago
    You usually use your HSM vendor tooling, in the KCs I was involved the backup secrets were not only split between multiple holders but each holder held its part on a PIN protected smartcard.
  - PeterisP 6y ago
    For card system master keys, you don't use open source tools, you buy a set of properly hardened hardware modules, and follow the appropriate (documented, tested, verified) 'rituals' on them. IIRC the set we used cost something in the ballpark of $50k-$100k, which is not really much compared to all the other things that are table stakes of doing it properly. You can do it much cheaper if the risks are lower and you need less tamper resistance and auditing because you're less likely to have (for example) one of the trusted authorised employees be malicious and willing to invest nontrivial effort in circumventing the system. Because losing these keys can be very, very, very expensive. For an example (somewhat similar to this Postbank case) see India Cosmos Bank 2018 incident (https://www.reuters.com/article/cyber-heist-india-idUSL4N1V551G https://www.reuters.com/article/cyber-heist-india-idUSL4N1V5... is one link) where criminals generated fake cards to cash out some $13 million; and replacing 12M bank cards also has an huge cost to replace the cards (perhaps roughly $12M - $1 per card replacement is plausible though possibly on the cheap side) even if we ignore the reputation cost.
  - fortran77 6y ago
thoraway1010 6y ago
HSM's properly implemented do not result in clearext backups of these highly important keys. You can do backups without that.

Probably a backup. It makes sense to have an offline backup in cleartext (for DR), as long as you have the appropriate storage and security controls in place to protect it.

brohee 6y ago

It makes absolutely no sense. Such highly valuable secrets are usually saved using Shamir's secret sharing with parts of the split secret held by people unlikely to collude. Key ceremonies are done in a way that at no point a human being is in position to single-handedly extract the secret from its HSM. This is a huge failure.

m4rtink 6y ago
What surprises me is that you can extract the private key at all - I would expect it to be firmly inside a HSM, that only accepts signing requests and the key never leaves the module & the HSM wipes the key if it detects tampering (there is usually a battery inside to power the tampering detection even if the device is not plugged in). So just exporting the private key so easily without some pretty involved hight-tech HSM key extraction sounds insane.
- closeparen 6y ago
  I worked on a new internal PKI, nothing as high stakes as banking, but we did have a security consultant who had been around the block. We did, of course, purchase such HSMs and design a signing ceremony. But in his opinion it was also normal and expected to keep a decryptable copy of the private key, in case of e.g. changing HSM vendors. It would be even less accessible, but it would exist.
  - m4rtink 6y ago
    That's a good point - I guess if you know what you are doing then I guess it's fine.
- brohee 6y ago
  One article said they key was left in clear on a laptop, so maybe a HSM migration was involved.
briffle 6y ago
Interesting.. What Open source tools do you use for this? I would love to read further how I can not have a printed copy of a master key in a safety deposit box.
- stefan_ 6y ago
  Here is an implementation of this: https://www.cloudflare.com/dns/dnssec/root-signing-ceremony/ https://www.cloudflare.com/dns/dnssec/root-signing-ceremony/ They have a page for each signing ceremony: https://www.iana.org/dnssec/ceremonies/41 https://www.iana.org/dnssec/ceremonies/41 With a script of everything done: https://data.iana.org/ksk-ceremony/41/KC41_Script.pdf https://data.iana.org/ksk-ceremony/41/KC41_Script.pdf
- brohee 6y ago
  You usually use your HSM vendor tooling, in the KCs I was involved the backup secrets were not only split between multiple holders but each holder held its part on a PIN protected smartcard.
- PeterisP 6y ago
  For card system master keys, you don't use open source tools, you buy a set of properly hardened hardware modules, and follow the appropriate (documented, tested, verified) 'rituals' on them. IIRC the set we used cost something in the ballpark of $50k-$100k, which is not really much compared to all the other things that are table stakes of doing it properly. You can do it much cheaper if the risks are lower and you need less tamper resistance and auditing because you're less likely to have (for example) one of the trusted authorised employees be malicious and willing to invest nontrivial effort in circumventing the system. Because losing these keys can be very, very, very expensive. For an example (somewhat similar to this Postbank case) see India Cosmos Bank 2018 incident (https://www.reuters.com/article/cyber-heist-india-idUSL4N1V551G https://www.reuters.com/article/cyber-heist-india-idUSL4N1V5... is one link) where criminals generated fake cards to cash out some $13 million; and replacing 12M bank cards also has an huge cost to replace the cards (perhaps roughly $12M - $1 per card replacement is plausible though possibly on the cheap side) even if we ignore the reputation cost.
- fortran77 6y ago

thoraway1010 6y ago

HSM's properly implemented do not result in clearext backups of these highly important keys. You can do backups without that.