3 ms·

If these keys are so important, why are they not treated with similar security measures as CA keys? Why is it just a handful of keys shared by dozens, hundreds

by crote 3y ago

If these keys are so important, why are they not treated with similar security measures as CA keys? Why is it just a handful of keys shared by dozens, hundreds, or likely even thousands of servers? Where is the HSM-backed root key, per-DC intermediate, and per-server transient signing key? Where is the certificate revocation list? Is it even possible to secure a bearer token from an attack like this? Should we go back to a nonce which we present to the originating service to securely retrieve the payload?

vladvasiliu 3y ago
It's not clear to me why there aren't more signing keys. After all, the public part is exposed at an URL you can retrieve whenever you're verifying the token. I've seen service provider implementations where the access token is exchanged for a cookie or other similar token, in which case you can't really do anything about it anymore for the lifetime of the cookie. But if you're always verifying the access token with a reasonably short cached key list, the service provider should be able to refuse the token reasonably quickly once the key is no longer advertised.
- ThomasRooney 3y ago
  Anecdotally, I ran an experiment with Envoy to see how far the number of signing keys could scale. This was for a B2B “API Key” auth solution; we wanted user keys to be self revocable, but just be a relatively standard JWT format for maintainability. The hypothesis was that, rather than running a whitelist or blacklist, we could improve the security signature by have 1 signing key for each JWT. When we ran some stress tests, turned out Envoy could happily run with ~300K signing keys in its JWK Set before noticeable service degradation occurred. Even then, by bumping up the memory on the validation servers, there was a small sacrifice of a few ms per extra 100K keys. This makes me fully agree that, for many applications, there’s probably an opportunity to vastly improve the security surface by bumping up the number of signing keys dramatically. As long as both Keys and Signing keys define a KID, key verification is prefaced only by a hash table lookup or a tight loop through a keyset to find the appropriate Signing Key, before the slower verification procedure.
  - vladvasiliu 3y ago
    I guess 640K keys ought to be enough for anybody (TM)(r) (c) More seriously, though, I wonder how AzureAD is implemented and how hard it would be to scope keys per tenant, if not per application. If I'm not mistaken, SAML certificates are per application.
  - specialist 3y ago
    ELI5, sorry: Taking your POC just a bit further and you've got the basis for zero trust networking, right? That's the Future Perfect Correct Answer™, right?
    - mistrial9 3y ago
      zero trust, or zero stability ?
      - specialist 3y ago
        LMGTFY https://en.wikipedia.org/wiki/Zero_trust_security_model https://en.wikipedia.org/wiki/Zero_trust_security_model
  - rvdginste 3y ago
    If you want 1 signing key per JWT, you would need to generate a new key pair for each JWT; wouldn’t that be too expensive? Or was the generation included in your tests?
jrockway 3y ago
> If these keys are so important, why are they not treated with similar security measures as CA keys? No incentive, right? If you mess up as a CA, you go out of business because every browser drops you. But if you're an identity provider and you have an incident, maybe 1% of your customers opens an angry support ticket and you give them 5% off their bill next month. So why bother? Are YOU going to move your 100,000 employee organization off of Microsoft because of this? No? Then why would they care at all? They won the sale and you're stuck with what you get. Here's a list of companies that have gone out of business because they didn't take information security seriously: <list not found> Here's a list of billionaires who had to sell their yachts because they didn't prioritize security over other concerns: <list not found> Yeah.
- fauigerzigerk 3y ago
  >No incentive, right? If you mess up as a CA, you go out of business because every browser drops you. Not going out of business is not the only possible incentive though. Security is a pretty key part of the sales pitch of every cloud provider: "We can do this better than your in-house team" they keep saying. No one is going to move 100,000 employees off Microsoft over this, but what about signing up the next 100,000? Will it impact next quarter's growth figures? And for the people directly responsible for security at Microsoft breaches like these may be a career risk or at least a huge embarrassment. There will be meetings. People will get blamed. Some will feel they haven't met their own professional standards. There's no lack of incentives.
  - mistrial9 3y ago
    you claim that contracts at this level are executed on the basis of technical merit?
    - fauigerzigerk 3y ago
      I'm not sure I would call it technical merit, but sentiment and reputation do play a role in negotiations. No cloud provider wants to be perceived as having a weakness in terms of security compared to its closest competitors. There are debates about risks when large companies move to the cloud - in some industries more than in others. Opposing groups within companies as well as competitors' sales people will be looking for weak spots to make their case. In isolation, incidents like this don't make a difference. If it looks like a pattern that would be bad for business though.
DANmode 3y ago
> why are they not treated with similar security measures as CA keys? Well, that's an unintentionally loaded question if I've ever seen one.

by crote 3y ago

vladvasiliu 3y ago

It's not clear to me why there aren't more signing keys. After all, the public part is exposed at an URL you can retrieve whenever you're verifying the token. I've seen service provider implementations where the access token is exchanged for a cookie or other similar token, in which case you can't really do anything about it anymore for the lifetime of the cookie. But if you're always verifying the access token with a reasonably short cached key list, the service provider should be able to refuse the token reasonably quickly once the key is no longer advertised.

ThomasRooney 3y ago
Anecdotally, I ran an experiment with Envoy to see how far the number of signing keys could scale. This was for a B2B “API Key” auth solution; we wanted user keys to be self revocable, but just be a relatively standard JWT format for maintainability. The hypothesis was that, rather than running a whitelist or blacklist, we could improve the security signature by have 1 signing key for each JWT. When we ran some stress tests, turned out Envoy could happily run with ~300K signing keys in its JWK Set before noticeable service degradation occurred. Even then, by bumping up the memory on the validation servers, there was a small sacrifice of a few ms per extra 100K keys. This makes me fully agree that, for many applications, there’s probably an opportunity to vastly improve the security surface by bumping up the number of signing keys dramatically. As long as both Keys and Signing keys define a KID, key verification is prefaced only by a hash table lookup or a tight loop through a keyset to find the appropriate Signing Key, before the slower verification procedure.
- vladvasiliu 3y ago
  I guess 640K keys ought to be enough for anybody (TM)(r) (c) More seriously, though, I wonder how AzureAD is implemented and how hard it would be to scope keys per tenant, if not per application. If I'm not mistaken, SAML certificates are per application.
- specialist 3y ago
  ELI5, sorry: Taking your POC just a bit further and you've got the basis for zero trust networking, right? That's the Future Perfect Correct Answer™, right?
  - mistrial9 3y ago
    zero trust, or zero stability ?
    - specialist 3y ago
      LMGTFY https://en.wikipedia.org/wiki/Zero_trust_security_model https://en.wikipedia.org/wiki/Zero_trust_security_model
- rvdginste 3y ago
  If you want 1 signing key per JWT, you would need to generate a new key pair for each JWT; wouldn’t that be too expensive? Or was the generation included in your tests?

jrockway 3y ago

> If these keys are so important, why are they not treated with similar security measures as CA keys? No incentive, right? If you mess up as a CA, you go out of business because every browser drops you. But if you're an identity provider and you have an incident, maybe 1% of your customers opens an angry support ticket and you give them 5% off their bill next month. So why bother? Are YOU going to move your 100,000 employee organization off of Microsoft because of this? No? Then why would they care at all? They won the sale and you're stuck with what you get. Here's a list of companies that have gone out of business because they didn't take information security seriously: <list not found> Here's a list of billionaires who had to sell their yachts because they didn't prioritize security over other concerns: <list not found> Yeah.

fauigerzigerk 3y ago
>No incentive, right? If you mess up as a CA, you go out of business because every browser drops you. Not going out of business is not the only possible incentive though. Security is a pretty key part of the sales pitch of every cloud provider: "We can do this better than your in-house team" they keep saying. No one is going to move 100,000 employees off Microsoft over this, but what about signing up the next 100,000? Will it impact next quarter's growth figures? And for the people directly responsible for security at Microsoft breaches like these may be a career risk or at least a huge embarrassment. There will be meetings. People will get blamed. Some will feel they haven't met their own professional standards. There's no lack of incentives.
- mistrial9 3y ago
  you claim that contracts at this level are executed on the basis of technical merit?
  - fauigerzigerk 3y ago
    I'm not sure I would call it technical merit, but sentiment and reputation do play a role in negotiations. No cloud provider wants to be perceived as having a weakness in terms of security compared to its closest competitors. There are debates about risks when large companies move to the cloud - in some industries more than in others. Opposing groups within companies as well as competitors' sales people will be looking for weak spots to make their case. In isolation, incidents like this don't make a difference. If it looks like a pattern that would be bad for business though.

DANmode 3y ago

> why are they not treated with similar security measures as CA keys? Well, that's an unintentionally loaded question if I've ever seen one.