6 ms·

I mentioned this on Twitter[1], the reason to encrypt before sending over SSL is not about double encryption, it’s about how a large backend system is designed.

I mentioned this on Twitter[1], the reason to encrypt before sending over SSL is not about double encryption, it’s about how a large backend system is designed. Often, you have load balancers that are SSL endpoints, so the data is decrypted at that point. You can start to see the problem already. What if there is a different bug, and so a dev starts logging requests somewhere down the line? You accidentally start logging cleartext passwords. Oops. Facebook was fined for this not that long ago. But if the password is encrypted, then it’s not really an issue, and the black box blob can be forwarded to a login microservice. There, the team decrypting will be on higher alert. So depending on the structure of various teams, you now have fewer teams that need that kind of security oversight and can move faster. Smaller blast radius of something goes wrong. [1] https://twitter.com/sroussey/status/1347688753221931010?s=21 https://twitter.com/sroussey/status/1347688753221931010?s=21

xwdv 6y ago
So what you’re saying is if you know what you’re doing and you don’t log any clear text passwords it’s fine to not double encrypt?
- colechristensen 6y ago
  If you are immune from mistakes, sure. Making decisions as if you are immune to mistakes is a good way to ensure you prove yourself wrong in the worst way. Making decisions to prevent things which should be impossible is how you build reliability.
  - remexre 6y ago
    And, adding assertions / very loud logs is a good way to be sure your assumptions hold: e.g. causing your tests to fail if the test user's password (or other privileged info, etc) appears in any logs is a fairly good extra safety layer.
    - sroussey 6y ago
      That’s a great test idea I have not used myself, thanks! I’m also working on forks of logging systems to make them smart about what they log in the first place, but not quite like the Palantir does it.
- ehsankia 6y ago
  It's extremely naive to think that bugs and mistakes only happen to those who "don't know what they're doing".
  - xwdv 6y ago
    At some point we must simply “know what we are doing” and do something. Software would be a tedious affair if you have to approach everything from the perspective that maybe you don’t really know what you’re doing and need to implement some way to verify.
    - Kalium 6y ago
      What are unit tests, linters, SAST tools, and more all for? Naively, it seems to me that we swim in a great sea of tools to verify that we do, in fact, know what we're doing.
      - xwdv 6y ago
        How do you know your tests are correctly testing what you need to test? Tests all the way down?
        Kalium 6y ago
        An excellent question! One reflecting true wisdom. The answer is that you never have true certainty. What you get is smaller error bars with each set of tools and tests until you find the error bars and effort involved both acceptable.
    - ehsankia 6y ago
      The other comment mentions tests, but I'd like to that to that. The reason why it's impossible for anyone, even the single greatest programmer on the planet, to not create bugs, is due to the high complexity of the software we write. It's not so much that you'll make a mistake when creating a new program. It's that over time, as you make more and more changes, it's impossible to actually trace out the exponential complexity that arises from your changes. The purpose of unit tests is to catch these, and yes as you mention lower down even those aren't 100% infallible but they greatly help. That's why even with some of the greatest programmers and extensive testing, we still constantly see major bugs from every single top tech company. I don't think there exists a single completely bug-free software of non-negligible size out there.
codexon 6y ago
Why not hash the password instead?
- andreareina 6y ago
  Hashing the password client-side makes a leak of the password hashes (which happens all the time) equivalent to leaking the password itself. https://en.wikipedia.org/wiki/Pass_the_hash https://en.wikipedia.org/wiki/Pass_the_hash
  - deleted 6y ago
    [deleted]
  - codexon 6y ago
    Then just add a time sensitive seed to it? I don't think it is equivalent to leaking plaintext. It can't be used to guess passwords on other websites. If your SSL layer is compromised, you can't trust the client-side encryption. The attacker can send arbitrary javascript.
    - Kalium 6y ago
      > Then just add a time sensitive seed to it? I don't think it is equivalent to leaking plaintext. It can't be used to guess passwords on other websites. You're reinventing password hashing and salting. Further, there's no guarantee that the has cannot be used to guess passwords on other sites. For what benefit, exactly? Your hash is now the password, and basically as dangerous as it was in a more conventional arrangement. Pass-the-hash is a real kind of vulnerability that has been used to exploit real systems. We might be better off sticking with design approaches that don't have this problem instead of trying to fix out way out of the problem. > If your SSL layer is compromised, you can't trust the client-side encryption. The attacker can send arbitrary javascript. Are you sure this is what it's guarding against? A sophisticated application architecture might involve a load balancer decrypting and doing the initial routing, several sets of data handoffs, and then the application that needs it handling the password. Any one of them could mishandle or leak the password, but only the one at the end actually needs it in the clear.
tialaramex 6y ago
I agree this may be why they did it. However a better approach to this problem is to not rely on shared secrets. Use public key signature tech to stop worrying about mistakenly logging a secret. If you never had it, you can't lose it. If you were to log literally every byte of the plaintext traffic when I sign into GitHub (e.g. maybe you're a GitHub ops person), you don't get the ability to sign into GitHub as me. There's a WebAuthn signature step, my signature is authentic, and you can even verify that from your log if you want, but you'd need to make a new signature to sign in, and you can't do that because the key needed to make my signature never left my hands. Even better, GitHub defuses their liability because as well as a (presumably hashed) password that could be broken by a hypothetical attacker they've got a public key for me, and learning that public key doesn't help the attacker do anything, at their site or anywhere else. Even - unlike with the SSH public keys GitHub holds - to identify people, since WebAuthn public keys are deliberately uncorrelated you can't match my GitHub key against a Facebook key for example.
staticassertion 6y ago
Yeah, I do hashing client side as well as server side. It's not ideal - what would be ideal is zero knowledge proofs for such a thing. But it's basically: hash(static-pepper, username, password) * 250k That + tagging the password with "password++" or something means that you're a lot safer against the major issue of leaking a password before it's stored, for example the mistake that definitely happens everywhere of "let me just add request logging, whoops there's everyone's plaintext passwords". You can always search your logs for the 'password++' tag and alert if you find it, and if that does happen at least you know an attacker isn't going to have an easy time extracting a plaintext password - it buys you time. And if an attacker gets SQLi or whatever and dumps the passwords they're that much harder to crack - you've added hundreds of thousands of iterations of key stretching, and it's totally distributed to clients so you don't even have to worry about it blowing up your db/ auth service CPU. And it's trivial to implement, which is the really important part. ZKP is a lot more work, but what I described is like 5 extra minutes and pretty trivial.
- darkr 6y ago
  > hash(static-pepper, username, password) * 250k Though this is obviously better from a wire-interception PoV, it means that you can't enforce any password policies, or maintain a list of leaked/bad passwords (e.g HIBP)
  - staticassertion 6y ago
    You can enforce password policy client side. A technical user can go way out of their way to bypass it, but honestly, at that point who cares? If you really want to you can send up some metadata or something I guess. Another option is to send the password at user creation time, but then to rely on a hash at login time. Now there's leak potential, but it's you just have to audit for leaks in one part of the codebase. There's a lotta stuff you can do to improve upon the very quick version I'd mentioned.

xwdv 6y ago

So what you’re saying is if you know what you’re doing and you don’t log any clear text passwords it’s fine to not double encrypt?

colechristensen 6y ago
If you are immune from mistakes, sure. Making decisions as if you are immune to mistakes is a good way to ensure you prove yourself wrong in the worst way. Making decisions to prevent things which should be impossible is how you build reliability.
- remexre 6y ago
  And, adding assertions / very loud logs is a good way to be sure your assumptions hold: e.g. causing your tests to fail if the test user's password (or other privileged info, etc) appears in any logs is a fairly good extra safety layer.
  - sroussey 6y ago
    That’s a great test idea I have not used myself, thanks! I’m also working on forks of logging systems to make them smart about what they log in the first place, but not quite like the Palantir does it.
ehsankia 6y ago
It's extremely naive to think that bugs and mistakes only happen to those who "don't know what they're doing".
- xwdv 6y ago
  At some point we must simply “know what we are doing” and do something. Software would be a tedious affair if you have to approach everything from the perspective that maybe you don’t really know what you’re doing and need to implement some way to verify.
  - Kalium 6y ago
    What are unit tests, linters, SAST tools, and more all for? Naively, it seems to me that we swim in a great sea of tools to verify that we do, in fact, know what we're doing.
    - xwdv 6y ago
      How do you know your tests are correctly testing what you need to test? Tests all the way down?
      - Kalium 6y ago
        An excellent question! One reflecting true wisdom. The answer is that you never have true certainty. What you get is smaller error bars with each set of tools and tests until you find the error bars and effort involved both acceptable.
  - ehsankia 6y ago
    The other comment mentions tests, but I'd like to that to that. The reason why it's impossible for anyone, even the single greatest programmer on the planet, to not create bugs, is due to the high complexity of the software we write. It's not so much that you'll make a mistake when creating a new program. It's that over time, as you make more and more changes, it's impossible to actually trace out the exponential complexity that arises from your changes. The purpose of unit tests is to catch these, and yes as you mention lower down even those aren't 100% infallible but they greatly help. That's why even with some of the greatest programmers and extensive testing, we still constantly see major bugs from every single top tech company. I don't think there exists a single completely bug-free software of non-negligible size out there.

codexon 6y ago

Why not hash the password instead?

andreareina 6y ago
Hashing the password client-side makes a leak of the password hashes (which happens all the time) equivalent to leaking the password itself. https://en.wikipedia.org/wiki/Pass_the_hash https://en.wikipedia.org/wiki/Pass_the_hash
- deleted 6y ago
  [deleted]
- codexon 6y ago
  Then just add a time sensitive seed to it? I don't think it is equivalent to leaking plaintext. It can't be used to guess passwords on other websites. If your SSL layer is compromised, you can't trust the client-side encryption. The attacker can send arbitrary javascript.
  - Kalium 6y ago
    > Then just add a time sensitive seed to it? I don't think it is equivalent to leaking plaintext. It can't be used to guess passwords on other websites. You're reinventing password hashing and salting. Further, there's no guarantee that the has cannot be used to guess passwords on other sites. For what benefit, exactly? Your hash is now the password, and basically as dangerous as it was in a more conventional arrangement. Pass-the-hash is a real kind of vulnerability that has been used to exploit real systems. We might be better off sticking with design approaches that don't have this problem instead of trying to fix out way out of the problem. > If your SSL layer is compromised, you can't trust the client-side encryption. The attacker can send arbitrary javascript. Are you sure this is what it's guarding against? A sophisticated application architecture might involve a load balancer decrypting and doing the initial routing, several sets of data handoffs, and then the application that needs it handling the password. Any one of them could mishandle or leak the password, but only the one at the end actually needs it in the clear.

tialaramex 6y ago

I agree this may be why they did it. However a better approach to this problem is to not rely on shared secrets. Use public key signature tech to stop worrying about mistakenly logging a secret. If you never had it, you can't lose it. If you were to log literally every byte of the plaintext traffic when I sign into GitHub (e.g. maybe you're a GitHub ops person), you don't get the ability to sign into GitHub as me. There's a WebAuthn signature step, my signature is authentic, and you can even verify that from your log if you want, but you'd need to make a new signature to sign in, and you can't do that because the key needed to make my signature never left my hands. Even better, GitHub defuses their liability because as well as a (presumably hashed) password that could be broken by a hypothetical attacker they've got a public key for me, and learning that public key doesn't help the attacker do anything, at their site or anywhere else. Even - unlike with the SSH public keys GitHub holds - to identify people, since WebAuthn public keys are deliberately uncorrelated you can't match my GitHub key against a Facebook key for example.

staticassertion 6y ago

Yeah, I do hashing client side as well as server side. It's not ideal - what would be ideal is zero knowledge proofs for such a thing. But it's basically: hash(static-pepper, username, password) * 250k That + tagging the password with "password++" or something means that you're a lot safer against the major issue of leaking a password before it's stored, for example the mistake that definitely happens everywhere of "let me just add request logging, whoops there's everyone's plaintext passwords". You can always search your logs for the 'password++' tag and alert if you find it, and if that does happen at least you know an attacker isn't going to have an easy time extracting a plaintext password - it buys you time. And if an attacker gets SQLi or whatever and dumps the passwords they're that much harder to crack - you've added hundreds of thousands of iterations of key stretching, and it's totally distributed to clients so you don't even have to worry about it blowing up your db/ auth service CPU. And it's trivial to implement, which is the really important part. ZKP is a lot more work, but what I described is like 5 extra minutes and pretty trivial.

darkr 6y ago
> hash(static-pepper, username, password) * 250k Though this is obviously better from a wire-interception PoV, it means that you can't enforce any password policies, or maintain a list of leaked/bad passwords (e.g HIBP)
- staticassertion 6y ago
  You can enforce password policy client side. A technical user can go way out of their way to bypass it, but honestly, at that point who cares? If you really want to you can send up some metadata or something I guess. Another option is to send the password at user creation time, but then to rely on a hash at login time. Now there's leak potential, but it's you just have to audit for leaks in one part of the codebase. There's a lotta stuff you can do to improve upon the very quick version I'd mentioned.