5 ms·

It's just such an easy mistake to make. Some new ops guys logs all the traffic through a subsystem and boom - plaintext passwords.

Stuckinsofa 8y ago
Yeps. And at Facebook-scale, when you realize the issue after it has went live you have already "stored hundreds of millions of user passwords"
- kerng 8y ago
  That's a good point in favor of less hacking things together and more engineering - Facebooks seems to have not embraced strict engineering processes which are the only way to tackle these things. Facebook has to change it's own principles to succeed long term.
  - EnFinlay 8y ago
    I would say that this needs to change on a broader scale within the software development community than just with Facebook.
LeifCarrotson 8y ago
Why does traffic going through the subsystem contain plaintext passwords? It was my understanding yhat my password, on a properly configured login page, never left my browser, much less crossed multiple machines that had the ability to read it.
- detaro 8y ago
  No, your browser sends the password to the server in the HTTP request, only protected during transport by it being HTTPS. On Facebook's side, the HTTPS gets decrypted and the request is passed on (again potentially encrypted in transport, but the machines handling the request obviously need to see what is in the request, which includes your PW)
  - CodeSheikh 8y ago
    Ideally Facebook's website or any website should encrypt the password client side and send that via HTTP to the server where the server decrypts in internal and authenticate login session? Unfortunately, we are in 2019 and client-side hashing is rare because people use SSL instead. One can argue that company like Facebook that is well stocked with tech resources should have figured this out already but here we are.
    - httpz 8y ago
      Let's say a company does implement client side hashing. Now the server just needs the hash to log you in. Isn't the hash now basically the password? Now if the hash leaks or gets logged, a malicious user can still login to the company service using that hash. Only difference is, since it's not a plaintext password, with proper salting the user is somewhat protected from having their other services with similar password hacked.
      - ndiscussion 8y ago
        FB could also rotate the hash any time, with a short overlap period to allow for already-loaded pages, to eliminate this issue.
      - gylterud 8y ago
        Consider this approach: Let H be a hashing function and p be the users password. In the database you store a nounce n, and H(H(p⊕n)). When authenticating the server sends n, and client responds with x=H(p⊕n). Now server can compute H(x) and compare with the stored value to authenticate the client. Finally, after it has been authenticated, the client generates a new nounce n', and sends H(H(p⊕n')) and n' to the server, which is stored in the database for the next log in. Replace the outer H with a proper key derivation function for extra credits. This avoids sending any secret value over to the server, so no server side logging will cause a problem.
      - jeltz 8y ago
        SCRAM has solved this and with it the server can verify that the client has the plain text password while only storing the hashed password a d without exchanging a plain text password.
- jeltz 8y ago
  No. It would be nice if this was the case but browser sadly just send the password to the servers instead of using something like SCRAM which only requires you send the password when setting it and not on every authentication.
wy35 8y ago
An easy mistake with serious potential consequences. I'd be interested in hearing how Facebook will prevent this from happening again in the future
- sethammons 8y ago
  A lot of folks are angry about this, but I'm not seeing much in they way of "this is how that should be handled" aside from "be vigilant." The closest is "encrypt logs," but those logs exist to be seen, else they wouldn't exist.
  - qpiox 8y ago
    Hash client side. Send hash. This is how it is done for 20 years (by those who truly care about the users).
    - NoodleIncident 8y ago
      If this is so common, it should be easy for you to provide a link to the javascript password hasher on a site you use. Does this site do that? Pretty sure it doesn't. But you sound very confident about this, so I'm sure you'll have no trouble finding some other site that handles its passwords this way.
    - crazygringo 8y ago
      What?! No!!! As many have said before, transmitting the hash simply turns the hash into the password itself. Anyone who has your hash, has your password. The biggest reason for hashing is that, even if an attacker gets access to the hashed passwords, they still can't use them to log in. Client-side hashing completely and utterly undoes that benefit. Passwords in transit should be protected by SSL. Not a hash.
      - jeltz 8y ago
        You should check out SCRAM. It solves this problem without sending the hash or the plain text.
tonyjstark 8y ago
But then it's unbelievable that they don't have watchdog for this in place given how big and user-facing that company is. Also out of those 2000 employees having access at least some would have noticed that, wouldn't they? And it still took far to long to correct that.

Stuckinsofa 8y ago

Yeps. And at Facebook-scale, when you realize the issue after it has went live you have already "stored hundreds of millions of user passwords"

kerng 8y ago
That's a good point in favor of less hacking things together and more engineering - Facebooks seems to have not embraced strict engineering processes which are the only way to tackle these things. Facebook has to change it's own principles to succeed long term.
- EnFinlay 8y ago
  I would say that this needs to change on a broader scale within the software development community than just with Facebook.

LeifCarrotson 8y ago

Why does traffic going through the subsystem contain plaintext passwords? It was my understanding yhat my password, on a properly configured login page, never left my browser, much less crossed multiple machines that had the ability to read it.

detaro 8y ago
No, your browser sends the password to the server in the HTTP request, only protected during transport by it being HTTPS. On Facebook's side, the HTTPS gets decrypted and the request is passed on (again potentially encrypted in transport, but the machines handling the request obviously need to see what is in the request, which includes your PW)
- CodeSheikh 8y ago
  Ideally Facebook's website or any website should encrypt the password client side and send that via HTTP to the server where the server decrypts in internal and authenticate login session? Unfortunately, we are in 2019 and client-side hashing is rare because people use SSL instead. One can argue that company like Facebook that is well stocked with tech resources should have figured this out already but here we are.
  - httpz 8y ago
    Let's say a company does implement client side hashing. Now the server just needs the hash to log you in. Isn't the hash now basically the password? Now if the hash leaks or gets logged, a malicious user can still login to the company service using that hash. Only difference is, since it's not a plaintext password, with proper salting the user is somewhat protected from having their other services with similar password hacked.
    - ndiscussion 8y ago
      FB could also rotate the hash any time, with a short overlap period to allow for already-loaded pages, to eliminate this issue.
    - gylterud 8y ago
      Consider this approach: Let H be a hashing function and p be the users password. In the database you store a nounce n, and H(H(p⊕n)). When authenticating the server sends n, and client responds with x=H(p⊕n). Now server can compute H(x) and compare with the stored value to authenticate the client. Finally, after it has been authenticated, the client generates a new nounce n', and sends H(H(p⊕n')) and n' to the server, which is stored in the database for the next log in. Replace the outer H with a proper key derivation function for extra credits. This avoids sending any secret value over to the server, so no server side logging will cause a problem.
    - jeltz 8y ago
      SCRAM has solved this and with it the server can verify that the client has the plain text password while only storing the hashed password a d without exchanging a plain text password.
jeltz 8y ago
No. It would be nice if this was the case but browser sadly just send the password to the servers instead of using something like SCRAM which only requires you send the password when setting it and not on every authentication.

wy35 8y ago

An easy mistake with serious potential consequences. I'd be interested in hearing how Facebook will prevent this from happening again in the future

sethammons 8y ago
A lot of folks are angry about this, but I'm not seeing much in they way of "this is how that should be handled" aside from "be vigilant." The closest is "encrypt logs," but those logs exist to be seen, else they wouldn't exist.
- qpiox 8y ago
  Hash client side. Send hash. This is how it is done for 20 years (by those who truly care about the users).
  - NoodleIncident 8y ago
    If this is so common, it should be easy for you to provide a link to the javascript password hasher on a site you use. Does this site do that? Pretty sure it doesn't. But you sound very confident about this, so I'm sure you'll have no trouble finding some other site that handles its passwords this way.
  - crazygringo 8y ago
    What?! No!!! As many have said before, transmitting the hash simply turns the hash into the password itself. Anyone who has your hash, has your password. The biggest reason for hashing is that, even if an attacker gets access to the hashed passwords, they still can't use them to log in. Client-side hashing completely and utterly undoes that benefit. Passwords in transit should be protected by SSL. Not a hash.
    - jeltz 8y ago
      You should check out SCRAM. It solves this problem without sending the hash or the plain text.

tonyjstark 8y ago

But then it's unbelievable that they don't have watchdog for this in place given how big and user-facing that company is. Also out of those 2000 employees having access at least some would have noticed that, wouldn't they? And it still took far to long to correct that.