4 ms·

It's mentioned in nested comments, but (as you'd probably expect) meta does not intend to store passwords in plaintext. There was a bug where they were logging

by foota 2y ago

It's mentioned in nested comments, but (as you'd probably expect) meta does not intend to store passwords in plaintext. There was a bug where they were logging plaintext passwords for some period of time e.g., when someone tried to log in etc.,.

gm3dmo 2y ago
Sanitize your outputs?
delusional 2y ago
> Meta does not intend Is an odd concept. Is the argument that nobody noticed? If somebody noticed, but the cleanup was deffered, them they did "intend to". It's like defending a bank robber by saying that he didn't intend to rob the bank, he just had a gun in his hand, and then he figured the damage was already done, so he may as well get some money.
- locallost 2y ago
  I think the comment is the context of being a software developer. "Everyone" knows you shouldn't do that, so it would be a bit odd if the company of Facebook's size would. But if it was accidental, then it makes it clearer how it happened. It's still a grave mistake, but not unthinkable. I personally write bugs all the time.
  - osullip 2y ago
    Logging passwords on the fly is probably common. Some debug or log action setup and forgotten. However, if you ever see a password in plain text you should raise alarms to the highest level. In this case, I don't think the alarm was raised.
    - cogman10 2y ago
      I agree, but also I know of devs that don't understand the basic security implications of passwords being in logs. I could easily see how someone, maybe even a couple of people, could see these logs and think nothing of them.
    - vidarh 2y ago
      Vast quantities of logs are never reviewed by anyone....
  - cogman10 2y ago
    I gotta level with you, not everyone knows you shouldn't do that. There's a number of devs that don't think twice about storing sensitive keys in a git repo. I could 100% see how someone would do this, see log messages with passwords in plain text, and then faff off being the last person to actually look at those logs. "K, this case is done, what's next"
    - vidarh 2y ago
      I worked somewhere not long ago where the "solution" was to run code to scrub repos before building release packages because these packages were in some cases installed in customer networks that drastically increased the chance they might leak. At no point did it seem developers or the devops team realised that the fact they saw the need to do this meant that maybe they should apply the same checks used to scrub in a hook to root them on commit in the first place.
  - tengwar2 2y ago
    "Everyone" does not know that. Avoiding plain text passwords is the commonest method used on the Internet, but if you get in to mobile telecoms, you find shared secrets stored in hardware-secure write-only enclaves in clear text. This is actually done because in that specific environment it increases security. It's not a general solution of course, but "only store encrypted passwords" and "only store password hashes" don't always apply either.
sschueller 2y ago
Boeing did not intend to have its plane crash when it installed the MCAS.
- sealeck 2y ago
  And it turns out that plane crashes are much more serious failures than logging sensitive data internally. Not to say what Facebook has done isn't an embarrasing failure that really shouldn't happen, but they're clearly not the same thing.
oefrha 2y ago
To the (intentionally?) obtuse responses: intending to store passwords in plaintext usually means storing plaintext passwords in databases and doing authentication with that; and that’s what the gazillion of commenters replying to the title are implying. Mistakenly logging credentials because of e.g. badly interacting HTTP middleware is still a very nasty bug, but it doesn’t count as intending to store passwords in plaintext. And something similar has happened to me, and I’m sure many others: logging middleware storing sensitive user data that shouldn’t be visible to engineers with mere access to logs (not as bad as logging credentials in my case, but still).
- nomilk 2y ago
  hmm.. that's a very sympathetic take. Most frameworks blot out passwords from logs by default, so even a newbie programmer on their first day doesn't make the mistake of logging plaintext passwords, yet facebook somehow made that mistake... It should raise eyebrows when the security practices of SWEs at a billion dollar company are outperformed by any newbie developer working a toy project.
  - yunohn 2y ago
    I think it’s actually much easier to make such mistakes at large companies with sprawling codebases and potential for settings that inadvertently end up logging sensitive data. Especially for nobody to notice either.
    - nomilk 2y ago
      True but they're also better resourced in terms of humans (and their experience) and tooling that should prevent or at least catch any blunders quickly.
      - yunohn 2y ago
        You’d be surprised, but from my first hand FAANG experience that is definitely not the case. Tooling can capture things that are known to it - but not everything is setup perfectly.
croes 2y ago
That's why the fine is only $102M
nolok 2y ago
And they're not fined for storing in plaintext, nor for storing in plaintext by mistake, they're fined because the law give a time limit for you to notify the regulator after you notice it and they waited too long. And in this specific case just to be clear it's not about taking too long to notify the public / customer, but about taking too long to notify regulator (the delay is much shorter). And they're not supposed to have perfect facts when they notify it, there is no sanction for notifying and saying "but we're not sure yet" if you're being honest, or coming back later with correction, there is a sanction for not telling them in time. We often see "companies should be responsible / should have to inform me" and that's part of that regulation, and it only works if there are clear defined delay and sanction when they're not respected.
- jart 2y ago
  How long has that even been a regulation and in which countries does it apply? Software engineers are trained to view these kinds of things as bugs. Legal isn't trained to monitor bug trackers.
  - JCharante 2y ago
    it's GDPR from the subheader > The Irish Data Protection Commission found that the company violated several GDPR rules. this is why lots of websites block the EU from accessing. You basically need to consult with lawyers to make sure you're not accidentally breaking the law when writing a codebase.
    - omnimus 2y ago
      “lots” not really as most companies want accesss to european market. Also no you dont need to consult lawyers when writing code. You just dont track and save data and do questionable stuff with it. Saving passwords in logs is surely security issue first before its GDPR issue.
      - JCharante 2y ago
        yes it's a security issue but you wouldn't "expect" to get fined millions of dollars. Do I think we should punish companies for storing passwords in plaintext? Yes. Would I expect that a bug and devs untrained in GDPR best practices could lead to fines? No. Usually in software engineering you don't get your company fined for making terrible mistakes unless you're in a field like finance. This was just passwords which most sites have, not something like PCI DSS stuff
nudgeee 2y ago
Indeed, this is a common vector for leaking PII and sensitive data. For example, what looks like an innocuous logging/print statement in an exception handler ends up leaking all sorts of data. And it gets more messy when you start to ingest and warehouse data logs for on-call monitoring/analytics/etc, and now you have PII floating around in all sorts of data stores that need to be scrubbed. In a previous job, we handled credit card numbers. We added PII detectors to logging libraries that would scrub anything that looked like a credit card number. We used client-side encryption where the credit card numbers are encrypted on the client before sending to the backend, so the backend systems never see the plain credit card numbers, except for the system that tokenizes them.
deleted 2y ago
[deleted]
tzs 2y ago
Also, it is not against GDPR per se to store passwords in plain text. It is required to keep user data safe from unauthorized access and processing and encryption is one way to help with that, but if you have it sufficiently protected by other means it would be OK under GDPR to have it in plain text. Avoiding ever storing passwords (or credit cards) in plain text [1] is actually harder than you might think. Even if you outsource password handling by using some third party authorization service so passwords should never even touch your servers, and handle credit card payments by having them handled on your checkout page by a frame or JavaScript or something that only ever sends them directly to your payment processor, you can still end up with the damn things in plain text on your servers. How? Because you receive emails to support that look like this: > Hello; I'm a subscriber to your service, account "Bob666", password "S3cr!t", billed to my credit card 4111111111111111 with security code 123. That card is expiring. My new card is 4012888888881881 with security code 712, expiration date May 2026. Please use my new card for renewals. Thanks! You may also receive messages like that in chat if you offer support via chat. And maybe even to email addresses other than support. So now you've got a plain text password and two plain text credit numbers along with their security codes stored in the inboxes of everyone who is on the support list, and possibly also somewhere in the database of your ticketing system if mail to support automatically creates a ticket. It gets worse. If you offer support by phone and that goes to voicemail after support hours you will find passwords and credit card numbers in there. [1] Note: technically probably almost nothing is actually stored in plain text nowadays. It's almost always going to stored on a filesystem that is using filesystem level encryption, and that filesystem is likely on a block device that is doing block level encryption. But I believe when people talk about "plain text" storage it means at a higher level. If I store the string "this is a secret" in foo.txt, that counts as plain text even though foo.txt is on an encrypted filesystem on an encrypted disk.

by foota 2y ago

gm3dmo 2y ago

Sanitize your outputs?

delusional 2y ago

> Meta does not intend Is an odd concept. Is the argument that nobody noticed? If somebody noticed, but the cleanup was deffered, them they did "intend to". It's like defending a bank robber by saying that he didn't intend to rob the bank, he just had a gun in his hand, and then he figured the damage was already done, so he may as well get some money.

locallost 2y ago
I think the comment is the context of being a software developer. "Everyone" knows you shouldn't do that, so it would be a bit odd if the company of Facebook's size would. But if it was accidental, then it makes it clearer how it happened. It's still a grave mistake, but not unthinkable. I personally write bugs all the time.
- osullip 2y ago
  Logging passwords on the fly is probably common. Some debug or log action setup and forgotten. However, if you ever see a password in plain text you should raise alarms to the highest level. In this case, I don't think the alarm was raised.
  - cogman10 2y ago
    I agree, but also I know of devs that don't understand the basic security implications of passwords being in logs. I could easily see how someone, maybe even a couple of people, could see these logs and think nothing of them.
  - vidarh 2y ago
    Vast quantities of logs are never reviewed by anyone....
- cogman10 2y ago
  I gotta level with you, not everyone knows you shouldn't do that. There's a number of devs that don't think twice about storing sensitive keys in a git repo. I could 100% see how someone would do this, see log messages with passwords in plain text, and then faff off being the last person to actually look at those logs. "K, this case is done, what's next"
  - vidarh 2y ago
    I worked somewhere not long ago where the "solution" was to run code to scrub repos before building release packages because these packages were in some cases installed in customer networks that drastically increased the chance they might leak. At no point did it seem developers or the devops team realised that the fact they saw the need to do this meant that maybe they should apply the same checks used to scrub in a hook to root them on commit in the first place.
- tengwar2 2y ago
  "Everyone" does not know that. Avoiding plain text passwords is the commonest method used on the Internet, but if you get in to mobile telecoms, you find shared secrets stored in hardware-secure write-only enclaves in clear text. This is actually done because in that specific environment it increases security. It's not a general solution of course, but "only store encrypted passwords" and "only store password hashes" don't always apply either.

sschueller 2y ago

Boeing did not intend to have its plane crash when it installed the MCAS.

sealeck 2y ago
And it turns out that plane crashes are much more serious failures than logging sensitive data internally. Not to say what Facebook has done isn't an embarrasing failure that really shouldn't happen, but they're clearly not the same thing.

oefrha 2y ago

To the (intentionally?) obtuse responses: intending to store passwords in plaintext usually means storing plaintext passwords in databases and doing authentication with that; and that’s what the gazillion of commenters replying to the title are implying. Mistakenly logging credentials because of e.g. badly interacting HTTP middleware is still a very nasty bug, but it doesn’t count as intending to store passwords in plaintext. And something similar has happened to me, and I’m sure many others: logging middleware storing sensitive user data that shouldn’t be visible to engineers with mere access to logs (not as bad as logging credentials in my case, but still).

nomilk 2y ago
hmm.. that's a very sympathetic take. Most frameworks blot out passwords from logs by default, so even a newbie programmer on their first day doesn't make the mistake of logging plaintext passwords, yet facebook somehow made that mistake... It should raise eyebrows when the security practices of SWEs at a billion dollar company are outperformed by any newbie developer working a toy project.
- yunohn 2y ago
  I think it’s actually much easier to make such mistakes at large companies with sprawling codebases and potential for settings that inadvertently end up logging sensitive data. Especially for nobody to notice either.
  - nomilk 2y ago
    True but they're also better resourced in terms of humans (and their experience) and tooling that should prevent or at least catch any blunders quickly.
    - yunohn 2y ago
      You’d be surprised, but from my first hand FAANG experience that is definitely not the case. Tooling can capture things that are known to it - but not everything is setup perfectly.

croes 2y ago

That's why the fine is only $102M

nolok 2y ago

And they're not fined for storing in plaintext, nor for storing in plaintext by mistake, they're fined because the law give a time limit for you to notify the regulator after you notice it and they waited too long. And in this specific case just to be clear it's not about taking too long to notify the public / customer, but about taking too long to notify regulator (the delay is much shorter). And they're not supposed to have perfect facts when they notify it, there is no sanction for notifying and saying "but we're not sure yet" if you're being honest, or coming back later with correction, there is a sanction for not telling them in time. We often see "companies should be responsible / should have to inform me" and that's part of that regulation, and it only works if there are clear defined delay and sanction when they're not respected.

jart 2y ago
How long has that even been a regulation and in which countries does it apply? Software engineers are trained to view these kinds of things as bugs. Legal isn't trained to monitor bug trackers.
- JCharante 2y ago
  it's GDPR from the subheader > The Irish Data Protection Commission found that the company violated several GDPR rules. this is why lots of websites block the EU from accessing. You basically need to consult with lawyers to make sure you're not accidentally breaking the law when writing a codebase.
  - omnimus 2y ago
    “lots” not really as most companies want accesss to european market. Also no you dont need to consult lawyers when writing code. You just dont track and save data and do questionable stuff with it. Saving passwords in logs is surely security issue first before its GDPR issue.
    - JCharante 2y ago
      yes it's a security issue but you wouldn't "expect" to get fined millions of dollars. Do I think we should punish companies for storing passwords in plaintext? Yes. Would I expect that a bug and devs untrained in GDPR best practices could lead to fines? No. Usually in software engineering you don't get your company fined for making terrible mistakes unless you're in a field like finance. This was just passwords which most sites have, not something like PCI DSS stuff

nudgeee 2y ago

Indeed, this is a common vector for leaking PII and sensitive data. For example, what looks like an innocuous logging/print statement in an exception handler ends up leaking all sorts of data. And it gets more messy when you start to ingest and warehouse data logs for on-call monitoring/analytics/etc, and now you have PII floating around in all sorts of data stores that need to be scrubbed. In a previous job, we handled credit card numbers. We added PII detectors to logging libraries that would scrub anything that looked like a credit card number. We used client-side encryption where the credit card numbers are encrypted on the client before sending to the backend, so the backend systems never see the plain credit card numbers, except for the system that tokenizes them.

deleted 2y ago

[deleted]

tzs 2y ago

Also, it is not against GDPR per se to store passwords in plain text. It is required to keep user data safe from unauthorized access and processing and encryption is one way to help with that, but if you have it sufficiently protected by other means it would be OK under GDPR to have it in plain text. Avoiding ever storing passwords (or credit cards) in plain text [1] is actually harder than you might think. Even if you outsource password handling by using some third party authorization service so passwords should never even touch your servers, and handle credit card payments by having them handled on your checkout page by a frame or JavaScript or something that only ever sends them directly to your payment processor, you can still end up with the damn things in plain text on your servers. How? Because you receive emails to support that look like this: > Hello; I'm a subscriber to your service, account "Bob666", password "S3cr!t", billed to my credit card 4111111111111111 with security code 123. That card is expiring. My new card is 4012888888881881 with security code 712, expiration date May 2026. Please use my new card for renewals. Thanks! You may also receive messages like that in chat if you offer support via chat. And maybe even to email addresses other than support. So now you've got a plain text password and two plain text credit numbers along with their security codes stored in the inboxes of everyone who is on the support list, and possibly also somewhere in the database of your ticketing system if mail to support automatically creates a ticket. It gets worse. If you offer support by phone and that goes to voicemail after support hours you will find passwords and credit card numbers in there. [1] Note: technically probably almost nothing is actually stored in plain text nowadays. It's almost always going to stored on a filesystem that is using filesystem level encryption, and that filesystem is likely on a block device that is doing block level encryption. But I believe when people talk about "plain text" storage it means at a higher level. If I store the string "this is a secret" in foo.txt, that counts as plain text even though foo.txt is on an encrypted filesystem on an encrypted disk.