4 ms·

> I don't know anything about FB's infrastructure This explains a lot of the comments here. Facebook’s scale is not like anything most engineers have worked on

> I don't know anything about FB's infrastructure This explains a lot of the comments here. Facebook’s scale is not like anything most engineers have worked on. Facebook probably has logs in the 100s of terabytes. Ensuring that sensitive data isn’t logged takes more than some occasional greps.

EpicEng 8y ago
My point is that it's irrelevant. Have more data? Need more auditing. Any system which touches sensitive data is subject to security review. Yes, FB systems are massive. They should have massive oversight as well. You may as well be defending a nuclear meltdown because desiging nuclear power plants is hard. >Ensuring that sensitive data isn’t logged takes more than some occasional greps. Right; it requires investment into process, requirements, testing, and oversight. Most importantly, it requires a company wide, top down mentality that your customer's privacy and protection is more important than your margins. If you can't (won't) dedicate the resources required to ensure your customers data is protected then you have no business operating at such a scale.
- ryandrake 8y ago
  Exactly. Can every piece of data logged be tied back to a legitimate business purpose? What’s needed here is a mentality change: These logs should be thought of as liabilities rather than assets. You should log only what you need, while you need it, and then turn off the log when you’re done. If your mentality is “log everything, always, because maybe we’ll need it later” then these privacy and security trash fires should be expected.
  - viraptor 8y ago
    Log everything: you don't care about privacy. Treat logs as liabilities: why can't you solve the issue I experienced yesterday? You can't win. Either you log more than you think you need right now or you can't do engineering investigations on past data. You're going to end up somewhere in the middle realistically.
- matz1 8y ago
  Everything is a tradeoff, how much cost that fb incure due to this incident, I would guess not so much, at least not big enough to warrant massive resources needed.
  - EpicEng 8y ago
    Well, that's the problem really; they obviously don't care.
    - NoodleIncident 8y ago
      According to the timeline of events in The Fine Article, this story only exists because someone cared. At many companies the story would end at "one diff reviewer noticed passwords getting logged in one diff". All of the numbers in this story come from an internal investigation to see where else they're making the same mistake, _so they stop doing that_. That's not what you do if you don't care.
      - EpicEng 8y ago
        If FB truly cared it would likely never have happened, let alone gone on for years. I'm not saying no employee at FB cares; I'm saying FB as an organization doesn't, and we have plenty of "I'm sorry, we'll do better" statements to back that up.
- noidea_ 8y ago
  Sorry, you're pushing your slapdash opinion everywhere in here and you're wrong. You have what seems like a basic and small understanding of real security programs. >> Any system which touches sensitive data is subject to security review. This is such a meaningless statement. Facebook has one of the best AppSec programs in the world. They regularly face attacks from nation states. You absolutely could not handle the threat model that Facebook deals with. With thousands of engineers, hundreds of acquisitions, and data at a scale you cannot even fathom, mistakes happen. >> Right; it requires investment into process, requirements, testing, and oversight Again, this is nonsense. Facebook has a more mature program than you do, full stop. >> it requires a company wide, top down mentality that your customer's privacy and protection is more important than your margins. You are mixing things up. >> If you can't (won't) dedicate the resources required to ensure your customers data is protected then you have no business operating at such a scale. They have a larger security program than you. Alex Stamos is more capable than whomever you work for.
  - EpicEng 8y ago
    It's hard to take you seriously when your argument boils down to name calling and how obviously awesome FB is.
    - noidea_ 8y ago
      I'm indifferent to that. You shit on very competent engineers, thousands of whom are trying to do the right thing. You're trying to score cheap points by insulting people with a no nuance view. There's nothing worse than mediocre sysadmin's pontificating about security. Just remember this conversation whenever whatever you're responsible for gets popped.
      - EpicEng 8y ago
        I've been called many things, but never a sysadmin.
  - dang 8y ago
    Crossing into personal attack isn't allowed here, regardless of how right you are or feel you are. Most of your comments to HN have unfortunately been like this. Would you mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the spirit of this site to heart? It's particularly important when your points are right, so as not to discredit them.
freeflight 8y ago
This feels like a slightly differently framed version of the "Too big to fail" argument and does pretty much nothing for me. Collecting data on such massive scales is literally FB's whole business. But with that also comes a responsibility that shouldn't simply be waved away with "But they are so big, it's so difficult!" Because when it's about monetizing their massive amounts of, often illegally collected, data then FB seem to have no issues having everything in order and getting stuff to work, regardless of how "difficult" it might be. Probably has to do with the fact that there's no money in protecting users data properly and FB seems to be pretty much immune from negative PR having any bad consequences.
dboreham 8y ago
>takes more than some occasional greps Of course at FB scale you'd automate this by creating a set of canary accounts with unique passwords that you perform a search for in the ETL pipeline, or some other handy place. This will at least catch inadvertent plaintext password logging.
- PixyMisa 8y ago
  Facebook ops team: "We're waist-deep in dead canaries."
zeckalpha 8y ago
Only 100s of terabytes?

EpicEng 8y ago

My point is that it's irrelevant. Have more data? Need more auditing. Any system which touches sensitive data is subject to security review. Yes, FB systems are massive. They should have massive oversight as well. You may as well be defending a nuclear meltdown because desiging nuclear power plants is hard. >Ensuring that sensitive data isn’t logged takes more than some occasional greps. Right; it requires investment into process, requirements, testing, and oversight. Most importantly, it requires a company wide, top down mentality that your customer's privacy and protection is more important than your margins. If you can't (won't) dedicate the resources required to ensure your customers data is protected then you have no business operating at such a scale.

ryandrake 8y ago
Exactly. Can every piece of data logged be tied back to a legitimate business purpose? What’s needed here is a mentality change: These logs should be thought of as liabilities rather than assets. You should log only what you need, while you need it, and then turn off the log when you’re done. If your mentality is “log everything, always, because maybe we’ll need it later” then these privacy and security trash fires should be expected.
- viraptor 8y ago
  Log everything: you don't care about privacy. Treat logs as liabilities: why can't you solve the issue I experienced yesterday? You can't win. Either you log more than you think you need right now or you can't do engineering investigations on past data. You're going to end up somewhere in the middle realistically.
matz1 8y ago
Everything is a tradeoff, how much cost that fb incure due to this incident, I would guess not so much, at least not big enough to warrant massive resources needed.
- EpicEng 8y ago
  Well, that's the problem really; they obviously don't care.
  - NoodleIncident 8y ago
    According to the timeline of events in The Fine Article, this story only exists because someone cared. At many companies the story would end at "one diff reviewer noticed passwords getting logged in one diff". All of the numbers in this story come from an internal investigation to see where else they're making the same mistake, _so they stop doing that_. That's not what you do if you don't care.
    - EpicEng 8y ago
      If FB truly cared it would likely never have happened, let alone gone on for years. I'm not saying no employee at FB cares; I'm saying FB as an organization doesn't, and we have plenty of "I'm sorry, we'll do better" statements to back that up.
noidea_ 8y ago
Sorry, you're pushing your slapdash opinion everywhere in here and you're wrong. You have what seems like a basic and small understanding of real security programs. >> Any system which touches sensitive data is subject to security review. This is such a meaningless statement. Facebook has one of the best AppSec programs in the world. They regularly face attacks from nation states. You absolutely could not handle the threat model that Facebook deals with. With thousands of engineers, hundreds of acquisitions, and data at a scale you cannot even fathom, mistakes happen. >> Right; it requires investment into process, requirements, testing, and oversight Again, this is nonsense. Facebook has a more mature program than you do, full stop. >> it requires a company wide, top down mentality that your customer's privacy and protection is more important than your margins. You are mixing things up. >> If you can't (won't) dedicate the resources required to ensure your customers data is protected then you have no business operating at such a scale. They have a larger security program than you. Alex Stamos is more capable than whomever you work for.
- EpicEng 8y ago
  It's hard to take you seriously when your argument boils down to name calling and how obviously awesome FB is.
  - noidea_ 8y ago
    I'm indifferent to that. You shit on very competent engineers, thousands of whom are trying to do the right thing. You're trying to score cheap points by insulting people with a no nuance view. There's nothing worse than mediocre sysadmin's pontificating about security. Just remember this conversation whenever whatever you're responsible for gets popped.
    - EpicEng 8y ago
      I've been called many things, but never a sysadmin.
- dang 8y ago
  Crossing into personal attack isn't allowed here, regardless of how right you are or feel you are. Most of your comments to HN have unfortunately been like this. Would you mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the spirit of this site to heart? It's particularly important when your points are right, so as not to discredit them.

freeflight 8y ago

This feels like a slightly differently framed version of the "Too big to fail" argument and does pretty much nothing for me. Collecting data on such massive scales is literally FB's whole business. But with that also comes a responsibility that shouldn't simply be waved away with "But they are so big, it's so difficult!" Because when it's about monetizing their massive amounts of, often illegally collected, data then FB seem to have no issues having everything in order and getting stuff to work, regardless of how "difficult" it might be. Probably has to do with the fact that there's no money in protecting users data properly and FB seems to be pretty much immune from negative PR having any bad consequences.

dboreham 8y ago

>takes more than some occasional greps Of course at FB scale you'd automate this by creating a set of canary accounts with unique passwords that you perform a search for in the ETL pipeline, or some other handy place. This will at least catch inadvertent plaintext password logging.

PixyMisa 8y ago
Facebook ops team: "We're waist-deep in dead canaries."

zeckalpha 8y ago

Only 100s of terabytes?