5 ms·

Seems like a pretty trivial automated test for so many PHD's to miss: create_user('Bob', 'BobPassword123') assert "BobPassword123" not in logfile

munk-a 7y ago
Hrm, so you checked the apache access logs, or maybe an error log, what about the system logs? Does the login request spin up a bash script and pass the password to it? I don't think it's trivial to guarantee non-existence.
- jlarocco 7y ago
  > I don't think it's trivial to guarantee non-existence. I disagree in this case. Log messages don't spontaneously appear in arbitrary places. If the developers understand what their software is doing and how their systems are configured then they should know where to check for the logging messages.
  - noselasd 7y ago
    Until someone turned on logging of the full request body on the load balancer/proxy or otherwise unknown-to-you middlebox that was the TLS termination point in production that you did not know about
    - fuzz4lyfe 7y ago
      So their incompetence lies in lacking configuration management in production as opposed to a lack of testing? Why are they manually configuring production servers anyway at that scale?
      - munk-a 7y ago
        They are managing a lot of things, they have full responsibility to be aware of these things as a company but I'd guess that with the scale of ops Facebook has regular developers never interact with the team that's managing their attempt at high availability by scattering servers across different hosts, data centers, legal jurisdictions etc. in fact I'd imagine at a op the size of Facebook that sort of problem has a dedicated department. That said, someone should have been watching for this stuff and failed to do so (or to exist), so I'm not excusing them - but this is not a trivial thing to protect against.
      - stickfigure 7y ago
        You speak with the hubris of a man that's never been in the arena.
        fuzz4lyfe 7y ago
        Facebook hires some of the top software developers and engineers on the planet, if not leaking plaintext passwords is too high of an expectation for them then nothing that isn't public knowledge should ever be put into any computer system. As a profession we should demand our peers do better than this.
        munk-a 7y ago
        We're all people of varying skills, I would never assume an innate high bar for any activity a human does - only by requiring the bar be maintained at a level and regularly checking and enforcing that requirement can we be reasonably sure it is - and this isn't just `echo $password` the way these passwords got into the log file is (from what I've been able to discern) pretty obscure and round about - Facebook is absolutely responsible and needs to be held to account, but the mistake is understandable.
      - icedchai 7y ago
        It's not really that simple. Sometimes you need to enable request logging in production... maybe for performance analysis, maybe for debugging.
        chopin 7y ago
        And you make sure that you can purge the logs afterwards.
    - jlarocco 7y ago
      And? That doesn't change anything I said. The new logging config isn't magic, either, and the new set of log files can be scanned, too. With billions of dollars and some the best software developers in the world (supposedly), Facebook should be able to figure this out.
- heavenlyblue 7y ago
  Run the deployment in docker, check every file changed in the container for it’s contents.
- wavefunction 7y ago
  yeah well we're not running apache >wut is this post?
kortilla 7y ago
Seems like a useless test considering how many actual production systems don’t log to local disk.
nkozyra 7y ago
Ah yes, the single, non-distributed, file-backed log Facebook uses.
- fuzz4lyfe 7y ago
  You have no problem with the create_user function? Obviously it's pseudocode my point is there are a finite number of log locations and checking for an instance of a known string among them isn't difficult.
  - UncleMeat 7y ago
    There are a finite number of log locations now. How do you intend to operate an integration test across the entirety of Facebook's stack to detect new log locations?
    - fuzz4lyfe 7y ago
      Change requests and a change review board staffed by professionals. Subject: Change Request Body: I would like to log the body of authentication requests in production. Subject Re: Change Request Body: How will you ensure personal data is not stored that shouldn't be? Subject: Re: Re: Change Request Body: I will add configuration Y to logging system X.
      - hw 7y ago
        Bureaucracy like this is what kills teams and products. There's no guarantee it works, and every change or commit should already have privacy and data protection in mind anyway. You also don't know what you don't know - you could for all your knowledge think that nothing is being logged, when in fact there's a subsystem outside your scope that's actually doing the logging. You assert that it's trivial, yet you're adding more layers to protect against something like that from happening. It's the naivety that all problems are trivial is what gets people and companies into trouble in the first place
        7y ago
gnulinux 7y ago
What if it logs "BobPassw[...]"? Is that more acceptable? What if there is a bug and some other function logs "[...]word123"?
skybrian 7y ago
If it turns out to be that simple to write a test, it probably didn't happen by accident. Many systems are more difficult to write tests for. Of course we (and they) should do it anyway, but it does often take an investment in making things testable.

munk-a 7y ago

Hrm, so you checked the apache access logs, or maybe an error log, what about the system logs? Does the login request spin up a bash script and pass the password to it? I don't think it's trivial to guarantee non-existence.

jlarocco 7y ago
> I don't think it's trivial to guarantee non-existence. I disagree in this case. Log messages don't spontaneously appear in arbitrary places. If the developers understand what their software is doing and how their systems are configured then they should know where to check for the logging messages.
- noselasd 7y ago
  Until someone turned on logging of the full request body on the load balancer/proxy or otherwise unknown-to-you middlebox that was the TLS termination point in production that you did not know about
  - fuzz4lyfe 7y ago
    So their incompetence lies in lacking configuration management in production as opposed to a lack of testing? Why are they manually configuring production servers anyway at that scale?
    - munk-a 7y ago
      They are managing a lot of things, they have full responsibility to be aware of these things as a company but I'd guess that with the scale of ops Facebook has regular developers never interact with the team that's managing their attempt at high availability by scattering servers across different hosts, data centers, legal jurisdictions etc. in fact I'd imagine at a op the size of Facebook that sort of problem has a dedicated department. That said, someone should have been watching for this stuff and failed to do so (or to exist), so I'm not excusing them - but this is not a trivial thing to protect against.
    - stickfigure 7y ago
      You speak with the hubris of a man that's never been in the arena.
      - fuzz4lyfe 7y ago
        Facebook hires some of the top software developers and engineers on the planet, if not leaking plaintext passwords is too high of an expectation for them then nothing that isn't public knowledge should ever be put into any computer system. As a profession we should demand our peers do better than this.
        munk-a 7y ago
        We're all people of varying skills, I would never assume an innate high bar for any activity a human does - only by requiring the bar be maintained at a level and regularly checking and enforcing that requirement can we be reasonably sure it is - and this isn't just `echo $password` the way these passwords got into the log file is (from what I've been able to discern) pretty obscure and round about - Facebook is absolutely responsible and needs to be held to account, but the mistake is understandable.
    - icedchai 7y ago
      It's not really that simple. Sometimes you need to enable request logging in production... maybe for performance analysis, maybe for debugging.
      - chopin 7y ago
        And you make sure that you can purge the logs afterwards.
  - jlarocco 7y ago
    And? That doesn't change anything I said. The new logging config isn't magic, either, and the new set of log files can be scanned, too. With billions of dollars and some the best software developers in the world (supposedly), Facebook should be able to figure this out.
heavenlyblue 7y ago
Run the deployment in docker, check every file changed in the container for it’s contents.
wavefunction 7y ago
yeah well we're not running apache >wut is this post?

kortilla 7y ago

Seems like a useless test considering how many actual production systems don’t log to local disk.

nkozyra 7y ago

Ah yes, the single, non-distributed, file-backed log Facebook uses.

fuzz4lyfe 7y ago
You have no problem with the create_user function? Obviously it's pseudocode my point is there are a finite number of log locations and checking for an instance of a known string among them isn't difficult.
- UncleMeat 7y ago
  There are a finite number of log locations now. How do you intend to operate an integration test across the entirety of Facebook's stack to detect new log locations?
  - fuzz4lyfe 7y ago
    Change requests and a change review board staffed by professionals. Subject: Change Request Body: I would like to log the body of authentication requests in production. Subject Re: Change Request Body: How will you ensure personal data is not stored that shouldn't be? Subject: Re: Re: Change Request Body: I will add configuration Y to logging system X.
    - hw 7y ago
      Bureaucracy like this is what kills teams and products. There's no guarantee it works, and every change or commit should already have privacy and data protection in mind anyway. You also don't know what you don't know - you could for all your knowledge think that nothing is being logged, when in fact there's a subsystem outside your scope that's actually doing the logging. You assert that it's trivial, yet you're adding more layers to protect against something like that from happening. It's the naivety that all problems are trivial is what gets people and companies into trouble in the first place
      - 7y ago

gnulinux 7y ago

What if it logs "BobPassw[...]"? Is that more acceptable? What if there is a bug and some other function logs "[...]word123"?

skybrian 7y ago

If it turns out to be that simple to write a test, it probably didn't happen by accident. Many systems are more difficult to write tests for. Of course we (and they) should do it anyway, but it does often take an investment in making things testable.