5 ms·

Could one single person really inflict that much damage without a balance/check or code deployment review process in place?

bink 5y ago
With the right access it wouldn't be hard for someone to configure some key routers in such a way that all traffic is blocked and no one can get into them over the network. They'd need to send someone physically to the sites to reset or replace them. I'm still leaning heavily towards an "oopsie" with routing that accomplished the same thing, however.
- dilyevsky 5y ago
  Rollout gone bad is def more likely. People with that level of access aren’t stupid enough to get sued for tens of millions of $ in damages
  - newsclues 5y ago
    It would not be the first time
notjesse 5y ago
I think if you were smart enough, you may be able to mask some needed changes under some legitimate tickets. You make certain changes that you know will break stuff, but you assign a reviewer who doesn't know enough about the particular thing that they may think it seems fine. I am talking in a very generalized sense, not for this particular issue. But I don't think the code review/deployment process is entirely safe against internal bad actors.
- Graffur 5y ago
  Code reviews can potentially catch bugs and prevent an obvious inside attack but are mostly to keep the code-base healthy and consistent over time. Something that can take down multiple revenue streams for all customers should have some other check besides a peer code review.
- Sanzig 5y ago
  See: The Underhanded C Contest. http://underhanded-c.org/ http://underhanded-c.org/ The whole point is to write C that appears on the level at first, but actually has a subtle exploitable flaw. The flaw is supposed to appear like a simple mistake for plausible deniability. Some of the winning responses are very devious.
mrkramer 5y ago
System administrator with high authority can pretty much bring down the whole network stack if s/he wants to.
- lokar 5y ago
  In general, access levels are pretty liberal at FB (less so then in the past, but still liberal compared to others)
HelloNurse 5y ago
A network engineer with enough experience to handle Facebook's DNS and BGP configurations can probably design a plausibly deniable mistake/misunderstanding/unfortunate coincidence.
- polynomial 5y ago
  By the time it gets to that point, they may likely have already been identified as not a good cultural fit, unless they were a real sleeper.
caturopath 5y ago
I'm sure there are measures in place to prevent it, but at the end of the day someone almost certainly could and would have to be able to.

bink 5y ago

With the right access it wouldn't be hard for someone to configure some key routers in such a way that all traffic is blocked and no one can get into them over the network. They'd need to send someone physically to the sites to reset or replace them. I'm still leaning heavily towards an "oopsie" with routing that accomplished the same thing, however.

dilyevsky 5y ago
Rollout gone bad is def more likely. People with that level of access aren’t stupid enough to get sued for tens of millions of $ in damages
- newsclues 5y ago
  It would not be the first time

notjesse 5y ago

I think if you were smart enough, you may be able to mask some needed changes under some legitimate tickets. You make certain changes that you know will break stuff, but you assign a reviewer who doesn't know enough about the particular thing that they may think it seems fine. I am talking in a very generalized sense, not for this particular issue. But I don't think the code review/deployment process is entirely safe against internal bad actors.

Graffur 5y ago
Code reviews can potentially catch bugs and prevent an obvious inside attack but are mostly to keep the code-base healthy and consistent over time. Something that can take down multiple revenue streams for all customers should have some other check besides a peer code review.
Sanzig 5y ago
See: The Underhanded C Contest. http://underhanded-c.org/ http://underhanded-c.org/ The whole point is to write C that appears on the level at first, but actually has a subtle exploitable flaw. The flaw is supposed to appear like a simple mistake for plausible deniability. Some of the winning responses are very devious.

mrkramer 5y ago

System administrator with high authority can pretty much bring down the whole network stack if s/he wants to.

lokar 5y ago
In general, access levels are pretty liberal at FB (less so then in the past, but still liberal compared to others)

HelloNurse 5y ago

A network engineer with enough experience to handle Facebook's DNS and BGP configurations can probably design a plausibly deniable mistake/misunderstanding/unfortunate coincidence.

polynomial 5y ago
By the time it gets to that point, they may likely have already been identified as not a good cultural fit, unless they were a real sleeper.

caturopath 5y ago

I'm sure there are measures in place to prevent it, but at the end of the day someone almost certainly could and would have to be able to.