4 ms·

US flights grounded because engineer accidentally 'replaced one file'

10 points by eiiot 4y ago

olliej 4y ago
"US Flights grounded because a single mistake by a person not trying to do anything wrong". The question should be: why is a single error by a good faith actor able to take out flights across the US? What prevents a malicious actor grounding flights in the US given this obviously fragile mechanism?
- _8j50 4y ago
  Hindsight isn't a fair judge. Can one person's mistake bring down the internet for a day? It sure can. Can one person's mistake create a nuclear disaster, oil spill,war? Yup. And one person can also prevent those things except you rarely hear about averted disasters immediately.
  - Tanjreeve 4y ago
    Im curious what mistake could be made by one person to bring down the internet. You can definitely bring down a lot if you take out an AWS data centre or screw up DNS like incident at Facebook last year. But the internet even in its current chaotic form would seem a good example of having many redundancies to me. Closest thing I could think of is screw up a big DNS provider but even that would be partial.
    - _8j50 4y ago
      Bad routing update into the global routing table. Especially by a Tier1 transit isp. If at&t starts flooding with a lot if /32 (or /24 even) or makes itself the shortest AS path hop to a lot of ASes or important ASes it would be bad. Even with rpki and all that, ISPs that don't participate can cause a cascading failure. Just knocking out verisign's root server routes for example.
    - bradknowles 4y ago
      I accidentally took down all Internet e-mail for the entire world back in 1996. It was a basic DNS mis-configuration on my part for AOL, combined with the behaviour of typical e-mail servers at the time. This is a slightly popularized version of the actual events: https://www.theregister.com/2018/04/16/who_me/ https://www.theregister.com/2018/04/16/who_me/ Today, I think it would be easier to mess up the entire internet via a bad BGP routing advertisement. I think mail servers are now more robust against this kind of misconfiguration.
      - Tanjreeve 4y ago
        Damn, congratulations
karmakaze 4y ago
I'm curious what those "rules" were mentioned in another post[0] that were suspected of being circumvented. Yet another story[1] says: > Had the FAA’s new NOTAM system been in place, redundancies would likely have stopped the cascading failures. With the antiquated system in place, there was nothing to stop the outages, the official told ABC News. > "Now we have to understand how this could have happened in the first place. Why the usual redundancies that would stop it from being that disrupted, did not stop it from being disrupted this time, and what the original source of the errors or the corrupted files would have been," he said. So there were multiple failures that allowed this to happen. Redundancies if not routinely tested aren't failsafe. It's possible that this particular failure mode wouldn't have been caught even if redundancies were working as designed. [0] https://news.ycombinator.com/item?id=34381219#34381857 https://news.ycombinator.com/item?id=34381219#34381857 [1] https://abcnews.go.com/US/computer-failure-faa-impact-flights-nationwide/story?id=96358202 https://abcnews.go.com/US/computer-failure-faa-impact-flight...

10 points by eiiot 4y ago

"US Flights grounded because a single mistake by a person not trying to do anything wrong". The question should be: why is a single error by a good faith actor able to take out flights across the US? What prevents a malicious actor grounding flights in the US given this obviously fragile mechanism?

_8j50 4y ago
Hindsight isn't a fair judge. Can one person's mistake bring down the internet for a day? It sure can. Can one person's mistake create a nuclear disaster, oil spill,war? Yup. And one person can also prevent those things except you rarely hear about averted disasters immediately.
- Tanjreeve 4y ago
  Im curious what mistake could be made by one person to bring down the internet. You can definitely bring down a lot if you take out an AWS data centre or screw up DNS like incident at Facebook last year. But the internet even in its current chaotic form would seem a good example of having many redundancies to me. Closest thing I could think of is screw up a big DNS provider but even that would be partial.
  - _8j50 4y ago
    Bad routing update into the global routing table. Especially by a Tier1 transit isp. If at&t starts flooding with a lot if /32 (or /24 even) or makes itself the shortest AS path hop to a lot of ASes or important ASes it would be bad. Even with rpki and all that, ISPs that don't participate can cause a cascading failure. Just knocking out verisign's root server routes for example.
  - bradknowles 4y ago
    I accidentally took down all Internet e-mail for the entire world back in 1996. It was a basic DNS mis-configuration on my part for AOL, combined with the behaviour of typical e-mail servers at the time. This is a slightly popularized version of the actual events: https://www.theregister.com/2018/04/16/who_me/ https://www.theregister.com/2018/04/16/who_me/ Today, I think it would be easier to mess up the entire internet via a bad BGP routing advertisement. I think mail servers are now more robust against this kind of misconfiguration.
    - Tanjreeve 4y ago
      Damn, congratulations

karmakaze 4y ago

I'm curious what those "rules" were mentioned in another post[0] that were suspected of being circumvented. Yet another story[1] says: > Had the FAA’s new NOTAM system been in place, redundancies would likely have stopped the cascading failures. With the antiquated system in place, there was nothing to stop the outages, the official told ABC News. > "Now we have to understand how this could have happened in the first place. Why the usual redundancies that would stop it from being that disrupted, did not stop it from being disrupted this time, and what the original source of the errors or the corrupted files would have been," he said. So there were multiple failures that allowed this to happen. Redundancies if not routinely tested aren't failsafe. It's possible that this particular failure mode wouldn't have been caught even if redundancies were working as designed. [0] https://news.ycombinator.com/item?id=34381219#34381857 https://news.ycombinator.com/item?id=34381219#34381857 [1] https://abcnews.go.com/US/computer-failure-faa-impact-flights-nationwide/story?id=96358202 https://abcnews.go.com/US/computer-failure-faa-impact-flight...