4 ms·
'in the words of the HAL 9000, "human error".' Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule it
by scoot 14y ago
'in the words of the HAL 9000, "human error".'
Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule itself was in error. The human error was with the bug in the Juniper firmware that caused this rule to crash this router, and arguably with the CloudFlare process that allows rules to be propagated to all routers concurrently, rather than segmenting the network and testing for success before further deployment.
- graycat 14y ago"or the rule itself was in error" I thought that the rule was in error: I couldn't read the rule clearly on the screen, but it seemed, or I guessed, that the problem with the rule was that the "humans" omitted the decimal points and, thus, asked for blocking packets with lengths 1000 larger than intended. The Juniper software got sick, i.e., allocated too much memory, only because it was trying to swallow working with such absurdly large packet sizes. But, then, I couldn't clearly read the screen capture with the rule.
- scoot 14y agoThe rule matched the output of the profiler, so in that sense was correct. It wasn't clear from the article whether the profiler output was correct, a result of intentionally malformed packets, or otherwise. Regardless, the Juniper should either have rejected the rule, or accommodated it.
- imbriaco 14y agoIt's pretty clear that it was a failure of CloudFlare's profiler in generating an obviously impossible rule, the engineer who attempted to apply that rule, and Juniper for allowing it. There's no one place to lay blame, nor is it relevant. I would imagine that while we have seen a public reason for outage statement that there is a lot more work going on inside CloudFlare as far as the post-mortem is concerned. There are a lot of angles to cover here to really understand how the _system_ failed.
- eastdakota 14y agoYup.
- kisielk 14y agoActually this outage report is a good example of compounding systematic errors. A among the things that went wrong: incorrect and impossible packet sizes were detected, the rule generator generated rules matching the impossible packet sizes, the human operator who looked at the rules and entered them in to the router didn't notice any problems, and finally the routers responded to the incorrect rules by starting to crash. Had any of these steps not gone wrong there likely would not have been an outage. It was a combination of failures that caused it.
- scoot 14y agoI don't disagree with you, but I was calling in to question the suggestion that the creation of the rule was the specific human error. By definition, every error you listed is a human error, as even if ultimately carried out but computers (routers), they were designed by humans.