4 ms·

To the couldflare folks; It's refreshing to see you take responsibility, but I think you've been a bit too hard on yourselves by taking all the blame. First of

by jcr 14y ago

To the couldflare folks; It's refreshing to see you take responsibility, but I think you've been a bit too hard on yourselves by taking all the blame. First of all, what you hit was a unknown bug in JunOS, and Juniper is to blame for their part. Using some form of staging to slow roll-out of rule changes might have saved you from a full meltdown, but when you're getting attacked, every second counts. Slow versus fast roll-out is one of those really tough balancing acts in your situation. You did a great job with it; by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again.

MichaelApproved 14y ago
"by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again." Not sure what your timeline shows but the "cloudflare is down" post hit the #1 spot on the front page just a few minutes after they went down. About 40 minutes after that, the services started to come back online for me. That's a significant outage. That's not reflecting on the job they did bringing things back online but your statement made it seem like a minor outage.
- druiid 14y ago
  Indeed. This was no few minute outage. There was very close to a full hour of zero traffic according to our MRTG.
  - gwern 14y ago
    Pingdom reported an hour downtime on my site too.
GigabyteCoin 14y ago
Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?
- eastdakota 14y ago
  Buck stops with us. We choose the hardware and software that runs on our network. We test and work around thousands of bugs in it. It was up to us to check range limits before applying them. While we'll never be perfect, one of the things I am most proud of with the CloudFlare team is how quickly we do learn from mistakes.
  - deleted 14y ago
    [deleted]
    - eastdakota 14y ago
      To be clear, while I wrote out the effective command that was pushed to the router, it was a script that generated it and the fault lies with the script not having adequate error handling.
      - deleted 14y ago
        [deleted]
  - cbsmith 14y ago
    There is more to this story than meets the eye. This had to be an IPv6 fragment attack. Why weren't you already advertising rejection of such packets, at least for DNS? Why would your analysis software and procedures not already be checking for memory problems with rules that would need yo assemble all the fragments before matching? Seems like there is more to this story than meets the eye.
- dubcanada 14y ago
  One would assume paying a company for an OS should be tested via the developers. Juniper SHOULD have tested that route scheme since they sell mission critical architecture. However you are most likely right, CloudFlare should have test ed it before rolling out.
  - packetslave 14y ago
deleted 14y ago
[deleted]

by jcr 14y ago

MichaelApproved 14y ago

"by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again." Not sure what your timeline shows but the "cloudflare is down" post hit the #1 spot on the front page just a few minutes after they went down. About 40 minutes after that, the services started to come back online for me. That's a significant outage. That's not reflecting on the job they did bringing things back online but your statement made it seem like a minor outage.

druiid 14y ago
Indeed. This was no few minute outage. There was very close to a full hour of zero traffic according to our MRTG.
- gwern 14y ago
  Pingdom reported an hour downtime on my site too.

GigabyteCoin 14y ago

Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?

eastdakota 14y ago
Buck stops with us. We choose the hardware and software that runs on our network. We test and work around thousands of bugs in it. It was up to us to check range limits before applying them. While we'll never be perfect, one of the things I am most proud of with the CloudFlare team is how quickly we do learn from mistakes.
- deleted 14y ago
  [deleted]
  - eastdakota 14y ago
    To be clear, while I wrote out the effective command that was pushed to the router, it was a script that generated it and the fault lies with the script not having adequate error handling.
    - deleted 14y ago
      [deleted]
- cbsmith 14y ago
  There is more to this story than meets the eye. This had to be an IPv6 fragment attack. Why weren't you already advertising rejection of such packets, at least for DNS? Why would your analysis software and procedures not already be checking for memory problems with rules that would need yo assemble all the fragments before matching? Seems like there is more to this story than meets the eye.
dubcanada 14y ago
One would assume paying a company for an OS should be tested via the developers. Juniper SHOULD have tested that route scheme since they sell mission critical architecture. However you are most likely right, CloudFlare should have test ed it before rolling out.
- packetslave 14y ago

deleted 14y ago

[deleted]