5 ms·

My feels go out to the ops folk at Cloudflare. Mistakes happen no matter how many years of experience people bring in, or how much they're paid. We're all human

by ultimoo 14y ago

My feels go out to the ops folk at Cloudflare. Mistakes happen no matter how many years of experience people bring in, or how much they're paid. We're all humans after all. It must be a pressurizing task to be responsible for potentially millions of dollars of losses during this downtime. I hope the issue is resolved soon and if a person caused it, they're not in too much trouble.

moot 14y ago
Amen.
- shanelja 14y ago
  I guess you're here because 4chan is down? Something I always wondered - do you use Cloudflare simply for the CDN in respects to photos, or does 4chan also frequently become the target of DDoS and other external attacks?
  - moot 14y ago
    We use CloudFlare for everything -- CDN, DDoS mitigation, Railgun (see below), et cetera. They serve ~1.5 petabytes of bandwidth on our behalf every month and proxy billions upon billions of requests. I'm a huge fan of the product and team, despite hiccups like this one. More on Railgun: http://arstechnica.com/information-technology/2013/02/cloudflare-blows-hole-in-laws-of-web-physics-with-go-and-railgun/ http://arstechnica.com/information-technology/2013/02/cloudf... Edit -- I am officially faster than carrier pigeon: me: was i the first human to notify you guys? | me: i caught it within the first 30-60 seconds i think | me: because i have no life and never sleep | CloudFlare Ops pal: yeah. you did. If anyone wants to hire me to check their site instead of Pingdom, feel free to ping me!
    - shanelja 14y ago
      It's expected that eventually things will go down, I've never once found a piece of software, server, person or traffic light which has performed at 100% for their entire lifetime, the best thing we can take from this however is response times. Barely an hour from outage to being live again - I barely had time to fire an email to my colleagues and update our Twitter before it came back online - however, what's worrying here is that there was no fallback in place whatsoever, we will be hearing from the Cloudflare team soon hopefully to let us know what went wrong, but if they are half the company they purport to be, they will learn a lot from whatever went wrong today.
    - shanelja 14y ago
      Regarding the Edit: How much would you charge to ping my site 24 hours per day? ^^
    - bowmessage 14y ago
      Even the creator of 4chan has to have his own 'first' moments once in a while :)
      - moot 14y ago
        What's the opposite of slowpoke.jpg?
  - deleted 14y ago
    [deleted]
laumars 14y ago
Human error is unacceptable. Online services should never have a single point of failure which can take their entire global presence offline. Not only because it minimizes the risk of catastrophic accidents, but also because hardware does occasionally break. This is why every piece of kid should be mirrored with a redundant backup and why many businesses even have entire duplicated standby systems for such disasters. Even if that gear cannot support the entire infrastructure, it's usually enough to at least publish an official status page. Having to use Twitter to update users is just amateurish in my opinion.
- vidarh 14y ago
  This response indicates lack of experience to me. No systems are immune to failure. No matter how much redundancy you have, chances are you have interdependencies you did not anticipate, and sooner or later run into failure scenarios that violate your expectations. It's very well possible that Cloudflare messed up here, but to claim so categorically that "human error is unacceptable" is a bit of a joke. We build systems to withstand the risks we know about, and guess at some we don't. But the number of possible failure scenarios we don't understand properly is pretty much infinite.
  - rdl 14y ago
    Losing DNS was fairly unforgivable. DNS as a protocol is designed to make it easy to deal with server and network outages (to the point of losing netblocks from the global routing tables). They added anycast DNS, which is great, but didn't split their DNS into multiple anycast netblocks.
    - vidarh 14y ago
      I run a number of DNS servers myself. And yes, it was "fairly unforgivable" to mess that part up. But what I was addressing was the blanket claim that human error is unacceptable. Anyone who runs a setup much larger than a calculator will deal with human errors - whether actual operational errors or human inability to engineer for resilience against all possible but unlikely scenarios - on a regular basis. Some things should be harder than others to break, and DNS are amongst them. Cloudflare no doubt have plenty of lessons to learn. But so have everyone else.
Corrado 14y ago
I would hope that they would treat this problem as an opportunity to improve their systems instead of haranguing an employee. All systems fail, the key to success is to turn that failure around and make sure it doesn't happen (the same way) again. If this was a human error, then change the process to make it difficult to do. If it was a software problem, then they have just found a new set of automated tests to write. If it was a vendor problem, contact the vendor to see how they plan to prevent this problem in the future. If it was a design problem, change the structure of the system to make this type of problem less likely. Quite a few big names have had similar outages in the past but the ones that I forgive (Google, Amazon) are the ones that talk openly about the issue and outline the changes that they are making to fix them. I'm looking forward to CloudFlare's explanation. BTW: My boss asked me this week for more information on Railgun and if we should change CDN providers. Railgun sounds incredible and I think it could really help our platform. How CloudFlare responds to this incident is critical to my decision to move forward with Railgun testing or to just forget it completely.

by ultimoo 14y ago

moot 14y ago

Amen.

shanelja 14y ago
I guess you're here because 4chan is down? Something I always wondered - do you use Cloudflare simply for the CDN in respects to photos, or does 4chan also frequently become the target of DDoS and other external attacks?
- moot 14y ago
  We use CloudFlare for everything -- CDN, DDoS mitigation, Railgun (see below), et cetera. They serve ~1.5 petabytes of bandwidth on our behalf every month and proxy billions upon billions of requests. I'm a huge fan of the product and team, despite hiccups like this one. More on Railgun: http://arstechnica.com/information-technology/2013/02/cloudflare-blows-hole-in-laws-of-web-physics-with-go-and-railgun/ http://arstechnica.com/information-technology/2013/02/cloudf... Edit -- I am officially faster than carrier pigeon: me: was i the first human to notify you guys? | me: i caught it within the first 30-60 seconds i think | me: because i have no life and never sleep | CloudFlare Ops pal: yeah. you did. If anyone wants to hire me to check their site instead of Pingdom, feel free to ping me!
  - shanelja 14y ago
    It's expected that eventually things will go down, I've never once found a piece of software, server, person or traffic light which has performed at 100% for their entire lifetime, the best thing we can take from this however is response times. Barely an hour from outage to being live again - I barely had time to fire an email to my colleagues and update our Twitter before it came back online - however, what's worrying here is that there was no fallback in place whatsoever, we will be hearing from the Cloudflare team soon hopefully to let us know what went wrong, but if they are half the company they purport to be, they will learn a lot from whatever went wrong today.
  - shanelja 14y ago
    Regarding the Edit: How much would you charge to ping my site 24 hours per day? ^^
  - bowmessage 14y ago
    Even the creator of 4chan has to have his own 'first' moments once in a while :)
    - moot 14y ago
      What's the opposite of slowpoke.jpg?
- deleted 14y ago
  [deleted]

laumars 14y ago

Human error is unacceptable. Online services should never have a single point of failure which can take their entire global presence offline. Not only because it minimizes the risk of catastrophic accidents, but also because hardware does occasionally break. This is why every piece of kid should be mirrored with a redundant backup and why many businesses even have entire duplicated standby systems for such disasters. Even if that gear cannot support the entire infrastructure, it's usually enough to at least publish an official status page. Having to use Twitter to update users is just amateurish in my opinion.

vidarh 14y ago
This response indicates lack of experience to me. No systems are immune to failure. No matter how much redundancy you have, chances are you have interdependencies you did not anticipate, and sooner or later run into failure scenarios that violate your expectations. It's very well possible that Cloudflare messed up here, but to claim so categorically that "human error is unacceptable" is a bit of a joke. We build systems to withstand the risks we know about, and guess at some we don't. But the number of possible failure scenarios we don't understand properly is pretty much infinite.
- rdl 14y ago
  Losing DNS was fairly unforgivable. DNS as a protocol is designed to make it easy to deal with server and network outages (to the point of losing netblocks from the global routing tables). They added anycast DNS, which is great, but didn't split their DNS into multiple anycast netblocks.
  - vidarh 14y ago
    I run a number of DNS servers myself. And yes, it was "fairly unforgivable" to mess that part up. But what I was addressing was the blanket claim that human error is unacceptable. Anyone who runs a setup much larger than a calculator will deal with human errors - whether actual operational errors or human inability to engineer for resilience against all possible but unlikely scenarios - on a regular basis. Some things should be harder than others to break, and DNS are amongst them. Cloudflare no doubt have plenty of lessons to learn. But so have everyone else.

Corrado 14y ago

I would hope that they would treat this problem as an opportunity to improve their systems instead of haranguing an employee. All systems fail, the key to success is to turn that failure around and make sure it doesn't happen (the same way) again. If this was a human error, then change the process to make it difficult to do. If it was a software problem, then they have just found a new set of automated tests to write. If it was a vendor problem, contact the vendor to see how they plan to prevent this problem in the future. If it was a design problem, change the structure of the system to make this type of problem less likely. Quite a few big names have had similar outages in the past but the ones that I forgive (Google, Amazon) are the ones that talk openly about the issue and outline the changes that they are making to fix them. I'm looking forward to CloudFlare's explanation. BTW: My boss asked me this week for more information on Railgun and if we should change CDN providers. Railgun sounds incredible and I think it could really help our platform. How CloudFlare responds to this incident is critical to my decision to move forward with Railgun testing or to just forget it completely.