5 ms·

I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke th

I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.

csabakissi 5y ago
I can't actually imagine something like that can happen. Single person with a simple change in a config can cause this.
- hulitu 5y ago
  They forgot to test it.
  - 2rsf 5y ago
    Testing is never complete nor can it be theoretically complete
  - alkonaut 5y ago
    Test “it”? The change in question wasn’t by fastly but a customer of theirs making a config change. It’s possible that this customer did validate their change somehow. Fastly obviously didn’t test their code (with the bug) enough, but testing of course can never prove the absence of bugs. Testing for a global deployment like a massive CDN happens to a large extent in prod because you don’t have another globe. You can test on a smaller scale but eventually you run into a problem that only shows itself at full scale.
    - that_guy_iain 5y ago
      Fastly even say it was a valid change. > We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. in the first sentence
      - alkonaut 5y ago
        Their change was bad, that was May 12. Since that seemed OK on May 13,14,… there wasn’t much indicating that change would blow up weeks later. For example if they roll it out gradually, they would reach 100% rollout with all lights being green The customer change was a valid configuration. That was yesterday.
- KirillPanov 5y ago
  Welcome to the CentralizedWeb (tm). Unfortunately we had to sunset the Internet you used to use.
  - ianlevesque 5y ago
    A web filled with DDOS attacks and scraping is a web that needs cloudflare and fastly. I’m not sure how to avoid this sorry state of things.
    - KirillPanov 5y ago
      This attitude is why we have only 2½ search engines on the entire Internet. Only Google, Bing, and Yandex run crawlers. Everybody else is just a reseller for them. Web crawlers are a feature not a bug. If your site shouldn't be crawled, it doesn't belong on the Internet.
      - motoboi 5y ago
        Search engines scrapping your content is not the problem. Competitors scraping your content is.
        KirillPanov 5y ago
        If you don't want your content crawled, don't put it on the public Internet. Your profitability is not the Internet's problem.
        motoboi 5y ago
        It’s a very interesting proposition. If you cannot generate revenue by your internet content, probably you can’t live from generating content for the internet. The consequence, IMHO, is that the internet would have this amount of content and usefulness. Newspapers? No. Can’t live from internet news if anyone can copy a reporter’s work and post it on his own site and dilute traffic. Online selling? Don’t look like a viable business model, as anyone can copy the photos you paid a photographer for, the descriptions you paid someone to write and the reviews your customers wrote. True reviews are priceless, you now? Even more now that an AI can detect computer generated reviews. Obviously an open and totally money-free internet is nice, but it wouldn’t be the internet people make a living from.
- alkonaut 5y ago
  That was the bug. A trivial example would be a bug that replaces the configuration for all customers with the last uploaded. Then when the next customer uploads a new (valid!) config, you have a problem. Obviously it wasn’t that trivial but the point is: it wasn’t the customer’s configuration change that was the problem but some code that managed the config change.
- nirvanis 5y ago
  It's more common than we imagine. That's usually the start of many of the historical network incidents. The important part, as usual, is to make sure the remediations of such incidents focus on how to limit blast radius of small changes, and how to accomplish that without imposing artificial gatekeeping and bureaucracy into the change process.
Santosh83 5y ago
Isn't it a shame and a cause for alarm that the supposed decentralised, damage resistant Internet has been reduced to this?
- arbitrage 5y ago
  damage resistance isn't profitable.
Cthulhu_ 5y ago
Really weird case of infinite number of monkeys. I wonder if fuzz testing would have found this one.
pdonis 5y ago
> broke the entire internet Breaking 85% of Fastly's servers is not the same as breaking the entire internet.
- pimlottc 5y ago
  GP said "/basically/ broke the entire internet", clearly they were using rhetorical hyperbole.
Jakobeha 5y ago
Sometimes when I access a webpage I think to myself "what if I do X and the whole website crashes?" Or if the webpage crashes I think "heh, what if I brought down the whole server?" It looks like this guy did just that. And for Fastly. Wow. "So, did I just hear three distinct light switch clicks?"

csabakissi 5y ago

I can't actually imagine something like that can happen. Single person with a simple change in a config can cause this.

hulitu 5y ago
They forgot to test it.
- 2rsf 5y ago
  Testing is never complete nor can it be theoretically complete
- alkonaut 5y ago
  Test “it”? The change in question wasn’t by fastly but a customer of theirs making a config change. It’s possible that this customer did validate their change somehow. Fastly obviously didn’t test their code (with the bug) enough, but testing of course can never prove the absence of bugs. Testing for a global deployment like a massive CDN happens to a large extent in prod because you don’t have another globe. You can test on a smaller scale but eventually you run into a problem that only shows itself at full scale.
  - that_guy_iain 5y ago
    Fastly even say it was a valid change. > We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. in the first sentence
    - alkonaut 5y ago
      Their change was bad, that was May 12. Since that seemed OK on May 13,14,… there wasn’t much indicating that change would blow up weeks later. For example if they roll it out gradually, they would reach 100% rollout with all lights being green The customer change was a valid configuration. That was yesterday.
KirillPanov 5y ago
Welcome to the CentralizedWeb (tm). Unfortunately we had to sunset the Internet you used to use.
- ianlevesque 5y ago
  A web filled with DDOS attacks and scraping is a web that needs cloudflare and fastly. I’m not sure how to avoid this sorry state of things.
  - KirillPanov 5y ago
    This attitude is why we have only 2½ search engines on the entire Internet. Only Google, Bing, and Yandex run crawlers. Everybody else is just a reseller for them. Web crawlers are a feature not a bug. If your site shouldn't be crawled, it doesn't belong on the Internet.
    - motoboi 5y ago
      Search engines scrapping your content is not the problem. Competitors scraping your content is.
      - KirillPanov 5y ago
        If you don't want your content crawled, don't put it on the public Internet. Your profitability is not the Internet's problem.
        motoboi 5y ago
        It’s a very interesting proposition. If you cannot generate revenue by your internet content, probably you can’t live from generating content for the internet. The consequence, IMHO, is that the internet would have this amount of content and usefulness. Newspapers? No. Can’t live from internet news if anyone can copy a reporter’s work and post it on his own site and dilute traffic. Online selling? Don’t look like a viable business model, as anyone can copy the photos you paid a photographer for, the descriptions you paid someone to write and the reviews your customers wrote. True reviews are priceless, you now? Even more now that an AI can detect computer generated reviews. Obviously an open and totally money-free internet is nice, but it wouldn’t be the internet people make a living from.
alkonaut 5y ago
That was the bug. A trivial example would be a bug that replaces the configuration for all customers with the last uploaded. Then when the next customer uploads a new (valid!) config, you have a problem. Obviously it wasn’t that trivial but the point is: it wasn’t the customer’s configuration change that was the problem but some code that managed the config change.
nirvanis 5y ago
It's more common than we imagine. That's usually the start of many of the historical network incidents. The important part, as usual, is to make sure the remediations of such incidents focus on how to limit blast radius of small changes, and how to accomplish that without imposing artificial gatekeeping and bureaucracy into the change process.

Santosh83 5y ago

Isn't it a shame and a cause for alarm that the supposed decentralised, damage resistant Internet has been reduced to this?

arbitrage 5y ago
damage resistance isn't profitable.

Cthulhu_ 5y ago

Really weird case of infinite number of monkeys. I wonder if fuzz testing would have found this one.

pdonis 5y ago

> broke the entire internet Breaking 85% of Fastly's servers is not the same as breaking the entire internet.

pimlottc 5y ago
GP said "/basically/ broke the entire internet", clearly they were using rhetorical hyperbole.

Jakobeha 5y ago

Sometimes when I access a webpage I think to myself "what if I do X and the whole website crashes?" Or if the webpage crashes I think "heh, what if I brought down the whole server?" It looks like this guy did just that. And for Fastly. Wow. "So, did I just hear three distinct light switch clicks?"