12 ms·

One Fastly customer triggered internet meltdown

287 points by JulianMorrison 5y ago

geerlingguy 5y ago
Not in the article: how one Fastly customer triggered the bug. Just a quote and a promise that an RCA will be posted.
- tyingq 5y ago
  Fastly doesn't appear to be sharing that detail. Their own blog post is similarly vague about the exact cause. https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Edit: That blog post does say this: "On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances." The scheduled maintenance on May 12 was this: https://status.fastly.com/incidents/dlsphjqst537 https://status.fastly.com/incidents/dlsphjqst537 Based on that, it sounds like maybe a configuration change could deploy new cache nodes with ip addresses that a customer hasn't explicitly allowed to talk to their backend: "When this change is applied, customers may observe additional origin traffic as new cache nodes retrieve content from origin. Please be sure to check that your origin access lists allow the full range of Fastly IP addresses"
  - abluecloud 5y ago
    Why is everyone banging on about this. It's a blog post from the same day, a decent post mortem takes a while to put together and assuming the bug isn't fully patched across their entire CDN, why would they post the information.
    - tyingq 5y ago
      I wasn't "banging on", I was answering why the article didn't mention the cause...because the source didn't either.
    - notacoward 5y ago
      Everyone is "banging on" because there are important lessons to be learned from such incidents, and people want to learn. They hunger for more details about the generalizable aspects of the bug, even if a full post mortem that also covers internal processes etc. might take longer to do. Having participated in many post mortems, in many roles, for systems just as complex, I believe it's entirely possible to provide that information the next (not same) day. Is that still setting the bar too high? Perhaps. Fastly deserves kudos for providing even the level of information that they have, since that's already above the pathetic industry standard, but I don't think there's anything bad about wanting more. Defensiveness is the enemy of effective post mortems.
    - jffry 5y ago
      Plus it would be weird to present just that specific information, outside of the context of a post mortem / failure chain analysis type discussion.
      - tyingq 5y ago
        That's true, though they are also saying things like "We created a permanent fix for the bug and began deploying it at 17:25.". "Permanent fix" sort of implies they understood the issue really well.
        jffry 5y ago
        That's my point though. Even though they may understand the immediate flaw in their code that caused the issue, there's not much use (for them or their customers) in just talking in detail about that specific flaw. I'd go so far as to argue that the specifics of the flaw are immaterial right now. At this stage, the important thing is that they have identified a specific code change that was the proximate cause of the issue, and have a mitigation in place. This is contrasted with more mysterious and hard-to-track-down failures. ("We are working to understand why our systems are down and will post another update in 30 minutes") What will take time, and the thing which will be interesting, is failure tree analysis. (You might hear the phrase "failure chain" or "root cause" but IMO it's quite rare for things to be so linear). That can help identify opportunities to improve processes at many different levels of the product lifecycle. Humans are fallible, and there's no way we can write bug-free software, so the solution has to be more robust than "hope that every member of our organization never makes a mistake again"
        tyingq 5y ago
        Yes, I was saying I would have avoided words like "permanent fix", because it sets unrealistic expectations.
  - addingnumbers 5y ago
    The paragraph you quoted is just describing a side effect of adding nodes. That excerpt appears in every one of their capacity expansion announcements, going back years.
    - tyingq 5y ago
      Ah, interesting. Though the change itself reads like it was just adding capacity in one location (Newark). I don't see any other changes mentioned for that date.
- Dobbs 5y ago
  My uneducated hypothesis is that Fastly runs varnish. Presumably they have some process that collects data from their config system generates the VCL (varnishes psuedo-C configuration language which compiles directly to C). Somehow a customer configured something in such a way that it generated a bad VCL file which then caused it to lose all configuration, or caused one domain to incorrectly garble up traffic for all domains. I can poke plenty of holes in this hypothesis, like fastly likely not deploying configuration to all nodes but only subsets. Looking forward to the deeper post.
  - tyingq 5y ago
    It's interesting to me that they still run Varnish, since it doesn't have https/tls built-in. I do get that VCL is more expressive than similar capabilities in Nginx, HAproxy, etc. But it would seem like less work to add expressiveness to one of those (via Lua maybe?) than to maintain both Varnish and the separate components needed for both tls ingress and egress.
    - foobarbazetc 5y ago
      Varnish and VCL were the hotness at the time Fastly was coming up so it sort of makes sense, but Varnish also doesn’t support WebSockets, can’t proxy GRPC, etc so they’re very limited in functionality vs CloudFlare. I doubt they’d build it on Varnish today, but it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) and will have to support that for eternity. They can run two or more serving stacks side by side though if it comes to that. And to add to your point, they also have a separate process that speaks QUIC. It’s an interesting tech stack with a lot of technical debt.
      - tyingq 5y ago
        >it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) Ah, okay. I took a look, and it appears they at least didn't allow varnish modules or inline C. But, still, a fairly hefty anchor for the future.
      - 5y ago
- jedimastert 5y ago
  I assume that the bug isn't 100% fixed yet and the instant they publish how the bug took place 1000 yahoos will immediately try to re-create it.
- staticassertion 5y ago
  If you find out you have a DOS that can take down the internet you might be wary about sharing details until you've hammered things out.
KingOfCoders 5y ago
"Fastly senior engineering executive Nick Rockwell said: "This outage was broad and severe - and we're truly sorry for the impact to our customers and everyone who relies on them."" I wonder here, are they sorry enough to run the company on two different tech and software stacks and data centers? Like people don't buy disk drives (SSDs) not all from the same vendor? How much would they spend for the "sorry".
- mhandley 5y ago
  You'd have to consider what actually happens when 50% of your infrastructure goes down. Can the remaining 50% cope with 100% of the load? If not, then you still get a complete failure. So then the question becomes can you reprovision between stack A and stack B very rapidly, both running from the same hardware pool, while 50% of your infrastructure is down. Now you've introduced the potential for correlated failures (single hardware pool and network), plus added complexity and load due to reprovisioning just when things are already overloaded. Not easy to get this right, so might not actually increase reliability, as now you've got two different stacks that can each independently fail and get you into this mess.
- ceejayoz 5y ago
  Without more details on what/where the "software bug" was, it's hard to say if that would've helped at all.
- stingraycharles 5y ago
  That only protects against one type of failure, eg one of your vendors going down. The article suggests this was a problem in their own software, which in all likelihood would not be protected against by using different stacks.
  - KingOfCoders 5y ago
    Two developers and two product managers would not create the same bugs but different ones I'd think. From writing thousands of correct lines and implementing features without bugs, it would be a coincidende (except perhaps not well understood complex parts) where two developers create the same bugs. Reading studies about bugs in the past where developers where told to write some code resulted in different bugs.
Simplicitas 5y ago
Still impressed at how quickly Fastly recovers. Kudos.
- ic4l 5y ago
  Even if they fixed it in 20 minutes the chain reaction caused by Fastly being down took much more than 20 minutes to resolve itself. An example is imgix: https://status.imgix.com https://status.imgix.com It took them 11 hours to recover from Fastly going down for their claimed 40 minutes.
  - busymom0 5y ago
    > We are still seeing increased origin load as a result of the earlier outage from our service provider. Does this mean that some companies using Fastly could have major costs because of the increased origin load?
    - axlee 5y ago
      Absolutely.
rlv-dan 5y ago
https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage
m3kw9 5y ago
I remember there was a similar bug that hit Cloud Flare before??
Dah00n 5y ago
Again, The Cloud is just someone else's server. Most things in The Cloud doesn't belong there in my opinion but it is yet another trend everyone must follow and check off on their list from management. Let's all chant in unison and play out the rituals sent down from above. Umbasa!
- gilbertbw 5y ago
  Unless you are a very large company a CDN is not something you can realistically build and run in house.
  - rocqua 5y ago
    If you are not a very large company, you might not need the full functionality of a very large CDN either. Now, building the functionality you require can still be unrealistic to build yourself. One thing that springs to mind is DDoS protection.
    - doublerabbit 5y ago
      Colocation is cheap. But your correct with DDoS; that is the only thing I am not prepared for. I will forever go with colocation. 1Gbit is nothing nowadays and can be saturated in seconds; my purposes do not justify the cost of 10g transit. Even owning 4U when I only need a VPS is overkill. But I like owning a small dusty cube of internet. So there's that.
    - aliasEli 5y ago
      When your site can expect visitors from all around the world you need a CDN that also has a world wide presence. Using a CDN can have a pretty big impact on the performance of your site, and it is well known that users avoid sites that are slow.
- ceejayoz 5y ago
  > Again, The Cloud is just someone else's server. Sure, but in a lot of cases, that "someone else" is an entire team of experts in their particular niche that can do a better job of the specific task at hand than I ever can hope to. Is this always the case? No. Is it sometimes the case? Yes.
napolux 5y ago
cat /users/*.conf >> global.conf && ./restart_everything
pentagone 5y ago
> The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet. Big surprise
paul_f 5y ago
I love how it almost suggests it was a customer's fault. If only they hadn't changed their settings!
- deleted 5y ago
  [deleted]
- politelemon 5y ago
  An edge case triggered an edge condition at our edge location.
  - Frost1x 5y ago
    "Edge cases" and "conditions" are some of my trigger phrases. Given enough time or users, they are inevitable and passing them off as rare to avoid dealing with them, especially when you're aware of their existence, drives me up the wall. Unless it's an edge use case you're not supporting, don't sell me your cost avoidance on any production systems.
    - bennyp101 5y ago
      And edge case is just that though? Something that hadn't been thought of, and makes stuff break, then you can figure out a fix. I don't know any dev that would agree with "passing them off as rare to avoid dealing with them" - rather, "it's a low priority, but it needs fixing" or "ok, this is an edge case, but holy crap its a bad one"
  - mjthompson 5y ago
    A very edgy comment! Well done.
    - smlss_sftwr 5y ago
      quite edge-ucational I must say
- iainmerrick 5y ago
jcims 5y ago
Just an unsolicited plug of an interesting podcast I started listening to recently (co-hosted by HNer slackerIII)- https://downtimeproject.com/ https://downtimeproject.com/ Hope this one gets the treatment.
- slackerIII 5y ago
  I'm not sure there is quite enough detail in this post to make a whole episode, but if any more details about the bug come out we will definitely cover it.
  - jcims 5y ago
    Awesome! I think my plug was a little forward for hump day on HN lol. Love the podcast.
bluedevil2k 5y ago
Why is Amazon dependent on someone else’s cloud infrastructure?
- busymom0 5y ago
  Someone answered this yesterday. CloudFront is good for video and large download assets (plus very low margins) but not for images and smaller stuff which Fastly is much faster at: https://www.streamingmediablog.com/2020/05/fastly-amazon-homepage.html https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...
  - deleted 5y ago
    [deleted]
- tyingq 5y ago
  I believe Google's Firebase also uses Fastly.
- Thaxll 5y ago
  Also large compagnies uses multiple CDN.
- typon 5y ago
  Hedging your bets
- rchaud 5y ago
  They can't afford Cloudfront bills either.
flareback 5y ago
fta: "The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet." I read that phrase everytime something like this happens and yet we all still rely on the same handful of companies.
- axiosgunnar 5y ago
  Would a smaller CDN provider have no outages at all?
  - sgtfrankieboy 5y ago
    No, but it would take down less sites if it does have one.
    - gpm 5y ago
      Is that a feature? If we have 12 small cdns have 12 outages in a year (combined, 1/year each), each time bringing down 10,000 websites, is that better than 1 large cdn having one outage during the year bringing down 120,000 websites? If I'm the website owner I think I prefer the latter, my customers blame the cdn instead of me. If I'm the cdn owner I definitely prefer the latter, more customers to amortize my costs over.
      - iso1210 5y ago
        From a customer point of view diversity is far better it's great - if Sainsburys is closed for some reason I'll go to Tesco. Certainly don't want a situation where all the shops are closed.
  - martius 5y ago
    It may be more acceptable to have more frequent outages if the impact radius (number of websites or services impacted) is smaller.
  - zelon88 5y ago
    Obviously that isn't an honest comparison. If you're asking whether or not 10 small CDN providers could provide a more robust, higher quality service with more uptime than one large CDN provider, then I think the answer is probably "yes."
mikesabbagh 5y ago
the thing with CDN is that they may have many edge locations, but the cache does not sit there. They frequently have a common caching servers located close by. so maybe every 10 or 100 edge locations have a single cache location. Edge is a reverse proxy and probably handles ssl handshakes. So if your cache is down, all your edge locations in that area are down
- uncertainrhymes 5y ago
  This may be true with some, but it is not true of Fastly. Each of their edge nodes is a varnish cache. Because they are multi-tenant, when varnish crashes it crashes hard and takes everyone with it. The question in various threads is why not have redundancy -- but the point of a CDN is to have extra servers and capacity and lots of locations to make individual crashes just flow elsewhere. But if the single customer with a valid-yet-crashable config had lots of traffic all over the world... it'll take everything out at once. Redundancy of CDN is more expensive, and still requires DNS failover. People do the calculation and usually decide that 30 min of downtime every couple years is worth the saving on vendors and code and hassle. They don't like it, but every site that was down made that decision.
arrty88 5y ago
Did they just miss a where clause on the update statement??
iainmerrick 5y ago
Throwing in my positive hot take among all the negative ones here: the immediate response and blog post from Fastly here is really good. A quick fix, a clear apology, enough detail to give an idea of what happened, but not so much detail that there might be a mistake they’ll have to clarify or retract. What more are you looking for? Apart from “not have the bug in the first place” -- and I hope and expect they’ll go into more detail later when they’ve had time for a proper post mortem -- I’d be interested to hear what anyone thinks they could have done better in terms of their immediate firefighting.
- oneeyedpigeon 5y ago
  I'm sorry, but I disagree. They gave the BBC enough detail that a very misleading headline was produced as a result. True, the main blame lies with the BBC, but it also comes across — to me, anyway, maybe I'm being too cynical — as a bit of an excuse from Fastly.
  - mytailorisrich 5y ago
    "But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing "85% of our network to return errors" They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.
    - oneeyedpigeon 5y ago
      I know — as I've said, the main blame lies with the bbc. However, as it's reported, it comes across very much as Fastly trying to save face. Maybe the blame is entirely on the bbc, maybe Fastly were naive in thinking that giving them this information wouldn't result in irresponsible headlines.
      - Jenk 5y ago
        How on earth do you figure that is anyone's fault but the BBC? Read Fastly's statement. There is nothing about it blaming the customer(s) at all. There is nothing trying to save face. What is your point here?
CalChris 5y ago
The BBC title is a terrible title. The customer didn’t cause the outage. The bug caused the outage.
- underscore_ku 5y ago
  this and the BBC bots that upvote every BBC news
- mypastself 5y ago
  Unless it’s been changed since your comment, I don’t think that’s what the title is claiming. A trigger is different from a cause.
- Zenst 5y ago
  That was my take as well and very much disenfranchises what happens with title's like that. If you have a design flaw, you don't hint at onus upon the first person to fall foul of it - and we all know how quick many are reading news that they will run fully upon the title alone (something we can all do). But we have seen a drive towards click-bait/search-bot friendly to garner hits - style headlines. Even the BBC over the years have IMHO learnt towards such tabloid style headlines more and that is just sad.
- tantalor 5y ago
  The bug is the root cause, the customer's action is the proximal cause (or "trigger")
  - hnbad 5y ago
    Placing the emphasis on "one customer" still creates a false narrative though. Note the headline doesn't even say it was a bug. This headline provokes the question "Who was it and what did they do?" rather than the more insightful "What was the bug?". If a single bug caused the "internet meltdown", it's fairly likely that the bug was triggered by one person so there's no need to emphasize that part.
    - tantalor 5y ago
oneeyedpigeon 5y ago
Shame on the BBC for such a misleading headline. But Fastly probably shouldn't have even given them detail — it's irrelevant and bound to be misreported. Just own up to your bug and be done with it.
koreanguy 5y ago
single point failure, you are what you run
desireco42 5y ago
Them pointing to customer as a source of issue is not OK. This reminds me of that Chase scandal, when they transferred huge amount of money to pay of full loan instead of installment, then they blamed on some guy in India for alleged mistake, where his 2 or 3 superiors that approved that and really horrific interface, but they decided to pin the blame on that guy. So, I am glad things are OK, but it definitely is not that one customer who is to blame for this outage.
- zomglings 5y ago
  I don't think that Fastly is pinning blame on the customer. That seems to be the BBC trying to bait us into reading the article. I am sad to report that their clickbait worked on me.
  - desireco42 5y ago
    I think you are right, it is my bad.
kevincox 5y ago
Global shared control plane updates are frighteningly common. For example Cloudflare regularly brags about how your configuration updates are pushed around the world in single-digit seconds. Sure, it is an amazing feature, but it opens you up to this sort of issue. All changes to critical infrastructure should be a gradual rollout (emergencies aside). Instant sounds nice, until it isn't. If this rolled out to one region for the first hour it likely would have been caught and Fastly could press the "stop all rollouts" button.
kissgyorgy 5y ago
So they have an infrastructure where customer data is not separated from other customers, that's pretty terrifying! This means another bug of the same kind can also cause a global disruption any time in the future (mistakes will happen).
- aliasEli 5y ago
  Yes, that might happen. But the only way to avoid that is to give each customer its own private hardware, which seems prohibitively expensive (and may not even prevent all failure sources).
NaturalPhallacy 5y ago
Modal free: https://archive.is/dKMHE https://archive.is/dKMHE
dang 5y ago
See also: Summary of June 8 outage - https://news.ycombinator.com/item?id=27444005 https://news.ycombinator.com/item?id=27444005 - June 2021 (ongoing)
alboaie 5y ago
it remembers ue about the inevitable tension between eficiency and resilience...

Not in the article: how one Fastly customer triggered the bug. Just a quote and a promise that an RCA will be posted.

tyingq 5y ago
Fastly doesn't appear to be sharing that detail. Their own blog post is similarly vague about the exact cause. https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Edit: That blog post does say this: "On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances." The scheduled maintenance on May 12 was this: https://status.fastly.com/incidents/dlsphjqst537 https://status.fastly.com/incidents/dlsphjqst537 Based on that, it sounds like maybe a configuration change could deploy new cache nodes with ip addresses that a customer hasn't explicitly allowed to talk to their backend: "When this change is applied, customers may observe additional origin traffic as new cache nodes retrieve content from origin. Please be sure to check that your origin access lists allow the full range of Fastly IP addresses"
- abluecloud 5y ago
  Why is everyone banging on about this. It's a blog post from the same day, a decent post mortem takes a while to put together and assuming the bug isn't fully patched across their entire CDN, why would they post the information.
  - tyingq 5y ago
    I wasn't "banging on", I was answering why the article didn't mention the cause...because the source didn't either.
  - notacoward 5y ago
    Everyone is "banging on" because there are important lessons to be learned from such incidents, and people want to learn. They hunger for more details about the generalizable aspects of the bug, even if a full post mortem that also covers internal processes etc. might take longer to do. Having participated in many post mortems, in many roles, for systems just as complex, I believe it's entirely possible to provide that information the next (not same) day. Is that still setting the bar too high? Perhaps. Fastly deserves kudos for providing even the level of information that they have, since that's already above the pathetic industry standard, but I don't think there's anything bad about wanting more. Defensiveness is the enemy of effective post mortems.
  - jffry 5y ago
    Plus it would be weird to present just that specific information, outside of the context of a post mortem / failure chain analysis type discussion.
    - tyingq 5y ago
      That's true, though they are also saying things like "We created a permanent fix for the bug and began deploying it at 17:25.". "Permanent fix" sort of implies they understood the issue really well.
      - jffry 5y ago
        That's my point though. Even though they may understand the immediate flaw in their code that caused the issue, there's not much use (for them or their customers) in just talking in detail about that specific flaw. I'd go so far as to argue that the specifics of the flaw are immaterial right now. At this stage, the important thing is that they have identified a specific code change that was the proximate cause of the issue, and have a mitigation in place. This is contrasted with more mysterious and hard-to-track-down failures. ("We are working to understand why our systems are down and will post another update in 30 minutes") What will take time, and the thing which will be interesting, is failure tree analysis. (You might hear the phrase "failure chain" or "root cause" but IMO it's quite rare for things to be so linear). That can help identify opportunities to improve processes at many different levels of the product lifecycle. Humans are fallible, and there's no way we can write bug-free software, so the solution has to be more robust than "hope that every member of our organization never makes a mistake again"
        tyingq 5y ago
        Yes, I was saying I would have avoided words like "permanent fix", because it sets unrealistic expectations.
- addingnumbers 5y ago
  The paragraph you quoted is just describing a side effect of adding nodes. That excerpt appears in every one of their capacity expansion announcements, going back years.
  - tyingq 5y ago
    Ah, interesting. Though the change itself reads like it was just adding capacity in one location (Newark). I don't see any other changes mentioned for that date.
Dobbs 5y ago
My uneducated hypothesis is that Fastly runs varnish. Presumably they have some process that collects data from their config system generates the VCL (varnishes psuedo-C configuration language which compiles directly to C). Somehow a customer configured something in such a way that it generated a bad VCL file which then caused it to lose all configuration, or caused one domain to incorrectly garble up traffic for all domains. I can poke plenty of holes in this hypothesis, like fastly likely not deploying configuration to all nodes but only subsets. Looking forward to the deeper post.
- tyingq 5y ago
  It's interesting to me that they still run Varnish, since it doesn't have https/tls built-in. I do get that VCL is more expressive than similar capabilities in Nginx, HAproxy, etc. But it would seem like less work to add expressiveness to one of those (via Lua maybe?) than to maintain both Varnish and the separate components needed for both tls ingress and egress.
  - foobarbazetc 5y ago
    Varnish and VCL were the hotness at the time Fastly was coming up so it sort of makes sense, but Varnish also doesn’t support WebSockets, can’t proxy GRPC, etc so they’re very limited in functionality vs CloudFlare. I doubt they’d build it on Varnish today, but it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) and will have to support that for eternity. They can run two or more serving stacks side by side though if it comes to that. And to add to your point, they also have a separate process that speaks QUIC. It’s an interesting tech stack with a lot of technical debt.
    - tyingq 5y ago
      >it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) Ah, okay. I took a look, and it appears they at least didn't allow varnish modules or inline C. But, still, a fairly hefty anchor for the future.
    - 5y ago
jedimastert 5y ago
I assume that the bug isn't 100% fixed yet and the instant they publish how the bug took place 1000 yahoos will immediately try to re-create it.
staticassertion 5y ago
If you find out you have a DOS that can take down the internet you might be wary about sharing details until you've hammered things out.

"Fastly senior engineering executive Nick Rockwell said: "This outage was broad and severe - and we're truly sorry for the impact to our customers and everyone who relies on them."" I wonder here, are they sorry enough to run the company on two different tech and software stacks and data centers? Like people don't buy disk drives (SSDs) not all from the same vendor? How much would they spend for the "sorry".

mhandley 5y ago
You'd have to consider what actually happens when 50% of your infrastructure goes down. Can the remaining 50% cope with 100% of the load? If not, then you still get a complete failure. So then the question becomes can you reprovision between stack A and stack B very rapidly, both running from the same hardware pool, while 50% of your infrastructure is down. Now you've introduced the potential for correlated failures (single hardware pool and network), plus added complexity and load due to reprovisioning just when things are already overloaded. Not easy to get this right, so might not actually increase reliability, as now you've got two different stacks that can each independently fail and get you into this mess.
ceejayoz 5y ago
Without more details on what/where the "software bug" was, it's hard to say if that would've helped at all.
stingraycharles 5y ago
That only protects against one type of failure, eg one of your vendors going down. The article suggests this was a problem in their own software, which in all likelihood would not be protected against by using different stacks.
- KingOfCoders 5y ago
  Two developers and two product managers would not create the same bugs but different ones I'd think. From writing thousands of correct lines and implementing features without bugs, it would be a coincidende (except perhaps not well understood complex parts) where two developers create the same bugs. Reading studies about bugs in the past where developers where told to write some code resulted in different bugs.

Simplicitas 5y ago

Still impressed at how quickly Fastly recovers. Kudos.

ic4l 5y ago
Even if they fixed it in 20 minutes the chain reaction caused by Fastly being down took much more than 20 minutes to resolve itself. An example is imgix: https://status.imgix.com https://status.imgix.com It took them 11 hours to recover from Fastly going down for their claimed 40 minutes.
- busymom0 5y ago
  > We are still seeing increased origin load as a result of the earlier outage from our service provider. Does this mean that some companies using Fastly could have major costs because of the increased origin load?
  - axlee 5y ago
    Absolutely.

rlv-dan 5y ago

https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage

m3kw9 5y ago

I remember there was a similar bug that hit Cloud Flare before??

Dah00n 5y ago

Again, The Cloud is just someone else's server. Most things in The Cloud doesn't belong there in my opinion but it is yet another trend everyone must follow and check off on their list from management. Let's all chant in unison and play out the rituals sent down from above. Umbasa!

gilbertbw 5y ago
Unless you are a very large company a CDN is not something you can realistically build and run in house.
- rocqua 5y ago
  If you are not a very large company, you might not need the full functionality of a very large CDN either. Now, building the functionality you require can still be unrealistic to build yourself. One thing that springs to mind is DDoS protection.
  - doublerabbit 5y ago
    Colocation is cheap. But your correct with DDoS; that is the only thing I am not prepared for. I will forever go with colocation. 1Gbit is nothing nowadays and can be saturated in seconds; my purposes do not justify the cost of 10g transit. Even owning 4U when I only need a VPS is overkill. But I like owning a small dusty cube of internet. So there's that.
  - aliasEli 5y ago
    When your site can expect visitors from all around the world you need a CDN that also has a world wide presence. Using a CDN can have a pretty big impact on the performance of your site, and it is well known that users avoid sites that are slow.
ceejayoz 5y ago
> Again, The Cloud is just someone else's server. Sure, but in a lot of cases, that "someone else" is an entire team of experts in their particular niche that can do a better job of the specific task at hand than I ever can hope to. Is this always the case? No. Is it sometimes the case? Yes.

napolux 5y ago

cat /users/*.conf >> global.conf && ./restart_everything

pentagone 5y ago

> The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet. Big surprise

paul_f 5y ago

I love how it almost suggests it was a customer's fault. If only they hadn't changed their settings!

deleted 5y ago
[deleted]
politelemon 5y ago
An edge case triggered an edge condition at our edge location.
- Frost1x 5y ago
  "Edge cases" and "conditions" are some of my trigger phrases. Given enough time or users, they are inevitable and passing them off as rare to avoid dealing with them, especially when you're aware of their existence, drives me up the wall. Unless it's an edge use case you're not supporting, don't sell me your cost avoidance on any production systems.
  - bennyp101 5y ago
    And edge case is just that though? Something that hadn't been thought of, and makes stuff break, then you can figure out a fix. I don't know any dev that would agree with "passing them off as rare to avoid dealing with them" - rather, "it's a low priority, but it needs fixing" or "ok, this is an edge case, but holy crap its a bad one"
- mjthompson 5y ago
  A very edgy comment! Well done.
  - smlss_sftwr 5y ago
    quite edge-ucational I must say
iainmerrick 5y ago

jcims 5y ago

Just an unsolicited plug of an interesting podcast I started listening to recently (co-hosted by HNer slackerIII)- https://downtimeproject.com/ https://downtimeproject.com/ Hope this one gets the treatment.

slackerIII 5y ago
I'm not sure there is quite enough detail in this post to make a whole episode, but if any more details about the bug come out we will definitely cover it.
- jcims 5y ago
  Awesome! I think my plug was a little forward for hump day on HN lol. Love the podcast.

bluedevil2k 5y ago

Why is Amazon dependent on someone else’s cloud infrastructure?

busymom0 5y ago
Someone answered this yesterday. CloudFront is good for video and large download assets (plus very low margins) but not for images and smaller stuff which Fastly is much faster at: https://www.streamingmediablog.com/2020/05/fastly-amazon-homepage.html https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...
- deleted 5y ago
  [deleted]
tyingq 5y ago
I believe Google's Firebase also uses Fastly.
Thaxll 5y ago
Also large compagnies uses multiple CDN.
typon 5y ago
Hedging your bets
rchaud 5y ago
They can't afford Cloudfront bills either.

flareback 5y ago

fta: "The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet." I read that phrase everytime something like this happens and yet we all still rely on the same handful of companies.

axiosgunnar 5y ago
Would a smaller CDN provider have no outages at all?
- sgtfrankieboy 5y ago
  No, but it would take down less sites if it does have one.
  - gpm 5y ago
    Is that a feature? If we have 12 small cdns have 12 outages in a year (combined, 1/year each), each time bringing down 10,000 websites, is that better than 1 large cdn having one outage during the year bringing down 120,000 websites? If I'm the website owner I think I prefer the latter, my customers blame the cdn instead of me. If I'm the cdn owner I definitely prefer the latter, more customers to amortize my costs over.
    - iso1210 5y ago
      From a customer point of view diversity is far better it's great - if Sainsburys is closed for some reason I'll go to Tesco. Certainly don't want a situation where all the shops are closed.
- martius 5y ago
  It may be more acceptable to have more frequent outages if the impact radius (number of websites or services impacted) is smaller.
- zelon88 5y ago
  Obviously that isn't an honest comparison. If you're asking whether or not 10 small CDN providers could provide a more robust, higher quality service with more uptime than one large CDN provider, then I think the answer is probably "yes."

mikesabbagh 5y ago

the thing with CDN is that they may have many edge locations, but the cache does not sit there. They frequently have a common caching servers located close by. so maybe every 10 or 100 edge locations have a single cache location. Edge is a reverse proxy and probably handles ssl handshakes. So if your cache is down, all your edge locations in that area are down

uncertainrhymes 5y ago
This may be true with some, but it is not true of Fastly. Each of their edge nodes is a varnish cache. Because they are multi-tenant, when varnish crashes it crashes hard and takes everyone with it. The question in various threads is why not have redundancy -- but the point of a CDN is to have extra servers and capacity and lots of locations to make individual crashes just flow elsewhere. But if the single customer with a valid-yet-crashable config had lots of traffic all over the world... it'll take everything out at once. Redundancy of CDN is more expensive, and still requires DNS failover. People do the calculation and usually decide that 30 min of downtime every couple years is worth the saving on vendors and code and hassle. They don't like it, but every site that was down made that decision.

arrty88 5y ago

Did they just miss a where clause on the update statement??

iainmerrick 5y ago

Throwing in my positive hot take among all the negative ones here: the immediate response and blog post from Fastly here is really good. A quick fix, a clear apology, enough detail to give an idea of what happened, but not so much detail that there might be a mistake they’ll have to clarify or retract. What more are you looking for? Apart from “not have the bug in the first place” -- and I hope and expect they’ll go into more detail later when they’ve had time for a proper post mortem -- I’d be interested to hear what anyone thinks they could have done better in terms of their immediate firefighting.

oneeyedpigeon 5y ago
I'm sorry, but I disagree. They gave the BBC enough detail that a very misleading headline was produced as a result. True, the main blame lies with the BBC, but it also comes across — to me, anyway, maybe I'm being too cynical — as a bit of an excuse from Fastly.
- mytailorisrich 5y ago
  "But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing "85% of our network to return errors" They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.
  - oneeyedpigeon 5y ago
    I know — as I've said, the main blame lies with the bbc. However, as it's reported, it comes across very much as Fastly trying to save face. Maybe the blame is entirely on the bbc, maybe Fastly were naive in thinking that giving them this information wouldn't result in irresponsible headlines.
    - Jenk 5y ago
      How on earth do you figure that is anyone's fault but the BBC? Read Fastly's statement. There is nothing about it blaming the customer(s) at all. There is nothing trying to save face. What is your point here?

CalChris 5y ago

The BBC title is a terrible title. The customer didn’t cause the outage. The bug caused the outage.

underscore_ku 5y ago
this and the BBC bots that upvote every BBC news
mypastself 5y ago
Unless it’s been changed since your comment, I don’t think that’s what the title is claiming. A trigger is different from a cause.
Zenst 5y ago
That was my take as well and very much disenfranchises what happens with title's like that. If you have a design flaw, you don't hint at onus upon the first person to fall foul of it - and we all know how quick many are reading news that they will run fully upon the title alone (something we can all do). But we have seen a drive towards click-bait/search-bot friendly to garner hits - style headlines. Even the BBC over the years have IMHO learnt towards such tabloid style headlines more and that is just sad.
tantalor 5y ago
The bug is the root cause, the customer's action is the proximal cause (or "trigger")
- hnbad 5y ago
  Placing the emphasis on "one customer" still creates a false narrative though. Note the headline doesn't even say it was a bug. This headline provokes the question "Who was it and what did they do?" rather than the more insightful "What was the bug?". If a single bug caused the "internet meltdown", it's fairly likely that the bug was triggered by one person so there's no need to emphasize that part.
  - tantalor 5y ago

oneeyedpigeon 5y ago

Shame on the BBC for such a misleading headline. But Fastly probably shouldn't have even given them detail — it's irrelevant and bound to be misreported. Just own up to your bug and be done with it.

koreanguy 5y ago

single point failure, you are what you run

desireco42 5y ago

Them pointing to customer as a source of issue is not OK. This reminds me of that Chase scandal, when they transferred huge amount of money to pay of full loan instead of installment, then they blamed on some guy in India for alleged mistake, where his 2 or 3 superiors that approved that and really horrific interface, but they decided to pin the blame on that guy. So, I am glad things are OK, but it definitely is not that one customer who is to blame for this outage.

zomglings 5y ago
I don't think that Fastly is pinning blame on the customer. That seems to be the BBC trying to bait us into reading the article. I am sad to report that their clickbait worked on me.
- desireco42 5y ago
  I think you are right, it is my bad.

kevincox 5y ago

Global shared control plane updates are frighteningly common. For example Cloudflare regularly brags about how your configuration updates are pushed around the world in single-digit seconds. Sure, it is an amazing feature, but it opens you up to this sort of issue. All changes to critical infrastructure should be a gradual rollout (emergencies aside). Instant sounds nice, until it isn't. If this rolled out to one region for the first hour it likely would have been caught and Fastly could press the "stop all rollouts" button.

kissgyorgy 5y ago

So they have an infrastructure where customer data is not separated from other customers, that's pretty terrifying! This means another bug of the same kind can also cause a global disruption any time in the future (mistakes will happen).

aliasEli 5y ago
Yes, that might happen. But the only way to avoid that is to give each customer its own private hardware, which seems prohibitively expensive (and may not even prevent all failure sources).

NaturalPhallacy 5y ago

Modal free: https://archive.is/dKMHE https://archive.is/dKMHE

dang 5y ago

See also: Summary of June 8 outage - https://news.ycombinator.com/item?id=27444005 https://news.ycombinator.com/item?id=27444005 - June 2021 (ongoing)

alboaie 5y ago

it remembers ue about the inevitable tension between eficiency and resilience...