5 ms·

Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration chang

Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout. Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale? Good on them for getting a statement out right away (although they basically had to) but seems to be lacking any useful details. Wonder if they were scrubbed by PR/legal in hopes of reducing the number of customers coming to ask for gibs.

abestic9 5y ago
It's an update on the situation until they confirm the bug fix is completely rolled out. I certainly wouldn't expect they tell their customer base how they can exploit a bug to bring down part of their business. With respect to partitioning - we don't know how or why an invalid configuration could poison so many nodes; if the config was physically present on them or if there was a cascade of healing/balancing issues stemming from it. I would leave speculation on many of your points at the doorstep until we see a full report.
plasma 5y ago
At a guess, perhaps each server can serve everyone, and the system-wide config file that is shipped to each server (to handle each customer) perhaps became corrupted as part of a customer update, and it was rolled out to every node assumed to be correct. A defense against this could be to ensure the system that applies the change validates some health-checks continue to work after the new file is made (or automatically rollback to previous configuration). I can see how this would happen, assuming thats what happened.
- EE84M3i 5y ago
  Yes, it sounds sort of like that. But this can be remediated by two things I was asking about: smoke tests and partitioning. When making a config change I'd assume they don't make it to all servers at once and instead roll it out gradually. If this caused the server to instantly start 503'ing all customers, presumably this would have been caught - perhaps it was more delayed though (resource leak, etc) and obviously that is somewhat more difficult to catch. If they're properly partitioning customers, ideally they wouldn't even ship the configs to all servers (slightly less good, but still pretty good they could ship them there but not parse/load them). It sounds like at the least this customer's config change effected 85% of servers, which seems absurd to me. So yes, I can see how it happened, but for Fastly, which runs one of the biggest CDNs, these don't seem like very reasonable mistakes.
  - sersi 5y ago
    One of the big competitive advantage Fastly has compared to say Akamai is that configuration changes roll out extremely fast. I could see them skimping on smoke tests to keep that advantage and not thinking that this could ever happen.
unityByFreedom 5y ago
> Is it a whale? TikTok is my guess. ByteDance is valued at 250 billion. Plus, the change was pushed in the middle of the night, which would be daytime in Asia. Certainly there are other development teams in Asia, but considering the scale of the change it likely comes from HQ, and Fastly's whale in Asia would be them. edit They may have lost TikTok at the end of last year, either partially or completely [1]. Anyone know what they use now? Akamai, or maybe they stealthily switched back to Fastly? [1] https://www.barrons.com/articles/fastly-lost-tiktok-but-ceo-says-ive-never-been-more-bullish-51603921961 https://www.barrons.com/articles/fastly-lost-tiktok-but-ceo-...
- Sander_Marechal 5y ago
  My guess is reddit. It was down yesterday due to this.
  - unityByFreedom 5y ago
    Why would reddit push such a change at 4am SF time? It's also unlikely that a high load at that time would trigger anything. Reddit's peak activity times are US daylight hours.
- Goz3rr 5y ago
  Middle of the night? It was at noon for most of Europe
  - unityByFreedom 5y ago
    What are Europe's whales?
    - vbsteven 5y ago
      Lots of tech companies have engineering offices with SRE responsabilities in Europe timezones. Could be anyone really.
      - 5y ago
notyourday 5y ago
At its core Fastly uses heavily modified Varnish 2 with customer's having full access to vcl providing an unprecedented level of control of the request processing and cache behavior. It is extremely difficult, if not impossible, to completely remove foot-guns while retaining this ability. They do amazing thing checking validity of the code and aborting broken requests but it is still vcl and in its core varnish 2 does not have multi-tenancy segmentation.
- notacoward 5y ago
  Thank you for that. That last clause seems like the real key here. With that much power in customers' hands and no hard multi-tenant isolation, unintentional DoS becomes almost inevitable. In effect, it puts every customer at the mercy of every other's diligence (or lack thereof). Even with diligence, that seems a bit fragile.
alkonaut 5y ago
Smoke tests or gradual rollout won’t help with the change by fastly on May 12 when they deployed the buggy change. It’s likely that this was gradually rolled out and looked ok with all customer configs existing on May 12. Obviously there should be other safeguards in place but gradual rollout in itself wouldn’t have helped since it would look green at a 100% rollout weeks ago.
- EE84M3i 5y ago
  I meant smoke testing and gradual rollout of the config, not the code, although obviously that's important too.
  - alkonaut 5y ago
    You’d think that individual customer configuration changes should only ever affect that customer, and that gradual vs instant rollout would be an option the customer handles when changing their configuration!
cowsandmilk 5y ago
> one customer that gets served from 85% of their servers There could be a feedback loop that is the opposite of a smoke test. 1. Validate customer configuration, if passes, assume it can roll out 2. Roll out customer configuration to node 3. Node goes down 4. Migrate all customers on node to new nodes 5. Node that problematic customer was migrated to goes down 6. Rinse and repeat as problematic customer migrates to every node and takes out every last one.
fierro 5y ago
guessing it's somehow BGP related. can't wait to find out

abestic9 5y ago

It's an update on the situation until they confirm the bug fix is completely rolled out. I certainly wouldn't expect they tell their customer base how they can exploit a bug to bring down part of their business. With respect to partitioning - we don't know how or why an invalid configuration could poison so many nodes; if the config was physically present on them or if there was a cascade of healing/balancing issues stemming from it. I would leave speculation on many of your points at the doorstep until we see a full report.

plasma 5y ago

At a guess, perhaps each server can serve everyone, and the system-wide config file that is shipped to each server (to handle each customer) perhaps became corrupted as part of a customer update, and it was rolled out to every node assumed to be correct. A defense against this could be to ensure the system that applies the change validates some health-checks continue to work after the new file is made (or automatically rollback to previous configuration). I can see how this would happen, assuming thats what happened.

EE84M3i 5y ago
Yes, it sounds sort of like that. But this can be remediated by two things I was asking about: smoke tests and partitioning. When making a config change I'd assume they don't make it to all servers at once and instead roll it out gradually. If this caused the server to instantly start 503'ing all customers, presumably this would have been caught - perhaps it was more delayed though (resource leak, etc) and obviously that is somewhat more difficult to catch. If they're properly partitioning customers, ideally they wouldn't even ship the configs to all servers (slightly less good, but still pretty good they could ship them there but not parse/load them). It sounds like at the least this customer's config change effected 85% of servers, which seems absurd to me. So yes, I can see how it happened, but for Fastly, which runs one of the biggest CDNs, these don't seem like very reasonable mistakes.
- sersi 5y ago
  One of the big competitive advantage Fastly has compared to say Akamai is that configuration changes roll out extremely fast. I could see them skimping on smoke tests to keep that advantage and not thinking that this could ever happen.

unityByFreedom 5y ago

> Is it a whale? TikTok is my guess. ByteDance is valued at 250 billion. Plus, the change was pushed in the middle of the night, which would be daytime in Asia. Certainly there are other development teams in Asia, but considering the scale of the change it likely comes from HQ, and Fastly's whale in Asia would be them. edit They may have lost TikTok at the end of last year, either partially or completely [1]. Anyone know what they use now? Akamai, or maybe they stealthily switched back to Fastly? [1] https://www.barrons.com/articles/fastly-lost-tiktok-but-ceo-says-ive-never-been-more-bullish-51603921961 https://www.barrons.com/articles/fastly-lost-tiktok-but-ceo-...

Sander_Marechal 5y ago
My guess is reddit. It was down yesterday due to this.
- unityByFreedom 5y ago
  Why would reddit push such a change at 4am SF time? It's also unlikely that a high load at that time would trigger anything. Reddit's peak activity times are US daylight hours.
Goz3rr 5y ago
Middle of the night? It was at noon for most of Europe
- unityByFreedom 5y ago
  What are Europe's whales?
  - vbsteven 5y ago
    Lots of tech companies have engineering offices with SRE responsabilities in Europe timezones. Could be anyone really.
    - 5y ago

notyourday 5y ago

At its core Fastly uses heavily modified Varnish 2 with customer's having full access to vcl providing an unprecedented level of control of the request processing and cache behavior. It is extremely difficult, if not impossible, to completely remove foot-guns while retaining this ability. They do amazing thing checking validity of the code and aborting broken requests but it is still vcl and in its core varnish 2 does not have multi-tenancy segmentation.

notacoward 5y ago
Thank you for that. That last clause seems like the real key here. With that much power in customers' hands and no hard multi-tenant isolation, unintentional DoS becomes almost inevitable. In effect, it puts every customer at the mercy of every other's diligence (or lack thereof). Even with diligence, that seems a bit fragile.

alkonaut 5y ago

Smoke tests or gradual rollout won’t help with the change by fastly on May 12 when they deployed the buggy change. It’s likely that this was gradually rolled out and looked ok with all customer configs existing on May 12. Obviously there should be other safeguards in place but gradual rollout in itself wouldn’t have helped since it would look green at a 100% rollout weeks ago.

EE84M3i 5y ago
I meant smoke testing and gradual rollout of the config, not the code, although obviously that's important too.
- alkonaut 5y ago
  You’d think that individual customer configuration changes should only ever affect that customer, and that gradual vs instant rollout would be an option the customer handles when changing their configuration!

cowsandmilk 5y ago

> one customer that gets served from 85% of their servers There could be a feedback loop that is the opposite of a smoke test. 1. Validate customer configuration, if passes, assume it can roll out 2. Roll out customer configuration to node 3. Node goes down 4. Migrate all customers on node to new nodes 5. Node that problematic customer was migrated to goes down 6. Rinse and repeat as problematic customer migrates to every node and takes out every last one.

fierro 5y ago

guessing it's somehow BGP related. can't wait to find out