4 ms·
A continuing theme in these broad outages that keep happening is that these company's are dogfooding their own services to provide the same service to their cus
by JohnMakin 3y ago
A continuing theme in these broad outages that keep happening is that these company's are dogfooding their own services to provide the same service to their customers.
I'm thinking of the AWS lambda outage taking out the control plane as a recent example.
I'm not sure or saying they're wrong for doing this, I just keep noticing this same pattern of "One of our products broke, now all of our tooling is failing." This seems.. wrong somehow?
- pixl97 3y agoSo they should use another tool with its own set of failure conditions?
- JohnMakin 3y agoNo, but maybe they should consider infrastructure where if a customer-facing service goes down their internal control mechanisms don’t fail and the only backup mechanism is break-the-glass procedures by on-call engineers that take half an hour to go live. It’s a big deal when CF services go down, this definitely impacted me, and I just expected more redundancy. It seems like their rollback mechanism failed as well.
- PrayagBhakar 3y agoBut isn’t this just a natural part of how distributed systems work? If one part fails it can have downstream effects that can cross multiple boundaries like end user clients crashing, or another business’s server crashing, or your own server crashing.