4 ms·
> I expected a brave tale of RedPanda SREs valiantly fixing things, or some cool automatic failover tech. It's a tale of how they set things up so they wouldn'
by gopher_space 1y ago
> I expected a brave tale of RedPanda SREs valiantly fixing things, or some cool automatic failover tech.
It's a tale of how they set things up so they wouldn't need to valiantly fix things, and I think the subtext is probably that Redpanda doesn't pass responsibility on to a third party.
There are plenty of domains and, more importantly, people who need uptime guarantees to mean "fix estimate from a real human working on the problem" and not eventual store credit. Payroll is a classic example.
- RadiozRadioz 1y agoNothing about the way they architected their system even mattered in this incident. Their service just wasn't using any of the infrastructure that failed - there was no event here that actually put their system design to the test. There just isn't a story here. It's like if the power went out in the building next door, and you wrote a blog post about how amazing the reliability of your office computers are compared to your neighbor. If your power had gone out too but you had provisioned a bunch of UPSs and been fine, then there's something to talk about. To extend the analogy, if the neighborhood had a reputation for brown-outs and you deliberately chose not to build an office there, then maybe you have something. But here, RedPanda's GCP offering is inside GCP, this failure in GCP has never happened before, they just got lucky.