5 ms·

Behind the scenes: Redpanda Cloud's response to the GCP outage

77 points by eatonphil 1y ago

Peterpanzeri 1y ago
“We got lucky as the way we designed it happened not to use the part of the service that was degraded” this is a stupid statement from them, hope they will be prepared next time
- mankyd 1y ago
  Why is that stupid? They did get lucky. They are acknowledging that, had they used that, they would have had problems. And now they will work to be more prepared. Acknowledging that one still has risks and that luck plays a factor is important.
- beefnugs 1y ago
  I learned a lesson : "use less cloud"
zzyzxd 1y ago
The article is unnecessarily long only to brag about "a service we didn't use went down so it didn't affect us". If I want to be picky, their architecture is also not perfect: - Their alerts were not durable. The outage took out the alert system so humans were just eyeballing dashboards during the outage. What if your critical system went down along with that alert system, in the middle of night? - The cloud marketplace service was affected by cloudflare outage and there's nothiing they could do. - Tiered stroage was down, disk usage went above normal level. But there's no anomaly detection and no alerts. It survived because t0 storage was massively over provisioned. - They took pride in using industry well-known designs like cell-based architecture, redundancy, multi-az...ChatGPT would be able to give me a better list And I don't get whey they had to roast Crowdstrike at the end. I mean, the Crowdstrike incident was really amateur stuff, like, the absolute lowest bar I can think of.
diroussel 1y ago
> Modern computer systems are complex systems — and complex systems are characterized by their non-linear nature, which means that observed changes in an output are not proportional to the change in the input. This concept is also known in chaos theory as the butterfly effect, This isn't quite right. Linear systems can also be complex, and linear dynamic systems can also exhibit the butterfly effect. That is why the butterfly effect is so interesting. Of course non-linear systems can have a large change in output based on a small input, because they allow step changes, and many other non-linear processes.
iwontberude 1y ago
[flagged]
- deleted 1y ago
  [deleted]
RadiozRadioz 1y ago
Hmm. Here's what I read from this article: RedPanda didn't happen to use any of the stuff in GCP that went down, so they were unaffected. They use a 3rd party for alerting and dashboarding, and that 3rd party went down, but RedPanda still had their own monitoring. When I read "major outage for a large part of the internet was just another normal day for Redpanda Cloud customers", I expected a brave tale of RedPanda SREs valiantly fixing things, or some cool automatic failover tech. What I got instead was: Google told RedPanda there was an issue, RedPanda had a look and their service was unaffected, nothing needed failing over, then someone at RedPanda wrote an article bragging about their triple-nine uptime & fault tolerance. I get it, an SRE is doing well if you don't notice them, but the only real preventative measure I saw here that directly helped with this issue, is that they over provision disk space. Which I'd be alarmed if they didn't do.
- echelon 1y ago
  > triple-nine uptime & fault tolerance. Haha, we used to joke that's how many nines our customer-facing Ruby on Rails services had compared against our resilient five nines payments systems. Our heavy infra handled billions in daily payment volume and couldn't go down. With the Ruby teams, we often playfully quipped, "which nines are those?" humorously implying the leading digit itself wasn't itself a nine.
  - sokoloff 1y ago
    AKA: "We're closing in on our third 8 of uptime..."
- gopher_space 1y ago
  > I expected a brave tale of RedPanda SREs valiantly fixing things, or some cool automatic failover tech. It's a tale of how they set things up so they wouldn't need to valiantly fix things, and I think the subtext is probably that Redpanda doesn't pass responsibility on to a third party. There are plenty of domains and, more importantly, people who need uptime guarantees to mean "fix estimate from a real human working on the problem" and not eventual store credit. Payroll is a classic example.
bdavbdav 1y ago
“We got lucky as the way we designed it happened not to use the part of the service that was degraded”
- smoyer 1y ago
  And we're oblivious enough about that luck that we're patting ourselves on the back in public.
  - belter 1y ago
    And we are linking our blog to the AWS doc on cell architectures, while talking about multiaz clusters on GCP azs that are nothing like that...
rybosome 1y ago
Must be hell inside GCP right now. That was a big outage, and they were tired of big outages years ago. It was already extremely difficult to move quickly and get things done due to the reliability red tape, and I have to imagine this will make it even harder.
- siscia 1y ago
  In fairness, their design does not seem to be regional. With problems in one region bringing down another, apparently not unrelated, region. With this kind of architecture, this sort of problems is just bound to happen. During my time in AWS, region independence was a must. And some services were able to operate at least for a while without degrading also when some core dependencies were not available. Think like loosing S3. And after that, the service would keep operating, but with a degraded experience. I am stunned that this level of isolation is not common in GCP.
  - valenterry 1y ago
    How does AWS do that though? Do the re-implement all the code in every region? Because even the slightest re-use of code could trigger a synchronous (possibly delayed) downtime of all regions.
    - crop_rotation 1y ago
      Reusing code doesn't trigger region dependencies. > Do the re-implement all the code in every region? Everyone does. The difference is AWS very strongly ensures that regions are independent failure domains. The GCP architecture is global with all the pros and cons that implies. e.g GCP has a truly global load balancer while AWS can not since everything is at core regional.
    - nijave 1y ago
      They definitely roll out code (at least for some services) one region at a time. That doesn't prevent old bugs/issues from coming up but it definitely helps prevent new ones from becoming global outages.
raverbashing 1y ago
Lol I love how they call "not spreading your services needlessly across many different servers" as an "Architectural Pattern" (Cell based arch) They are right, of course, but the way things, the obvious needs to be said sometimes
- macintux 1y ago
  Years ago I had the misfortune of helping a company recover from an outage. It turned out that they had services in two data centers for redundancy, but they divided their critical services between them. So if either data center went offline, their whole stack was dead. Brilliant design. That was a very long week; fortunately by now I’ve forgotten most of it.
deleted 1y ago
[deleted]

77 points by eatonphil 1y ago

“We got lucky as the way we designed it happened not to use the part of the service that was degraded” this is a stupid statement from them, hope they will be prepared next time

mankyd 1y ago
Why is that stupid? They did get lucky. They are acknowledging that, had they used that, they would have had problems. And now they will work to be more prepared. Acknowledging that one still has risks and that luck plays a factor is important.
beefnugs 1y ago
I learned a lesson : "use less cloud"

zzyzxd 1y ago

The article is unnecessarily long only to brag about "a service we didn't use went down so it didn't affect us". If I want to be picky, their architecture is also not perfect: - Their alerts were not durable. The outage took out the alert system so humans were just eyeballing dashboards during the outage. What if your critical system went down along with that alert system, in the middle of night? - The cloud marketplace service was affected by cloudflare outage and there's nothiing they could do. - Tiered stroage was down, disk usage went above normal level. But there's no anomaly detection and no alerts. It survived because t0 storage was massively over provisioned. - They took pride in using industry well-known designs like cell-based architecture, redundancy, multi-az...ChatGPT would be able to give me a better list And I don't get whey they had to roast Crowdstrike at the end. I mean, the Crowdstrike incident was really amateur stuff, like, the absolute lowest bar I can think of.

diroussel 1y ago

> Modern computer systems are complex systems — and complex systems are characterized by their non-linear nature, which means that observed changes in an output are not proportional to the change in the input. This concept is also known in chaos theory as the butterfly effect, This isn't quite right. Linear systems can also be complex, and linear dynamic systems can also exhibit the butterfly effect. That is why the butterfly effect is so interesting. Of course non-linear systems can have a large change in output based on a small input, because they allow step changes, and many other non-linear processes.

iwontberude 1y ago

[flagged]

deleted 1y ago
[deleted]

RadiozRadioz 1y ago

Hmm. Here's what I read from this article: RedPanda didn't happen to use any of the stuff in GCP that went down, so they were unaffected. They use a 3rd party for alerting and dashboarding, and that 3rd party went down, but RedPanda still had their own monitoring. When I read "major outage for a large part of the internet was just another normal day for Redpanda Cloud customers", I expected a brave tale of RedPanda SREs valiantly fixing things, or some cool automatic failover tech. What I got instead was: Google told RedPanda there was an issue, RedPanda had a look and their service was unaffected, nothing needed failing over, then someone at RedPanda wrote an article bragging about their triple-nine uptime & fault tolerance. I get it, an SRE is doing well if you don't notice them, but the only real preventative measure I saw here that directly helped with this issue, is that they over provision disk space. Which I'd be alarmed if they didn't do.

echelon 1y ago
> triple-nine uptime & fault tolerance. Haha, we used to joke that's how many nines our customer-facing Ruby on Rails services had compared against our resilient five nines payments systems. Our heavy infra handled billions in daily payment volume and couldn't go down. With the Ruby teams, we often playfully quipped, "which nines are those?" humorously implying the leading digit itself wasn't itself a nine.
- sokoloff 1y ago
  AKA: "We're closing in on our third 8 of uptime..."
gopher_space 1y ago
> I expected a brave tale of RedPanda SREs valiantly fixing things, or some cool automatic failover tech. It's a tale of how they set things up so they wouldn't need to valiantly fix things, and I think the subtext is probably that Redpanda doesn't pass responsibility on to a third party. There are plenty of domains and, more importantly, people who need uptime guarantees to mean "fix estimate from a real human working on the problem" and not eventual store credit. Payroll is a classic example.

bdavbdav 1y ago

“We got lucky as the way we designed it happened not to use the part of the service that was degraded”

smoyer 1y ago
And we're oblivious enough about that luck that we're patting ourselves on the back in public.
- belter 1y ago
  And we are linking our blog to the AWS doc on cell architectures, while talking about multiaz clusters on GCP azs that are nothing like that...

rybosome 1y ago

Must be hell inside GCP right now. That was a big outage, and they were tired of big outages years ago. It was already extremely difficult to move quickly and get things done due to the reliability red tape, and I have to imagine this will make it even harder.

siscia 1y ago
In fairness, their design does not seem to be regional. With problems in one region bringing down another, apparently not unrelated, region. With this kind of architecture, this sort of problems is just bound to happen. During my time in AWS, region independence was a must. And some services were able to operate at least for a while without degrading also when some core dependencies were not available. Think like loosing S3. And after that, the service would keep operating, but with a degraded experience. I am stunned that this level of isolation is not common in GCP.
- valenterry 1y ago
  How does AWS do that though? Do the re-implement all the code in every region? Because even the slightest re-use of code could trigger a synchronous (possibly delayed) downtime of all regions.
  - crop_rotation 1y ago
    Reusing code doesn't trigger region dependencies. > Do the re-implement all the code in every region? Everyone does. The difference is AWS very strongly ensures that regions are independent failure domains. The GCP architecture is global with all the pros and cons that implies. e.g GCP has a truly global load balancer while AWS can not since everything is at core regional.
  - nijave 1y ago
    They definitely roll out code (at least for some services) one region at a time. That doesn't prevent old bugs/issues from coming up but it definitely helps prevent new ones from becoming global outages.

raverbashing 1y ago

Lol I love how they call "not spreading your services needlessly across many different servers" as an "Architectural Pattern" (Cell based arch) They are right, of course, but the way things, the obvious needs to be said sometimes

macintux 1y ago
Years ago I had the misfortune of helping a company recover from an outage. It turned out that they had services in two data centers for redundancy, but they divided their critical services between them. So if either data center went offline, their whole stack was dead. Brilliant design. That was a very long week; fortunately by now I’ve forgotten most of it.

deleted 1y ago

[deleted]