20 ms·

Stripe CEO here. We're very sorry about this. We work hard to maintain extreme reliability in our infrastructure, with a lot of redundancy at different levels.

by pc 7y ago

Stripe CEO here. We're very sorry about this. We work hard to maintain extreme reliability in our infrastructure, with a lot of redundancy at different levels. This morning, our API was heavily degraded (though not totally down) for 24 minutes. We'll be conducting a thorough investigation and root-cause analysis.

perfect_wave 7y ago
I also look forward to reading the postmortem. Stripe puts out a lot of high quality blog posts.
- jamestimmins 7y ago
  The Stripe outage that occurred bc of a deleted DB index was particularly interesting. Someone deleted an index before the replacement was live due to a process error. This had cascading effects across the system and caused a large % of API requests to timeout. It's such a pedestrian problem but had an enormous impact.
JaimeThompson 7y ago
Will the results of the investigation and analysis be publicly available?
- filoleg 7y ago
  This. Reading well-written post-mortems of outages for big and complex services like Stripe is just pure joy to me and feels very educational too. I remember reading Gitlab post-mortems earlier this year, and it felt really fresh, given how honest and open they were in those.
  - sovnade 7y ago
    They're really helpful in preventing our own outages. Although it's ironic how many of them boil down to having things so well automated that one mistyped command can take down an entire environment with extreme efficiency.
calhoun137 7y ago
This is causing a big problem for my business right now, but I am not mad at Stripe because you earned that level of credibility and respect in my opinion. I understand these things happen and am glad to know a team as excellent as Stripe's is on the job.
simonebrunozzi 7y ago
Patrick, I think that it would be really nice to share the technical details of the post-mortem, once the dust has settled. Many folks don't have the privilege to run a massive scale operation like Stripe, and lots of people can learn from it.

by pc 7y ago

perfect_wave 7y ago

I also look forward to reading the postmortem. Stripe puts out a lot of high quality blog posts.

jamestimmins 7y ago
The Stripe outage that occurred bc of a deleted DB index was particularly interesting. Someone deleted an index before the replacement was live due to a process error. This had cascading effects across the system and caused a large % of API requests to timeout. It's such a pedestrian problem but had an enormous impact.

JaimeThompson 7y ago

Will the results of the investigation and analysis be publicly available?

filoleg 7y ago
This. Reading well-written post-mortems of outages for big and complex services like Stripe is just pure joy to me and feels very educational too. I remember reading Gitlab post-mortems earlier this year, and it felt really fresh, given how honest and open they were in those.
- sovnade 7y ago
  They're really helpful in preventing our own outages. Although it's ironic how many of them boil down to having things so well automated that one mistyped command can take down an entire environment with extreme efficiency.

calhoun137 7y ago

This is causing a big problem for my business right now, but I am not mad at Stripe because you earned that level of credibility and respect in my opinion. I understand these things happen and am glad to know a team as excellent as Stripe's is on the job.

simonebrunozzi 7y ago

Patrick, I think that it would be really nice to share the technical details of the post-mortem, once the dust has settled. Many folks don't have the privilege to run a massive scale operation like Stripe, and lots of people can learn from it.