6 ms·

Easier said than done. No system with real usage can achieve 100% up time. When problems inevitably occur, what do you expect Stripe to do? Wait until 10am the

Easier said than done. No system with real usage can achieve 100% up time. When problems inevitably occur, what do you expect Stripe to do? Wait until 10am the next morning??

izacus 4y ago
The "inevitability" of problems is what is the problem here. Toxic organisations always find ways to defend their deep issues as "inevitable". Google SRE book is now decade old. Read it.
- lozenge 4y ago
  "100% is probably never the right reliability target: not only is it impossible to achieve, it’s typically more reliability than a service’s users want or notice. Match the profile of the service to the risk the business is willing to take." "Things break; that’s life."
  - izacus 4y ago
    Yes, and with 14 global offices the company can fix them in worktime of one of their employees, not late at night. Let me reiterate: "inevitable" things are rarely inevitable and most commonly just a poor excuse for poor organization and toxic work practices. Even if you put out pithy quotes.
    - ZephyrBlu 4y ago
      In theory, yes. In practice, it's highly unlikely that there will always be someone with the right context to fix the problem in each time zone. The way my company does this is with a centralized incident management team that follows the sun across 4 timezones, and specific teams will be paged by them depending on what is broken.
      - jjtang1 4y ago
        For anyone that is interested, my team at Rootly (https://rootly.com/ https://rootly.com/) built an incident management platform that has been greatly inspired by the folks at Stripe!
- vasco 4y ago
  Even trying to implement solid SRE practices you'll still have inevitable problems, incidents and other disruptions. I agree what most people consider inevitable isn't, but your position is too extreme if you've ever been in a real company. Not everything can be predicted by an error budget, and in most non-google companies "stop deploying until you got your budget back" isn't going to allow your business to operate fast enough or flexibly enough. Reliability isn't all in business.
  - izacus 4y ago
    That's correct - but this argument is more often than not used to prop up "inevitability" of overtime that neither inevitable, nor necessary. Yes, bad stuff happens - but having it once per two years or once per few months as the article implies it a different scale and the MOST important part of it is to stop treating it as normal, required and necessary. Also, Stripe has 14 offices globally. That's already a size where you can have work-time rotation for those "late night" problems. Which then stop being late night problems. Again. Not inevitable.

Easier said than done. No system with real usage can achieve 100% up time. When problems inevitably occur, what do you expect Stripe to do? Wait until 10am the next morning??

izacus 4y ago

The "inevitability" of problems is what is the problem here. Toxic organisations always find ways to defend their deep issues as "inevitable". Google SRE book is now decade old. Read it.

lozenge 4y ago
"100% is probably never the right reliability target: not only is it impossible to achieve, it’s typically more reliability than a service’s users want or notice. Match the profile of the service to the risk the business is willing to take." "Things break; that’s life."
- izacus 4y ago
  Yes, and with 14 global offices the company can fix them in worktime of one of their employees, not late at night. Let me reiterate: "inevitable" things are rarely inevitable and most commonly just a poor excuse for poor organization and toxic work practices. Even if you put out pithy quotes.
  - ZephyrBlu 4y ago
    In theory, yes. In practice, it's highly unlikely that there will always be someone with the right context to fix the problem in each time zone. The way my company does this is with a centralized incident management team that follows the sun across 4 timezones, and specific teams will be paged by them depending on what is broken.
    - jjtang1 4y ago
      For anyone that is interested, my team at Rootly (https://rootly.com/ https://rootly.com/) built an incident management platform that has been greatly inspired by the folks at Stripe!
vasco 4y ago
Even trying to implement solid SRE practices you'll still have inevitable problems, incidents and other disruptions. I agree what most people consider inevitable isn't, but your position is too extreme if you've ever been in a real company. Not everything can be predicted by an error budget, and in most non-google companies "stop deploying until you got your budget back" isn't going to allow your business to operate fast enough or flexibly enough. Reliability isn't all in business.
- izacus 4y ago
  That's correct - but this argument is more often than not used to prop up "inevitability" of overtime that neither inevitable, nor necessary. Yes, bad stuff happens - but having it once per two years or once per few months as the article implies it a different scale and the MOST important part of it is to stop treating it as normal, required and necessary. Also, Stripe has 14 offices globally. That's already a size where you can have work-time rotation for those "late night" problems. Which then stop being late night problems. Again. Not inevitable.