4 ms·

I usually tell people you don't need as much reliability as you think. Three nines reliability is great for most purposes. 8 hours downtime a year. If your sy

by dmurray 17d ago

I usually tell people you don't need as much reliability as you think. Three nines reliability is great for most purposes. 8 hours downtime a year. If your system produces money at a constant rate, it captures 99.9% of the available money. Even two nines or one nine might be pretty good on that basis, when the alternative is spending 2x or 10x as much - let's build another unreliable system with that money that captures some other independent market opportunity. Poor reliability is a problem where you need to chain many systems together, or where the cost of a single failure is very large compared to a success. Or - as happens commonly because of load - if your periods of unreliability are correlated with periods of maximum opportunity, like an e-commerce site failing on Black Friday or a trading system failing when the market is most busy. But if you don't have one of those cases, evaluate whether investing in reliability is actually worth it to you. GitHub is an example where two nines of reliability ought to be OK. The argument against it is that it's bad marketing to have an unreliable service, especially one aimed at software engineers. And if GitHub is largely a marketing play by Microsoft anyway (do they really make back its cost in enterprise subscriptions?) then marketing considerations need to drive its reliability.

YawningAngel 17d ago
I'm not sure that having your expensive software engineers not work 2.4 days a year is very palatable
- Dylan16807 17d ago
  They have other stuff to do that's not bottlenecked on push/pull/CI. 1% github downtime should reduce productivity by a lot less than 1%.
- himata4113 17d ago
  There are n services and n tasks, the chances of overlap are minimal. In most cases people take a much needed break during downtime.
  - dmurray 17d ago
    Actually I think it's better if everything fails at once and everyone can take the day off (thanks, AWS!). Having your CI fail one day and your package repository the next might well cost you two days of productivity.
- vel0city 17d ago
  I fully agree with this take. We need to ensure we get every hour of work from our expensive engineers. This is why we got rid of coffee machines and bathrooms and moved to intravenous caffeine and other fluid drips and catheters. We cannot afford to lose productivity.
- dmurray 17d ago
  They already don't work for 130+ days of holidays, weekends, vacations and sick leave. What's another two days? Probably I already ask them to spend 2+ days attending HR or compliance training or listening to senior management tell them about sales targets. But the point is, that 1% extra productivity requires the sometimes staggering cost of making the software 10x or 100x more reliable.
- snapcaster 16d ago
  If your margin for error is this tight something is fundamentally broken in your business
prennert 16d ago
The thing is: its rarely constant rate anywhere. A short outage can snowball very easily in a lot of lost time. What I learned when working with enterprises is that above all they value reliability. This is for a reason. A short outage might at best trigger loads of paperwork for multiple hierarchies, big meetings etc. The org has no choice. It needs to evaluate if whatever happens is a threat to their business. In the worst case it is that, plus missing some crucial windows of delivery. This is because a system that is unavailable for a short time can cause backlogs that, like traffic jams, cascade as everyone has to slow down and then synchronously speed up again. Orgs have the option to create more resilience, but that is overhead similar to compliance. You need to drill all your backup plans all the time, otherwise they are worthless. The drills cost time and money. At scale it is infeasible to be robust to all failures. Therefore, enterprises (at least) often prefer reliable systems over sophisticated systems. Because this delegates the risk management to the vendors rather than adding an overhead to every employee. Because at some point the employee would just do drills all the time instead of work.

by dmurray 17d ago

YawningAngel 17d ago

I'm not sure that having your expensive software engineers not work 2.4 days a year is very palatable

Dylan16807 17d ago
They have other stuff to do that's not bottlenecked on push/pull/CI. 1% github downtime should reduce productivity by a lot less than 1%.
himata4113 17d ago
There are n services and n tasks, the chances of overlap are minimal. In most cases people take a much needed break during downtime.
- dmurray 17d ago
  Actually I think it's better if everything fails at once and everyone can take the day off (thanks, AWS!). Having your CI fail one day and your package repository the next might well cost you two days of productivity.
vel0city 17d ago
I fully agree with this take. We need to ensure we get every hour of work from our expensive engineers. This is why we got rid of coffee machines and bathrooms and moved to intravenous caffeine and other fluid drips and catheters. We cannot afford to lose productivity.
dmurray 17d ago
They already don't work for 130+ days of holidays, weekends, vacations and sick leave. What's another two days? Probably I already ask them to spend 2+ days attending HR or compliance training or listening to senior management tell them about sales targets. But the point is, that 1% extra productivity requires the sometimes staggering cost of making the software 10x or 100x more reliable.
snapcaster 16d ago
If your margin for error is this tight something is fundamentally broken in your business

prennert 16d ago

The thing is: its rarely constant rate anywhere. A short outage can snowball very easily in a lot of lost time. What I learned when working with enterprises is that above all they value reliability. This is for a reason. A short outage might at best trigger loads of paperwork for multiple hierarchies, big meetings etc. The org has no choice. It needs to evaluate if whatever happens is a threat to their business. In the worst case it is that, plus missing some crucial windows of delivery. This is because a system that is unavailable for a short time can cause backlogs that, like traffic jams, cascade as everyone has to slow down and then synchronously speed up again. Orgs have the option to create more resilience, but that is overhead similar to compliance. You need to drill all your backup plans all the time, otherwise they are worthless. The drills cost time and money. At scale it is infeasible to be robust to all failures. Therefore, enterprises (at least) often prefer reliable systems over sophisticated systems. Because this delegates the risk management to the vendors rather than adding an overhead to every employee. Because at some point the employee would just do drills all the time instead of work.