3 ms·

People very often argue that "it's hard to run large distributed systems" and also that "It's better to use SaaS as getting five-nines is really hard." I agree

by dijit 4y ago

People very often argue that "it's hard to run large distributed systems" and also that "It's better to use SaaS as getting five-nines is really hard." I agree, but I'm going to repeat the same thing I said yesterday: It really doesn't matter if your uptime is 80% as long as that 20% of downtime is happening when nobody is working. An 80% uptime architecture is hopelessly simple to maintain (and restore and so on). Complexity increases exponentially the more 9's you add, and those complexities need more and more people to paper over, eventually you end up with a false dichotomy. If you have 99.9% uptime and it's down when you need it with no clear resolution possible then you're not winning by having availability when you never needed it. The people arguing against self-hosting are so committed; I can't help but feel people know that it's a false economy but they're financially incentivised or they like having pretty green dots on a special website or something. Rachel by the Bay is US based (I am EU based) but even she has something to say on the subject: "My Nines are not your Nines": https://rachelbythebay.com/w/2019/07/15/giant/ https://rachelbythebay.com/w/2019/07/15/giant/

matthewowen 4y ago
I'm surprised by this perspective, I think in particular because it is hard to pick and choose availability. First, like many people, I work in a company that spans multiple time zones. At least 16 hours of the day are in someone on the eng team's working hours. Plenty of places _do_ practically speaking need to use version control 24/7 when you factor in both global teams and out of hours incidents, which at a large enough scale become routine. Second, when something fails, it doesn't _usually_ come back up of its own accord. It comes back up because someone fixes it. So you're either pulling people in out of hours to fix problems in these tools, or you're fixing them during work hours, which means that you have an outage when people are working! And if you work in an even slightly global team, oops, it might be both – it's in working hours for the people affected, but out of working hours for the people that maintain and can fix this system! Third, because operating these systems tends not to be someone's full time job, even trivial issues like "the disk is full from log entries because it got mistakenly set to be in debug mode" can wind up taking a surprisingly long time to diagnose and fix. What am I missing that makes it practical for a system to have 20% downtime, but only when people are not working?
- dijit 4y ago
  > I think in particular because it is hard to pick and choose availability I mean, human error is 95% of all outages, intentional changes to the running system and how it operates. You definitely can choose when you do those. Hardware or system failures outside of that are exceedingly rare, though I do submit they happen; simple systems are easier to stand up again if they fall over in the event that does happen. I mean; Stopping a database for a backup is a prime example of something that's significantly harder to do when you're running it 24/7 365. Similarly, performing a host migration, or running a potentially dangerous major version upgrade. What we do normally is limit the cost of rolling back, which is nice. But, instead of "iterate quickly and if it causes an outage we roll back" it can easily be: one guy stays a little bit late and upgrades the server and if it doesn't work then they roll back. -- impacting a much more limited set of people. Also: if you work globally, you can run upgrades on your edges. Also also: 20% unavailability is just an extreme case. I have 99.997% uptime with a single host machine that lives on a shelf in my room. I'm not saying this is how you should run systems, but it's pretty normal for even single nodes to have insanely high availability out of the box. Bonus: if it does have an outage, because it's so simple a restore takes 23 minutes. I know this because I run restores on another machine pretty often, and if I run a backup I usually run blue/green between these boxes.
RichieAHB 4y ago
In my experience “large distributed systems” don’t tend to be idling for 20% of the time? I think often such systems have some dependencies 24h a day. Even if it’s much lower in a 20% window you risk leaving clients out in the cold in that window, similarly to the post you linked, no?
- dijit 4y ago
  Put this way instead then: If your company is in Europe, and your gitlab instance goes down for 4hrs every day between 2.30am and 6.30am -- does anyone care? 0 people working those hours means 0 hours of lost dev time. If you have 5 minutes of availability lost at 9.30 every morning it's much worse (at least in GameDev, when everyone is pulling the latest generated binaries). 900 people unable to work for 5 minutes is 75 hours of lost dev time; quite a lot more than 0 hours, even if the actual availability of the system was significantly higher. If your audience is yourself, and people aren't supposed to be working, availability of those services is not necessary. Hell. the Google SRE workbook goes into exactly this, so I know I'm not crazy.
redleader55 4y ago
With enough retries, 5 ones turn into 5 nines, latency be damned!

by dijit 4y ago

matthewowen 4y ago

I'm surprised by this perspective, I think in particular because it is hard to pick and choose availability. First, like many people, I work in a company that spans multiple time zones. At least 16 hours of the day are in someone on the eng team's working hours. Plenty of places _do_ practically speaking need to use version control 24/7 when you factor in both global teams and out of hours incidents, which at a large enough scale become routine. Second, when something fails, it doesn't _usually_ come back up of its own accord. It comes back up because someone fixes it. So you're either pulling people in out of hours to fix problems in these tools, or you're fixing them during work hours, which means that you have an outage when people are working! And if you work in an even slightly global team, oops, it might be both – it's in working hours for the people affected, but out of working hours for the people that maintain and can fix this system! Third, because operating these systems tends not to be someone's full time job, even trivial issues like "the disk is full from log entries because it got mistakenly set to be in debug mode" can wind up taking a surprisingly long time to diagnose and fix. What am I missing that makes it practical for a system to have 20% downtime, but only when people are not working?

dijit 4y ago
> I think in particular because it is hard to pick and choose availability I mean, human error is 95% of all outages, intentional changes to the running system and how it operates. You definitely can choose when you do those. Hardware or system failures outside of that are exceedingly rare, though I do submit they happen; simple systems are easier to stand up again if they fall over in the event that does happen. I mean; Stopping a database for a backup is a prime example of something that's significantly harder to do when you're running it 24/7 365. Similarly, performing a host migration, or running a potentially dangerous major version upgrade. What we do normally is limit the cost of rolling back, which is nice. But, instead of "iterate quickly and if it causes an outage we roll back" it can easily be: one guy stays a little bit late and upgrades the server and if it doesn't work then they roll back. -- impacting a much more limited set of people. Also: if you work globally, you can run upgrades on your edges. Also also: 20% unavailability is just an extreme case. I have 99.997% uptime with a single host machine that lives on a shelf in my room. I'm not saying this is how you should run systems, but it's pretty normal for even single nodes to have insanely high availability out of the box. Bonus: if it does have an outage, because it's so simple a restore takes 23 minutes. I know this because I run restores on another machine pretty often, and if I run a backup I usually run blue/green between these boxes.

RichieAHB 4y ago

In my experience “large distributed systems” don’t tend to be idling for 20% of the time? I think often such systems have some dependencies 24h a day. Even if it’s much lower in a 20% window you risk leaving clients out in the cold in that window, similarly to the post you linked, no?

dijit 4y ago
Put this way instead then: If your company is in Europe, and your gitlab instance goes down for 4hrs every day between 2.30am and 6.30am -- does anyone care? 0 people working those hours means 0 hours of lost dev time. If you have 5 minutes of availability lost at 9.30 every morning it's much worse (at least in GameDev, when everyone is pulling the latest generated binaries). 900 people unable to work for 5 minutes is 75 hours of lost dev time; quite a lot more than 0 hours, even if the actual availability of the system was significantly higher. If your audience is yourself, and people aren't supposed to be working, availability of those services is not necessary. Hell. the Google SRE workbook goes into exactly this, so I know I'm not crazy.

redleader55 4y ago

With enough retries, 5 ones turn into 5 nines, latency be damned!