4 ms·

In some sense I suppose if AWS goes down in some fashion it's kind of like an act of nature. There's nothing individually you could do to prevent it. On the oth

In some sense I suppose if AWS goes down in some fashion it's kind of like an act of nature. There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix.

abraxas 6y ago
In other words we're in the "Nobody gets fired for choosing managed AWS" paradigm.
- dageshi 6y ago
  Basically. If you go down you're probably in the company of half the fortune 500. And of course AWS is a legitimately capable and useful platform that does remove a lot "ops" although it probably frontloads a lot of "config" in return for that.
- derefr 6y ago
  It's not just diffusion of responsibility. It's the fact that — assuming you're a B2B SaaS — when you're down, the customers depending on your service are probably down too, since they're very likely to also be dependent on the same service that knocked you offline. It's like a city-wide power outage, or a flood in a city that's in a flood-basin. The city's entire economy will "pause." If you're a B2B business affected by the outage/flood, you don't have to worry about losing sales/customers, as your customers won't even be trying to make any requests of you, since their capacity to serve requests to their customers is knocked out, and without that, they won't be making any requests of you. In fact, as long as they're down, they may not even notice that you're down!
- wdb 6y ago
  Still if I cause P2/P1 issue I still need to explain to the board of top 50 fortune 500 company. Had to do it once its no fun at all.
briffle 6y ago
There is also (and especially if your a small shop) the thing where if your self hosted stuff goes down, you have a crappy time fixing it, over weekend, holiday, etc, until its done. If AWS goes down, thousands of other people are working on it frantically. And most likely, many of them are smarter/better than you.
- tlarkworthy 6y ago
  Single tenancy systems are simpler and avoid neighbourhood noise and safety limits.
- mattmanser 6y ago
  So this just happened to a friend of mine who used managed.com, which was down for a week last week because of a ransomware attack. Didn't really get mentioned here, but that was a lot of websites. Instead of stressing about getting his sites back up, he was stressing that he'd never get any of his data or code back and that he couldn't contact his 100,000s of users because he didn't even have a recent backup of the database with all of the emails. Even his DNS settings went down as those sort of sites push you to making them the nameservers. I ended up telling him about wayback machine just so he could get a homepage set up. He'd relied on their claims that they managed his backups for him, but when they went down, it took down his ability to even access the backups, and they wouldn't do anything, I assume in case those backups were infected too. So in fact if you're blindly relying on an external provider, it's even more risky than hosting yourself. At least the stuff you self host you are almost pushed to back up to something like S3, and both won't go down.
  - hunter2_ 6y ago
    The entire premise of claiming to have backups is that you are the fork who ensures multiple copies have no commonalities, no overlap, in what keeps them healthy. You can delegate if you aren't interested in DIY and getting in the trenches, but you still need to apply the original premise by being a fork and delegating to multiple separate entities... Otherwise, you can't claim to have backups, despite the fact that the one basket you put your eggs in can legitimately claim to have backups.
NullPrefix 6y ago
>There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix. I feel like this statement perfectly embodies the problems that people face when choosing a closed source OS vs Linux/BSD. When stuff doesn't work on Windows there's pretty much nothing you can do so you have to just accept it and pretend that you wanted it to not work this way. As opposed to Linux where you can make it work your way, but it costs a lot of time to do that.

abraxas 6y ago

In other words we're in the "Nobody gets fired for choosing managed AWS" paradigm.

dageshi 6y ago
Basically. If you go down you're probably in the company of half the fortune 500. And of course AWS is a legitimately capable and useful platform that does remove a lot "ops" although it probably frontloads a lot of "config" in return for that.
derefr 6y ago
It's not just diffusion of responsibility. It's the fact that — assuming you're a B2B SaaS — when you're down, the customers depending on your service are probably down too, since they're very likely to also be dependent on the same service that knocked you offline. It's like a city-wide power outage, or a flood in a city that's in a flood-basin. The city's entire economy will "pause." If you're a B2B business affected by the outage/flood, you don't have to worry about losing sales/customers, as your customers won't even be trying to make any requests of you, since their capacity to serve requests to their customers is knocked out, and without that, they won't be making any requests of you. In fact, as long as they're down, they may not even notice that you're down!
wdb 6y ago
Still if I cause P2/P1 issue I still need to explain to the board of top 50 fortune 500 company. Had to do it once its no fun at all.

briffle 6y ago

There is also (and especially if your a small shop) the thing where if your self hosted stuff goes down, you have a crappy time fixing it, over weekend, holiday, etc, until its done. If AWS goes down, thousands of other people are working on it frantically. And most likely, many of them are smarter/better than you.

tlarkworthy 6y ago
Single tenancy systems are simpler and avoid neighbourhood noise and safety limits.
mattmanser 6y ago
So this just happened to a friend of mine who used managed.com, which was down for a week last week because of a ransomware attack. Didn't really get mentioned here, but that was a lot of websites. Instead of stressing about getting his sites back up, he was stressing that he'd never get any of his data or code back and that he couldn't contact his 100,000s of users because he didn't even have a recent backup of the database with all of the emails. Even his DNS settings went down as those sort of sites push you to making them the nameservers. I ended up telling him about wayback machine just so he could get a homepage set up. He'd relied on their claims that they managed his backups for him, but when they went down, it took down his ability to even access the backups, and they wouldn't do anything, I assume in case those backups were infected too. So in fact if you're blindly relying on an external provider, it's even more risky than hosting yourself. At least the stuff you self host you are almost pushed to back up to something like S3, and both won't go down.
- hunter2_ 6y ago
  The entire premise of claiming to have backups is that you are the fork who ensures multiple copies have no commonalities, no overlap, in what keeps them healthy. You can delegate if you aren't interested in DIY and getting in the trenches, but you still need to apply the original premise by being a fork and delegating to multiple separate entities... Otherwise, you can't claim to have backups, despite the fact that the one basket you put your eggs in can legitimately claim to have backups.

NullPrefix 6y ago

>There's nothing individually you could do to prevent it. On the other hand if you self host and there's an outage, the buck stops with you, it's your problem to diagnose and fix. I feel like this statement perfectly embodies the problems that people face when choosing a closed source OS vs Linux/BSD. When stuff doesn't work on Windows there's pretty much nothing you can do so you have to just accept it and pretend that you wanted it to not work this way. As opposed to Linux where you can make it work your way, but it costs a lot of time to do that.