5 ms·

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem.

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

christophilus 5y ago
I mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.
londons_explore 5y ago
I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend. Since Amazon failure modes are so varied, it's impossible to make the above tweaks fully automated. There'll always be a new way the cloud can misbehave, and often your service can at least maintain partial service by being nimble enough.
- technobabbler 5y ago
  If you're going closer-to-the-metal cloud, then yeah. I think that's part of the allure of newer, more abstracted cloud hosts like Vercel, Netlify, Gatsby, etc. They take care of all the infrastructure for ya, and if something breaks, you just twiddle your thumbs while they twiddle the dials. Of course this sort of laziness is not a valid approach if you're a huge enterprise or running some live-saving project that needs seventy nines of uptime, but for non-critical websites, it's a great sigh of relief...
ksec 5y ago
>making downtime somebody else's problem. It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.
- tuldia 5y ago
  > You can now simply point to AWS is down Heck, if that was even possible... "everything is green" dashboards... :-)
  - technobabbler 5y ago
    Obviously, the answer is to pay for a cloud dashboard-of-dashboards that show orange lights when AWS's is falsely green. "We need this new fancy dashboard to monitor our other dashboards in one place, more accurately!"
- SahAssar 5y ago
  Do people not take responsibility for their choice of vendors? If boeing crashed a bunch of planes and said "Not our fault, our subcontractor made shitty software" would we accept it? If a startup leaked all their customers data would we accept "Blame this other company, we just gave them all our data for analysis" as an excuse? If you accept that then it means that we need to investigate and scrutinize the whole supply chain of any product before trusting it and it would also mean that any company switching even the smallest part of their supply chain would require us to reevaluate their product. When we buy things (either products, services or software) we consider the company selling it to be responsible for the whole and deferring responsibility to a vendor is basically just saying that you were either promising more than you knew could deliver or that you did not do a good job picking vendors.
  - greedo 5y ago
    It used to be "No one got fired for buying IBM." Now it's AWS. Many CTOs are very risk averse, and having someone to blame is incredibly valuable to them. My company has a large multi-region set of datacenters, but we're using AWS for a lot of new projects, despite being able to deliver similar services with better performance and cost, on-prem. Some of the reasons are no one wants to be seen using old tech; stuck being the last COBOL dev. So there's a big incentive for developers and their managers to be using the latest tech; it looks good on the devs resume and grants them job security in the field, and it makes their managers look like they're sharp and on top of new technology. Also, recruiting people familiar with AWS services is often easier than finding someone versed in on-prem tech. And for companies in less than desirable locations, this helps remove a staffing issue since it can all be done remotely. And when something goes wrong on-prem, the CTO is expected by the rest of the C-suite to fix the issue. When it's a reputable 3rd party, he can absolve both himself and his team of responsibility.

christophilus 5y ago

I mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.

londons_explore 5y ago

I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend. Since Amazon failure modes are so varied, it's impossible to make the above tweaks fully automated. There'll always be a new way the cloud can misbehave, and often your service can at least maintain partial service by being nimble enough.

technobabbler 5y ago
If you're going closer-to-the-metal cloud, then yeah. I think that's part of the allure of newer, more abstracted cloud hosts like Vercel, Netlify, Gatsby, etc. They take care of all the infrastructure for ya, and if something breaks, you just twiddle your thumbs while they twiddle the dials. Of course this sort of laziness is not a valid approach if you're a huge enterprise or running some live-saving project that needs seventy nines of uptime, but for non-critical websites, it's a great sigh of relief...

ksec 5y ago

>making downtime somebody else's problem. It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.

tuldia 5y ago
> You can now simply point to AWS is down Heck, if that was even possible... "everything is green" dashboards... :-)
- technobabbler 5y ago
  Obviously, the answer is to pay for a cloud dashboard-of-dashboards that show orange lights when AWS's is falsely green. "We need this new fancy dashboard to monitor our other dashboards in one place, more accurately!"
SahAssar 5y ago
Do people not take responsibility for their choice of vendors? If boeing crashed a bunch of planes and said "Not our fault, our subcontractor made shitty software" would we accept it? If a startup leaked all their customers data would we accept "Blame this other company, we just gave them all our data for analysis" as an excuse? If you accept that then it means that we need to investigate and scrutinize the whole supply chain of any product before trusting it and it would also mean that any company switching even the smallest part of their supply chain would require us to reevaluate their product. When we buy things (either products, services or software) we consider the company selling it to be responsible for the whole and deferring responsibility to a vendor is basically just saying that you were either promising more than you knew could deliver or that you did not do a good job picking vendors.
- greedo 5y ago
  It used to be "No one got fired for buying IBM." Now it's AWS. Many CTOs are very risk averse, and having someone to blame is incredibly valuable to them. My company has a large multi-region set of datacenters, but we're using AWS for a lot of new projects, despite being able to deliver similar services with better performance and cost, on-prem. Some of the reasons are no one wants to be seen using old tech; stuck being the last COBOL dev. So there's a big incentive for developers and their managers to be using the latest tech; it looks good on the devs resume and grants them job security in the field, and it makes their managers look like they're sharp and on top of new technology. Also, recruiting people familiar with AWS services is often easier than finding someone versed in on-prem tech. And for companies in less than desirable locations, this helps remove a staffing issue since it can all be done remotely. And when something goes wrong on-prem, the CTO is expected by the rest of the C-suite to fix the issue. When it's a reputable 3rd party, he can absolve both himself and his team of responsibility.