7 ms·
Is AWS/Azure/GCE down? Remember: the cloud is someone's else computer. When it's broken, you cannot do anything
by IloveHN84 8y ago
Is AWS/Azure/GCE down?
Remember: the cloud is someone's else computer. When it's broken, you cannot do anything
- dullgiulio 8y agoWould be big news to discover that GNU.org runs on some cloud hosting ...
- solarkraft 8y agoI'd rather expect it to be a problem with some other provider. Proxy, or something.
- ajeet_dhaliwal 8y agoI’d need to see some spy shots of Stallman in an Uber, talking on an iPhone and tapping out a denial on a Surface to really believe this news was true.
- nik736 8y agoSo what? I would rather have Google/Amazon employees on the issue than some random DevOps dude.
- jrs95 8y agoThis just doesn’t make sense. Google/Amazon employees basically are some random DevOps “dudes”. Whereas your own people would be...whoever you decided to hire to work on your infrastructure.
- bootloop 8y agoIt's like an insurance. You wouldn't be able to hire as many and as highly paid random DevOps on your own. So you and others pay a 3th person to do it, just in case you need a competent person.
- leot 8y agoInfrastructure (ops, software, hiring, social) matters. Hiring some dude won't give you that.
- module0000 8y agoHiring the right dude will. The trouble is, most of them are happily employed elsewhere.
- allannienhuis 8y agoat AWS/Google? :P
- lameiam 8y agoyeah, by google and amazon.
- user5994461 8y agoThe right dude if both on week end AND on holidays right now. Please wait until Monday 20th to get a status on the issue. Thank you.
- devwastaken 8y agoProblem being that Infrastructure is made to be very, very compicated because they're selling tons of managed features that have reliance upon each other. I don't know why this idea of having your own devops is suddenly now bad. renting your own bare hardware and managing solutions yourself is still very much a thing and something you have to do if you're using lots of bandwidth. Dropbox for example made their own infrastructure to get off of AWS [1]. [1] https://techcrunch.com/2017/09/15/why-dropbox-decided-to-drop-aws-and-build-its-own-infrastructure-and-network/ https://techcrunch.com/2017/09/15/why-dropbox-decided-to-dro...
- hluska 8y agoI don't think that anyone thinks having your own infrastructure is bad, or if they do, they likely don't have much experience. Rather, I think it can be a nightmare if it's not properly managed, and it's hard to develop the skill to properly manage the process unless you've been burned in the past.
- apple4ever 8y agoI wouldn’t. Hire the right person and you have immediate response instead of waiting or somebody else. A large reason we are not going cloud for our new infrastructure.
- nik736 8y agoAnd this one person never sleeps and is 24/7 on-call, right?
- geofft 8y agoand isn't about to quit once they have a little more money saved up?
- hluska 8y agoBeware of the bus factor! Going in house is great, as long as everything is well documented and your company has good backup resources.
- user5994461 8y agoIt's Sunday. Please wait for Monday office hours for your immediate response.
- zaarn 8y agoAt work we have a 24/7 20min response time clause. If the phone for work emergency calls we are ready to help in 20 minutes at any time around the clock even on Sunday. Why would you do anything else for your sysop/sysadmin?
- user5994461 8y agoYou surely realize that no human being can be available 24/7 within 20 minutes. It's beyond slavery to expect that from any employee. You need at least 10 sysop/sysadmin to achieve anything close to that SLA, with a sustainable rota. Contrary to the parent posters who believe it can be done with THE right guy.
- zacharycohn 8y agoperson
- throwawaymath 8y agoIn fairness one "random DevOps dude" might be equally capable and less expensive for your infrastructure. Generally speaking any software company can succeed without a cloud provider's infrastructure, it's just a matter of cost and developing that competency in-house. There are many site reliability engineers who specialize in high availability and downtime resolution on baremetal hardware. StackExchange notably has this competency internally.
- user5994461 8y agoSite reliability is a new fancy name for the sucker who is on call. They will change career after being forced to work on week ends and holidays a few times. Incidentally, today is a Sunday AND the most taken holiday of the year.
- throwawaymath 8y agoNot all site reliability engineers have a bad work life balance.
- user5994461 8y agoThe ones at AWS and Google, maybe not. The ones who are the single "random DevOps dude" at a small company trying to emulate AWS and Google, do have zero balance.
- xd 8y agoWell I guess the whole essence of HN is now lost... no room for some "random" startup dude to do anything that can be trusted.
- zzzcpan 8y ago> So what? I would rather have Google/Amazon employees on the issue than some random DevOps dude. This is fine if three nines of availability is all you need. Doesn't matter much if you prefer a big brand employee fixing things or a small brand employee. It doesn't change the outcome. However there are a lot of things that simply cannot live with crappy three nines availability. And the only way to do better is to stop relying on any single cloud, which inevitably requires infrastructure engineers aka random devops dudes.
- i_r7al 8y agoThe random dude is your employee and his/her mani job to get it to work and it’s first proirity. You will never be AWS/GCP/Azure first priority.
- tobylane 8y agoHow could I be their second priority? For any one service and availability zone isn't it all or nothing (or slow).
- simonjgreen 8y agoWhy would you hire just one person?
- ofj28 8y agoIf this is DNS related, which it seems to be, it has little to do with cloud based endpoints DNS has always been subject to manipulation by malicious actors or glitches in DNS server code
- ReverseCold 8y agoLast time this happened for me it was Cloudflare's fault. Google and some other really large sites worked, but not much else.
- throwawaymath 8y agoThat's pretty vacuous. Everyone's computer is someone's computer. The more important point is how capable you are at managing it yourself. What you're trying to get at is this: would you rather trust your infrastructure to a large organization whose core competency it is to do so, or would you rather manage it yourself? For many companies it makes more sense to have someone else manage it because of division of labor. If you believe you're better suited to managing your own hardware for cost or capability reasons, you should. But of the arguments in favor of that decision, pointing out that "you cannot do anything" when GCP/AWS/Azure has downtime is a pretty poor one. It's an exceptional circumstance if you're 1) able to achieve better uptime than a cloud provider, 2) at nearly the same cost (in personnel, hardware and software), and 3) while being relatively unaffected by the downtime of major cloud providers anyway. The companies for which the calculus shifts in favor of managing their own hardware probably don't need to be told "the cloud is just someone else's computer." In contrast, most companies using a cloud provider do not have a readily available alternative because they do not have in-house talent capable of maintaining baremetal hardware (local or colocated). I consider myself personally capable of maintaining a baremetal distributed system with high availability, because I presently do that. But for the most part I wouldn't encourage companies using a cloud provider to invest in their own infrastructure. It's usually expensive in personnel, time or both.
- pigscantfly 8y agoSpeculating here, but I was woken a half-hour back by an SMS from our prod monitoring system. The people at Azure had required maintenance for some instances scheduled for this morning, which I had had performed during the scheduled window over the last two weeks, but they seem to have brought down two thirds of the instances anyways. Possibly unrelated; just my two cents.
- ekr 8y agoGNU.org relying on a cloud provider? Isn't that going against the GNU philosophy a bit?