5 ms·
Nobody ever got fired for using AWS.
by api 5y ago
Nobody ever got fired for using AWS.
- flatiron 5y agoIf you rely solely on east 1 maybe?
- omh2 5y agoAWS doesn't follow their own advice about hosting multi-regional. When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it. Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region
- jeremyjh 5y agoI didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.
- throwanem 5y agoIt wasn't reliable. I heard of many more who weren't able to get in that way than who were, and was in the former category myself. We didn't take any downtime, but if anything had gone wrong there would have been nothing we could do about it until IAM came back up.
- tyingq 5y agoThere are some services that do have hard US-EAST-1 dependencies. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And there's also the notion of "global endpoints" like https://sts.amazonaws.com https://sts.amazonaws.com... it's not clear why that exists, because it fails when us-east-1 does. It would be better to only have regional endpoints if the "global" ones are region-specific in reality. The endpoint thing is documented, but it's still confusing to people. The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.
- omh2 5y agoIf you're referring to Dec 7, it absolutely did. Metrics went down nearly across the board, which also means most auto-scaling setups were non-functional. Cloudfront metrics didn't properly recover until the next day Logging in with root credentials was not possible in any region, and even logging in with IAM creds in other regions yielded an intermittently buggy console and as is usual with us-east-1 outages management API calls were a complete crap shoot regardless of region
- debarshri 5y agoToday DO also went down. We could not login briefly.
- trabant00 5y agoTrue sad fact. I first thought it is a management problem but lately I see it is the tech bros who push for fads in the hopes of staying relevant and not asuming responsability for choices.
- datavirtue 5y agoOmg, this needs to be on a plaque or something. "Let's move our internal app with 50 users to k8s in the cloud." --true story
- Jupe 5y agoAmazing. And as long as "technological progress" sufficiently obscures the impact of such ridiculousness the more such projects will continue to occur. It's a real shame that the collective world of technology does not properly respect the simple solutions that work. It is almost funny the dichotomy here. Most technological people "admire" the simplicity, elegance and extensibility of the command line. But tell those same people that the best data store for the solution is a relational database and their nose crinkles up.
- datavirtue 5y agoYeah, after getting caught up in the hype for ten years I'm running back to proven tech that is flushed out (Java, Swing-omg it just works, wanting to try Ruby,even PHP is looking good at this point). Every dependency scrutinized and discarded if possible. I would probably work for free if someone setup their own on-prem cloud in Tanzu, Open shift, or Ranger and used old school proven frameworks for development. Working in AWS has been a real shitty experience at these large companies. All the nit picky problems (of which there are thousands) get dumped on devs who are trying to deliver working software.
- Jupe 5y ago(Accidentally down-voted, apologies! I would upvote to fix, but can't... Update: fixed) Agreed. Arguably, not using an existing cloud service is a red flag on any new hires. AWS being the primary, but experience using GCS or Azure are at least viable skills, even if your business is AWS-based. But the "fad-based-development" meme is not going away any time soon. The incentives in the business are built around it (really! No one want's to work on a boring old relational database solution any more). In the old days it was 4th generation languages, RUP, XML and Function Point Analysis... today it's functional programming, SDKs, big-three cloud PaaS experience or (shudder) block-chain. I think back to my much younger self, when I thought that technology was something to be mastered to solve real-world problems, and I laugh. Little did I know the real problem to be solved was to figure out how to solve those same-old business problems but with the technology of the season (Kubernetes, GraphQL or ML).
- alecbz 5y agoI wonder to what extent this actually becomes less of a problem the more people use AWS. At this point AWS being down just feels like "the internet is down", it's hard for customers to be too mad at any company being down when all their competitors are too. Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.
- AQuantized 5y agoThis seems like an interesting pendulum swing where the few companies not reliant on AWS could capture significant enough revenue by maintaining uptime during a potential busy season outage.
- ryanbrunner 5y agoThat depends on them being up the rest of the time. If they have an equal number of outages as AWS, it has the risk to make them look worse (since all attention is on them when they're out).
- hwers 5y agoThese two outages have been incredibly anomalous, I doubt you'd get much revenue betting that they'll be a common occurrence.
- tapan_jk 5y agoAgree. Now viable alternatives exist. The nextgen cloud providers will learn from the weaknesses of incumbents and innovate.
- FridayoLeary 5y agoMaybe they'll just find themselves DDOSed by the sudden influx of visitors? As a small example, i think HN was slightly slower when FB had their outage.
- pxue 5y agomaybe except a team at google? ;)
- falcolas 5y agoI have a story from only a few years ago where the finance section, and a good portion of management, of Google had no idea how poor their GAE solution was for uptime, until they tried to do business critical work using software that was hosted on GAE. Uptime improved rather dramatically after that.
- pxue 5y agoyeap. it's sloowwly getting there.