8 ms·
what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkma
by fletom 10y ago
what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page.
they just now put up a box at the top saying "We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region."
increased error rates? really?
Amazon, everything is on fire. you are not fooling anyone
edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- evtothedev 10y agoThe AWS Status page will lie to you: https://medium.com/@ev.dev.dev/the-aws-status-page-will-lie-to-you-4c24a68d8e0a#.1vr972hcx https://medium.com/@ev.dev.dev/the-aws-status-page-will-lie-...
- fletom 10y agoI like how this post says "if you look at the AWS Status Page, this what you see". but you can't see the image. because S3 is down.
- brianpgordon 10y agoI thought this was funny so I took a screenshot of the blog post and uploaded it to the company Slack. The upload failed because Slack uses S3. This is getting crazy.
- idlewords 10y ago@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."
- mpetrovich 10y agoThere are some real gems @Pinboard too: "Green checkmark = no lava in data center. Green checkmark with information icon = data center filling with lava https://status.aws.amazon.com" https://status.aws.amazon.com"
- deleted 10y ago[deleted]
- scarlac 10y agoWhile that may be true, that's not the reason you're seeing green. You should have been seeing a broken image or a status page not finishing loading if that was an issue.
- eknkc 10y agohttps://en.m.wikipedia.org/wiki/Sarcasm https://en.m.wikipedia.org/wiki/Sarcasm
- general_failure 10y agoThat was (obviously) sarcasm :)
- artursapek 10y agoI don't think they intentionally kept the checkmarks there. They probably just didn't update it as quickly as developers made a post on Hacker News (not surprising, they were probably investigating).
- fletom 10y agowhen you start investigating an outage, that is exactly when you should change your checkmark to yellow if not red. if you're as big as AWS there should not be any more than a minute or two between when your service goes down and when you actually update your status page to show that.
- CaptSpify 10y agoIt should actually just be automated.
- hobofan 10y agoAfter having seen multiple AWS outages/service disruptions, with nothing other than a green checkmark ever showing, I am now very confident that the checkmarks are hardcoded and there is no logic behind them.
- user5994461 10y agoIt's already been confirmed by amazon employees on HN that the color can only be changed manually by an employee and it needs a high level of approval. Also, there are incentives based on colors, so the managers really don't want to admit any failure.
- KnoopKnoop 10y agoYup probably some incentives due to SLA's for their larger customers.
- mschuster91 10y ago> Also, there are incentives based on colors, so the managers really don't want to admit any failure. A textbook case of "wrong incentives". #1 incentive should be satisfied customers.
- deleted 10y ago[deleted]
- gtrubetskoy 10y agothe non-green icon is probably hosted on s3 (i'm not trying to be funny)
- deleted 10y ago[deleted]
- j2kun 10y ago> Amazon, everything is on fire. you are not fooling anyone Fun story, when I was an intern at Amazon there was actually a warehouse fire. The result was a lot of manual database entry updating as products were determined to be destroyed or still fit for sale.
- marcoperaza 10y agoI'm curious about what happened to products that were no longer fit for sale, but still fit for use. Do you recall?
- robaato 10y agoIn the military, a warehouse fire or equivalent suddenly generates a ton of "backdated transfer requests" showing that various stock had been sent to the warehouse just previously!
- dTal 10y agoThis sounds like rank corruption. Surely such a thing is rare in the military?
- marcoperaza 10y agoTo be fair, there's a plausible explanation for what robaato describes that doesn't involve corruption. Suppose it's standard or common to move things first and then file such "backdated transfer requests". After a fire that destroys everything in a warehouse, there would be a flurry of activity to quickly account for everything that was destroyed, so paperwork that would otherwise have trickled in over a month or two might suddenly be hurriedly filed in a few days.
- dragonwriter 10y agoIt could just be lackadaisical administration that only gets urgently addressed when there is something perceived as a particular problem. The military is not exactly known for being great at keeping track of things that aren't nuclear weapons, and sometimes falls short even on those.
- stretchwithme 10y agoWhat we need are status pages that are driven by votes from verified customers, which could also serve to inform the provider about issues. This would address issues that are only visible from the outside.
- laughfactory 10y agoAnd system monitoring which isn't dependent on itself. Kind of a "duh" kind of thing...
- ATsch 10y agohttp://outage.report/ http://outage.report/ does this pretty much, except for the "verified customers" part.
- kevin_b_er 10y agoIf this isn't good evidence that amazon downright lies on their status page and that no green checkmark should ever be considered trustworthy, I don't know what is.
- rinze 10y ago> edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large So this is what centralization looks like.
- skywhopper 10y agoI hear that their process for updating the status page involves S3.
- fletom 10y agoit would appear that you are correct "The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates." https://twitter.com/awscloud/status/836656664635846656 https://twitter.com/awscloud/status/836656664635846656
- kevin_b_er 10y agoI very seriously thought that it was a joke to say that S3 was needed to show the red icon, but apparently they can't update the dashboard about the status of S3 because of S3.
- knaik94 10y agoSo... https://twitter.com/awscloud/status/836656664635846656 https://twitter.com/awscloud/status/836656664635846656 "The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."
- ckozlowski 10y ago(Disclaimer: I work for AWS.) The dashboard is not changing color due to the S3 issue. We're updating the banner in place of that. Edit: Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue. http://status.aws.amazon.com/ http://status.aws.amazon.com/
- perlgeek 10y agoMaybe you could encourage your colleagues to host the status page outside of AWS?
- johansch 10y agoMaybe with GCP? :)
- oxguy3 10y agoWe'll have to wait for the postmortem, but I bet it was an unintentional dependency on S3 that no one realized had come into place until S3 went down -- especially considering how fast they were able to remove the dependency and fix it.
- ckozlowski 10y agoS3 gets used to store a lot of static content. Can't speak for that team, but I'm sure they'll take that feedback. Happy the banner functionality remained unimpeded.
- idlewords 10y agoIt took them ~30 minutes.
- LunaSea 10y agoIt took them 2 hours actually.
- whafro 10y ago"Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard." Yep.
- _ao789 10y agoI love it, like that fixes the problem! ..now fix the REAL problem
- twistedpair 10y agoWhy is the status board hosted on AWS? Most providers host such pages on a 3rd party, specifically for this reason; correlated failure.