6 ms·
Show HN: AWS S3 outage test from across the world
- rbinv 11y agoI just re-ran the test and got an error rate of 1.23% (vs. 41.98%): https://pulse.turbobytes.com/results/55c88a0fecbe400bf800073e/ https://pulse.turbobytes.com/results/55c88a0fecbe400bf800073... edit: S3 seems to be back up and running according to the AWS status page.
- cddotdotslash 11y agoGot some alerts from a couple services that rely on S3 this morning. Perhaps this is related, but everything is back up for now.
- Sami_Lehtinen 11y ago"Oh no, our server made a boo boo. Please try again."
- chr15 11y agoIs there any really way to design your application to handle S3 failures like this? S3's SLA has 99.99% availability, but is there a way to handle the 1% so your application is not affected? Options I can think of: 1. Using a CDN to serve files can help in some cases 2. On-prem systems may be able to use gateway-cached volumes and use the local disk cache vs S3 Other ideas?
- lukasm 11y agoreplicate to different data center e.g. Azure.
- ohitsdom 11y agoSecond this, I would love to hear how companies handle S3 outages. Although one correction, it's 0.01% that you need to handle (if they deliver on their availability promise). That's less than an hour a year.
- dbarlett 11y agoCross-Region Replication http://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html http://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html
- toomuchtodo 11y agoWhich works if you fail to read-only, and simply switch to reading from the non-failed region. More complicated is attempting to continue writing to a non-failed region and then attempting to bring everything consistent once the failed region is available again. EDIT: If you were using hashing/uuids/guids for objects, this should be possible using a background task that'll scan various buckets in multiple regions and move objects when/where necessary to return to a consistent state.
- merb 11y agoMulti Cloud. Use S3 + Azure Cloud Files + Google Cloud Files
- untog 11y agoSlightly OT, but there's an interesting phenomenon at work now that so much of the internet depends on Amazon's infrastructure. When it goes down you might not even need to worry about it that much, as so many sites/apps will be broken that most users will just assume that the internet is broken.
- alpb 11y ago
- imrehg 11y agoYeah, I got 2.53% error rate, but that's nothing to worry about - using 79 servers, that's exactly 2 errors, both of them in China, which kinda makes it feel less Amazon's fault, than the Great Firewall's.... Maybe there should be some more meaningful error measure than the raw failure percentage.
- mechadock 11y agoError Rate: 41.98% Response Time (avg): 5,518 ms for me.
- sajal83 11y agoclick "Run test again". The OP link is test from 5 hours ago.
- scott_karana 11y agoVancouver, Tokyo, Cebu, Singapore, Bangkok, and Kharkiv all show EOF. Paris, Roubaix, Manchester, Portsmouth, Budapest, Tokyo, Bilthoven, Vleuten, Utrecht, Manila, Sovetskaya Gavan, Tyuven, Singapore, Bangkok, Taipei City, Kiev, Kherson, Ashburn, Mountain View, Rowley, amd Newark all show 503s. It's more than just the Great Firewall. :-)
- imrehg 11y agoIt depends, you have to re-run the test...