16 ms·
AWS down again?
- trevorriles 5y agoI'm still able to log in to the console. Seems to just be the AWS landing page that is down.
- tyingq 5y agoMe too. Though it does seem to be any page for that specific domain. They all return an HTTP 500 (Internal Server Error). Google "site: aws.amazon.com" and try any of the links.
- ff7c11 5y ago${jndi:ldap://AWS Down}?
- desdiv 5y agoFor those out of the loop: this is a reference to the Log4j RCE story that's also currently on the HN frontpage. Amazon uses a lot of Java, so the two stories might be actually connected somehow. [0] https://news.ycombinator.com/item?id=29504755 https://news.ycombinator.com/item?id=29504755
- netsec_burn 5y agoSpoiler: They are. AWS and Amazon both used Log4j exploitable from the homepage (/) of both sites through headers. Source: me
- perrohunter 5y agotoo soon man… hahahaha
- not1ofU 5y agolog4j? :-)
- Rudisimo 5y agoEvery time I see a situation like this one, I tell myself: "What a wonderful opportunity to go outside and smell the roses". P.S. A comment from an exhausted engineer to another. P.S.S. Seriously, I need to get out of the house more.
- kumarvvr 5y agoYes. I am trying to access the console and there is an error message. I am from India.
- raffraffraff 5y agoI've been getting the "We're sorry!" error for at least 15 minutes. No idea about regions or anything, I was just about to look up their docs about moving accounts between orgs etc. Anyone affected? Edit: Seems to be flapping between the 'sorry' error and a blank page. Thoughts and prayers with the SREs, if they call 'em that over there.
- bierjunge 5y agoSame here. And as always, AWS status page [0] is completely green and useless. [0] https://status.aws.amazon.com/ https://status.aws.amazon.com/
- my123 5y agoManagement console seems to be accessible fine to me, maybe it's just the aws.amazon.com website that's down.
- raffraffraff 5y agoOdd that a major landing page like aws.amazon.com isn't super HA. At a previous $work, the "marketing" page (ie: main dot com site) was horribly mis-managed and had countless outages due to various embarrassing reasons. Every time it was down, customers complained that the product was down (it wasn't, but people used the dot com page to get to the login page).
- Thaxll 5y agoBecause everything works fine? So a landing page does not work you need to put defcon 3?
- ziddoap 5y agoHow on earth is putting a little red X or whatever next to the service that isn't working, when it isn't working, defcon 3?
- markus_zhang 5y agoWe are sorry page is all I got.
- x0xrx 5y agoI wonder why the error message is also in … French? (I’m not located anywhere French speaking). > Désolés! > Une erreur s'est produite lorsque nous avons tenté de traiter votre requête. Soyez assuré que nous travaillons déjà à la résolution du problème que nous pensons trouver très rapidement.
- jpgvm 5y agoMine is in Japanese: > 申し訳ありません。 > リクエストの処理中に問題が発生しました。 現在問題を調査しておりますので、解決するまでもう少々お待ちください。
- themodelplumber 5y agoGotta love how polite standard business Japanese is. It's no wonder they named their moon probe after this national treasure of a philosophy.
- Waterluvian 5y agoI don't follow. It seems like an almost exact translation to all the other languages for this error page.
- themodelplumber 5y agoYou've never pondered the beauty and provenance of phrases like moushiwake arimasen or omachi kudasai, I take it? I mean, you could say they mean we're sorry and please wait, but that's also just wrong in a whole other way.
- tjpnz 5y agoJapanese has multiple politeness levels, in this particular case the message is using keigo which is the most polite. At a guess I'd say that the machine translation is identical all the way down.
- Waterluvian 5y ago
- kchoudhu 5y agoYup, my traders are getting the "sorry!" error when trying to log into QuickSight, so looks like another great day at AWS.
- redwood 5y agoStill no post mortem from Monday's right? We're all still in the dark?
- davewritescode 5y agoWhen I've worked with AWS and other providers before, they've typically given early indications about potential root causes under NDA. If you're a small customer, you probably get to wait with the rest of us but the big folks probably have some idea of the root cause.
- tr33house 5y agoCan confirm this
- naikrovek 5y ago> Still it's usually a month or more after a large outage to see the full breakdown on what happened. people expecting to see it the same week are kinda not living in reality.
- redwood 5y agoFair I guess I would have expected more in the way of official comms though. Most of what I've seen is based on deduction and speculation rather than from the horse's mouth
- markus_zhang 5y agobtw status page doesn't show anything
- CliffJumper 5y agoOFC not. They'll deny there's a problem until it goes away.
- natded 5y agoThe absolute STATE
- woeirua 5y agoAmazon’s going to have to start paying out on some SLAs if this goes on for much longer…
- deleted 5y ago[deleted]
- zwily 5y agoThere’s no SLA for the console, IIRC.
- js4ever 5y agoI see no impact on my infrastructure and the dashboard is still accessible on my side. So I'm not sure about exactly what is down.
- markus_zhang 5y agoGuess it's just the homepage then.
- rswail 5y agoI just logged in via standard console and things seem to be fine. That direct URL aws.amazon.com gives me a weird error, but everything else seems to be fine.
- kl4m 5y agohttps://console.aws.amazon.com https://console.aws.amazon.com up.
- deleted 5y ago[deleted]
- SCHiM 5y agoI think this adds some momentum to the pendulum swinging back the other way. Maybe cloud teams can patch your services better than your in-house team can (See 2 critical issues in Azure the last 3 months, _caused_ by MS itself). Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political pressures and weaselly worded SLAs (how do you mean you had an outage? Only 49% of your requests were failing!).
- tinalumfoil 5y agoEven if you can manage more uptime on your own than through the cloud (which I doubt), being on the cloud means downtime is correlated with downtime of other services. That's usually a good thing. Your customers will be more understanding if your outage is part if a wider outage that makes national news. Any services you integrate with are likely down too. If two services with 99% uncorrelated uptime together drops to about 98%. It doesn't drop if the downtime is perfectly correlated. Even if you don't directly integrate, your customer's workflow might. They see many services down and say, "cool, time to get caught up on laundry". If only you're down it's more aggrevating.
- 5e92cb50239222b 5y agoWha? We host everything on our hardware (which is nothing special) and haven't had any downtime in this year (yet). And we're just another run of the mill dev shop, very far from "superstars" who work on these (supposedly extremely stable) platforms.
- oxfordmale 5y agoTwo questions I have regarding your in-house hardware: 1. How easy can I access your physical servers ? 2. What happens if there is a catastrophic failure, for example local power outage or a major flooding 3. How secure is your server? Are you regularly patching your operation systems 4. If I want to run a project that requires double the capacity of your current hardware for a specific project, how long is it going to take to get it spun up?
- artembugara 5y agoGuess whos status page is all green?
- darrena092 5y agoThis still seems to work for me, replace region with your region: https://<region>.console.aws.amazon.com/ https://<region>.console.aws.amazon.com/
- burai 5y agoTestes this and it works fine. Thanks!
- raffraffraff 5y agoI don't think you said what you think you said
- burai 5y ago*tested
- spresso 5y agoThis worked for me too, thanks for the tip!
- hvgk 5y agoTurning my phone off right fucking now!
- ale42 5y agoIs web.archive.org using itself AWS? I tried archiving the error page, but I only get a "Sorry / Job failed" page...
- Sosh101 5y agoIt seems like AWS and Google have had more outages than normal recently. I guess this is to do with COVID disruptions? (remote working and such).
- acdha 5y agoThis morning I’d bet more on a rushed log4j patch. They use it heavily.
- x86_64Ubuntu 5y agoIt's kind of terrible that a logging library can lead to such downtime.
- bushbaba 5y agoWhen you need to force update all of prod asap. Yes it is terrible. The fact log4j even does remote network calls is crazy
- stjohnswarts 5y agowhy would remote workers matter? AWS is basically all remote.
- Sosh101 5y agoBecause change can be disruptive.
- ransom1538 5y agoWhen will people stop paying for Multi-AZ ops. You basically pay double for this - and - when one region goes down - aws is hosed.
- tomcooks 5y agoRemind me why this is better than a 10$/year VPS
- iso1631 5y agoYou can't possibly hope to match the reliability of AWS with a tiny VPS
- BenjiWiebe 5y agoMine has exceeded AWS' uptime in the last 5 years. I think. I wish I had the actual numbers. If not, it's only because I actually didn't care very much about uptime, and did an in-place system upgrade on the live server late in the evening, and when it broke I waited till the next morning to fix it. Also, the rather rare and brief maintenance window from the provider is always middle of the night for all my customers.
- easton 5y ago$10… per year? Where?
- rei_ayanami 5y agoYeah, lowest DO is $5 per month. Not even Vultr is that low.
- BenjiWiebe 5y agoIONOS has a $2/mo (or is it $2.50) option, so that's getting close to $10/yr. But that's not very capable.
- tomcooks 5y agoLearn to use search engines and translators, if you stop at Vultr you're BARELY scratching the surface.
- rei_ayanami 5y ago
- FrankSansC 5y ago(from France) at first: We're sorry! An error occurred when we tried to process your request. Rest assured, we're already working on the problem and expect to resolve it shortly. and now: This page isn’t working aws.amazon.com is currently unable to handle this request. HTTP ERROR 503
- clavicat 5y agoAnyone feel like Big Tech outages are happening more frequently recently? In recent months, we’ve seen Amazon Web Services, Facebook, Gmail, and Twitter go down. Are we at the point where the people who maintain the infrastructure are now completely different than the ones who built it, and are struggling to keep it running because they don’t understand it as well?
- tyingq 5y agoI think also just finally getting to the point where they have a tech debt burden comparable to everyone else. Starting fresh has its advantages...for a while.
- papito 5y agoEven on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.
- deleted 5y ago[deleted]
- willcipriano 5y agoIs there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.
- carlmr 5y agoI've never seen it. It's kind of like credit card debt. The people that have it get progressively more to pay off the previous debt until they reach bankruptcy. So I do think tech debt is accurate, in that the average human deludes themselves into thinking they will pay it off at some point.
- 5y ago
- 404mm 5y agoStatus page is all green. All must be ok! /s
- justintrtle 5y agoI'm not seeing any issues with my services is useast1 or uswest2.
- PedroBatista 5y agoOnly the most informed and rational companies will see how and why most of this "in the cloud" thing is a bad idea. Most will accept it as a fact of life and continue to pay for it both directly and indirectly, as long as there's cheap money going around the cloud business can't do wrong. But it's hilarious to see people indulging in byzantine "World scale" resilient systems that depend on a single vendor.
- Flankk 5y agoAre you arguing that purchasing and maintaining your own servers is cheaper than using AWS? Let's hear the rationale, then.
- mac-chaffee 5y agoYes, on-prem is generally cheaper in the long-run. For example, my company recently purchased $400k worth of hardware that would cost $80k/month if it was all EC2. There's amortized costs of the supporting infra (cooling, power) and ongoing maintenance/depreciation, but paying for the hardware in 5 months can't be beat.
- Flankk 5y agoYou forgot the salaries of the network technicians you wouldn't have otherwise. You now have to manage attacks on your infrastructure yourself. You left out bandwidth costs as well. Your servers will need to be replaced entirely every 5 years, and partially as hardware fails. It sounds like you're at sufficient scale that it makes sense. For most people, it doesn't make sense.
- eterm 5y agoOn the contrary, downtime is inevitable and I think I'd rather have downtime when all my competitors do too.
- cnity 5y agoHow about having uptime while your competitors have downtime?
- christophilus 5y agoI wonder if it's the log4j thing... AWS is built on Java, and I'd be very surprised if log4j isn't used heavily there.
- trollied 5y agoI think it is them frantically patching it. There are some Amazon staffers posting in the RCE reddit thread: https://www.reddit.com/r/programming/comments/rcxehp/rce_0day_exploit_found_in_log4j_a_popular_java/ https://www.reddit.com/r/programming/comments/rcxehp/rce_0da... > Our slack group for this issue is at 3,400 people, haha. It'd be funny if I wasn't one of them. > Where do y'all work that has 5000 employees on a single issue?? > One that has an arrow under it's name.
- zinekeller 5y agoJust went up for me, but it was indeed down earlier.
- sydney6 5y agoPerhaps AWS just hasn't paid their networking egress fees to Amazon, you know?
- tr33house 5y agoThe worst part of AWS outages is that their status page is always green. It's OK to fail but it's not OK to hide the failure and expect all the customers to stumble upon the failure especially for a company as big as AWS. Not OK!
- kreitje 5y agoI get the "We're sorry!" page on this link, but when I go to console.aws.amazon.com it works.
- koprulusector 5y agoI expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the good ol’ days (perhaps with some “member berries” https://youtu.be/mPs8-ZBSjok https://youtu.be/mPs8-ZBSjok), or you’ve just straight up forgotten what it takes to build and run your own infra from the ground up. And to those who happen across this comment before posting your own: please contribute something to the discussion beyond the above. Thanks.
- stjohnswarts 5y agoThere is nothing wrong with pointing out the warts. I wonder if you've ever been at the mercy of google/aws and their lack of customer service if you aren't representing the business $$ of a fortune 500 company?
- cosmotic 5y agoI don't think anyone is saying they could do better than AWS at making their own infrastructure. I think the complaint is that it's a single point of failure for a huge swath of the internet.
- technobabbler 5y agoY'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.
- christophilus 5y agoI mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.
- gerbilly 5y agoThe whole thing reminds me of the early nineties. There was a whole mature infrastructure built around UNIX and open standards at the time. We were told to scrap that and replace it with Windows NT and its descendants. While UNIX wasn't perfect¹, Windows was an order of magnitude worse. Compared to our UNIX machines, those systems got the internet protocols wrong, were full of security holes and crashed all the time. Objectively worse performance and security was tolerated for a decade or more, just because we all convinced each other that this was the way things were going. It took Microsoft maybe ten fifteen years to work their problems out. And now there's a Linux env built into Windows. This cloud thing isn't the end of computing history, just like Windows on the server wasn't the end of history. Cloud isn't better than on prem. It's just popular right now. 1: https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm
- robomartin 5y agoAh, the debate between managing your own infrastructure or renting it from someone else. Always a fun debate, often centering around uptime far more so than cost. I'll go ahead and make what will seem like a weird comparison. I sometimes think about this as I do about Microsoft Office. What? Over the years the MS Office suite has reached an asymptotic limit with regards to functionality users would actually be willing to pay for. I'll say that somewhere between 2013 and 2016 the suite had everything most users would need to do the usual stuff one does with Word, Excel, PowerPoint, etc. In other words, it became good enough and maybe even better than good enough. This is the parallel I see with Linux servers and the Internet infrastructure needed to run a reliable site or service. Not just the OS, but the system as a whole has, over time, approached an asymptotic limit on functionality, reliability, uptime, ease of use, capacity, bandwidth, management, etc. Not sure we are at that limit yet. It sure feels like we must be close. As many have echoed, today one can deploy a range of servers in multiple configurations and achieve extremely good uptimes with as much functionality as needed without necessarily having to go to cloud service providers. Is it the same? Not sure. I haven't operated at the highest scales in building web services, so I don't know. My gut feeling is that short of things like seriously large DDoS attacks, a self-owned infrastructure could do just as well as something hosted by a cloud provider. With knowledgeable engineers running the show there should not be any issues in setting-up, managing, maintaining and supporting such a structure. Note that I am not hating on cloud services. No. What I am saying is that because technology tends to get better over time, it is only natural that the alternative approach has gotten better and better to the point that it would be perfectly sensible for a business to consider rolling their own rather than automatically resorting to cloud providers.
- hdjjhhvvhga 5y agoI know it is not recommended, but for small (-to-medium) websites you can use DNS failover. I run non-critical services but still I like to have above-average uptime, so I have them configured in two separate physical locations by two different providers. From the point of view of visitors, they had zero downtime in the last 7 years, in spite of several hardware failures (disks, motherboard, power supply - all resolved within an hour). I know it because I tested it multiple times from various location checking what happens if one of them dies. Of course if you have millions of visitors a day, you'll do better with a more robust setup, but mine costs me €80/month.
- FerdSlav 5y agoHave a couple LightSail instances that all went offline around the time that the homepage did and still cannot connect. Everything in the dashboard says they are healthy and online but all attempts to access them (either just access the webpage or SSH) timeout. Edit: SSH access has been restored if using their web client, everything else still times out