17 ms·
Google Outage in Europe
- everydaypanos 5y agoreCaptcha and translate are also down. Even search is affected it is super slow.
- jbverschoor 5y agoYeah had recaptcha issues too
- jacquesm 5y agoAnd meet. Meetings starting is super slow, sometimes it times out.
- mschuster91 5y agoGoogle Forms definitely has issues, too. I just had an issue where I couldn't fill out a form.
- synchton 5y agoIs it me or are big provider outages getting more common these days?
- that_guy_iain 5y agoI think they're reasonable common it's just becoming more of a think to notice.
- scrollaway 5y agoAs time passes, the more big cloud providers there are, and the more complex they individually get (more products). Assuming the chance of an outage is actually static. If there is one (highly reliable and trusted) provider with five products one year, and three providers with ten products the next year, the chance of you seeing an outage has gone way up because of surface area.
- querez 5y agoExcept that Google services don't run within GCP, but apart from it (mostly on Borg I'd guess).
- scrollaway 5y agoThe above applies to cloud as understood by techies just as b2c cloud as understood by laypeople.
- jsty 5y agoTheir internal platform is likely analogous in terms of accumulating complexity (and a bit of cruft) over time though
- nefitty 5y agoIf anyone else is curious, Borg is Google's cluster management software: https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/ https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...
- deepstack 5y agoMore reason to promote for self-host, and decentralised systems like smtp, matrix.org, ActivityPub. The idea of having all the data and server concentrated in a few player such as google, amazon, ccp is not reliable for the digital operation of the planet.
- altgoogler 5y agoDisclaimer: am Googler. Self-hosting is a good option, especially when mixed with multi-cloud offerings. The big rub is that it's really hard to approach the same level of availability the cloud offerings already give you. Depending on work-load, self-hosting is typically more expensive. This is why the SRE book talks about availability budget. You can't have 100% uptime, so how much do you want to pay to get close to it?
- kertoip_1 5y agoThe question is: should we trust small, underfunded, hobbyist servers more than large corporations that have a money-driven reason to maintain high quality of services?
- sva_ 5y agoAlmost like bad weather. Perhaps we'll have cloud outage forecasts at some point.
- yunohn 5y agoThere’s always this question on every “X is down” post on HN. Quite boring commentary tbh.
- afc 5y agoI think it's just you, probably recency bias. I recall the days of Gmail being globally down for hours, more than once (3 times, IIRC), back in 2009.
- The-Bus 5y agoI also can't think of the last time I saw Twitter's Failwhale, which almost became their default mascot at one point. (Yes, I know they discontinued it, but I haven't seen the service "over capacity" in some time. I say this as a casual web user of it, not an active account holder or app user).
- riffic 5y agoTwitter had a global outage this year (April 16). Facebook had an October outage. This happens quite regularly with distributed and complex systems.
- deleted 5y ago[deleted]
- riffic 5y agoask anybody who researches complex and distributed systems and see what they say.
- 1cvmask 5y agoAnd many people blame their internet provider as google outages are rare.
- vanderZwan 5y agoI don't think it's that so much as the fact that that half of the internet services that rely on Google along the line somewhere (but not visibly to the users) are down, at least for me. And yet the Google search page can be reached (again, speaking from my European perspective, sample size one). To make it even more confusing: these issues are limited to my WiFi. If I use my phone connection as a hotspot the problems disappear. So from the perspective of a regular person it is very reasonable to conclude that it is their internet provider that is down.
- kaba0 5y agoAnd I think no internet provider has such a good name where an outage on their part would be unheard of. Much more so than google.
- ratww 5y agoNow that I think of it, I might have blamed my provider or at least restarted my modem if Hacker News was ever down...
- rablackburn 5y agoIt’s the perfect connection test. 9 times out of 10 if you can’t connect to hn the problem is you, not them. Probably even 99/100
- paganel 5y agoI usually check the HN website to see if my internet is down, both on my mobile and on my work computer. I used to just type "test" in the browser address bar hoping for the related Google SERPs but checking for HN seems way faster nowadays.
- 5y ago
- intunderflow 5y agoI felt a great disturbance in the force, as if millions of pagers suddenly cried out in terror
- deleted 5y ago[deleted]
- kroolik 5y agoThere were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages https://en.m.wikipedia.org/wiki/Google_services_outages
- alpb 5y ago> evaluation and promotion window ends around december Googler here, that is not when those promotion/perf evaluation windows happen.
- oblio 5y agoHe he. It's kind of hilarious. Maybe they should stagger reviews to ensure high availability? :-D
- kroolik 5y agoNothing wrong with it. People make errors, distributed systems aren't easy. More frequent changes - more likely a bug was introduced. My post is just a speculation, let's wait for the actual technical details doc from Google.
- oblio 5y agoWell, there are various whitepapers that reformulate this exact truth you've highlighted: issues tend to happen more when changes are made. Regarding the technical details doc, Google will never state that outright in individual postmortems. And they will definitely not draw this to the logical conclusion regarding the spiky yearly activity.
- yunohn 5y ago> logical conclusion regarding the spiky yearly activity Why is it logical? There’s tons of changes being deployed at all times, at all large companies. Across products, verticals, everything - hundreds of changes at any given point. Some of these changes can introduce hard-to-predict bugs in globally distributed systems. Most of the time, external users don’t even notice before they’re fixed. Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies.
- tomudding 5y agoProbably related to the Google Cloud "service disruption" that started earlier this morning: https://status.cloud.google.com/incidents/1xkAB1KmLrh5g3v9ZEZ7 https://status.cloud.google.com/incidents/1xkAB1KmLrh5g3v9ZE... It was very frustrating to see DNS working perfectly fine (since DNS is always the problem) and the connections just timing out.
- demosito666 5y agoI was also getting 500 error in GCP console
- ddek 5y agoYep, we're hit by that right now. It's not a total outage, we're losing about 10-15% of our calls to bigquery from within cloudfunctions. What we have using a VPC connector is ok. Fortunately areas affected are ancillary (mainly monitoring, ironically), our service is still running.
- peteri 5y agoDocs was a bit flaky earlier on.
- elp 5y agoThe Google imap service has also been unstable in South Africa this morning.
- ulzeraj 5y agoDidn't even noticed. Work doesn't use any GCloud components and personally its been a while since I've degoogled myself. No Google docs, mail or search. Not even the quad 8 nameservers.
- yosito 5y agoNot sure why you got downvoted. I'm also a degoogled European and didn't even notice.
- rossdavidh 5y agoDownvoted from people who are jealous, perhaps?
- dredmorbius 5y agoAlso reported / dupe here: https://news.ycombinator.com/item?id=29197479 https://news.ycombinator.com/item?id=29197479
- nudpiedo 5y agoSeems that google is not anymore the company it used to be advertised. The myth prevails though. It’s a pity as some of its services are frankly among the best, and I depend on them.
- southerntofu 5y ago> Seems that google is not anymore the company it used to be advertised. Do you mean about the "Don't be evil" slogan it started with? Even back then, it was a pretty obvious move for any movie villain. Or do you mean you actually bought into the reliability promises of cloud providers thinking downtime would not exist anymore?
- nudpiedo 5y agodue their reliability and over the tp. It always overfelivered. during its first decade, If anyone read any of the books from its founders, papers, stories related to the company and them it seemed to be the new "non zero day" infrastructure, however the longer the company alive is the more it feels just like any other. Unrelated: not sure what is going on on HN since a few months but almost all comments I get to write get automatically downvoted, even little things that no one should really care much about.
- bla3 5y agoLooks like it might be resolved now.
- sebow 5y agoGood Maybe these outages will make people realize how dependent they are on a few monopolistic companies.Not only these will make rise of competitors but also hopefully make people's information more independent
- rzr 5y agoLet me crosslink to invidious tracker: https://github.com/iv-org/invidious/issues/2577 https://github.com/iv-org/invidious/issues/2577
- tobyhinloopen 5y agoWe just released a rewrite of an (web)app today so lot’s of support calls and mails. Really unfortunate timing, hah
- rossdavidh 5y agoSo what you're saying is, you brought down Google with your rewrite. :)
- intsunny 5y agoOutage reports in UTC, amazing! (glares at AWS)
- tex0 5y agoHahaha. This one made me laugh hard.
- andrelaszlo 5y agoNo it will make you laugh in about 30 minutes. Oh wait, DST. Never mind.
- NaturalPhallacy 5y agoI want to smack whomever invented time zones. Everyone is bad at them. Trading desks, international trade offices, space related offices all have a set of clocks on the wall because even really smart people just suck at figuring out time zones. I even use https://everytimezone.com/ https://everytimezone.com/ in lieu of a set of clocks. What a needless complication. I wish we could all just switch to UTC and stop daylight savings time.
- judge2020 5y agoI can't imagine anyone taking kindly to a change as big as eradicating time zones. Get rid of DST, yes, but only if the U.S. goes up an hour (the extra sunlight at 6pm during DST is nice).
- NaturalPhallacy 5y agoI can imagine hordes of programmers everywhere taking kindly to it. It's a simpler system.
- markusgammer 5y agoWoW
- iso1210 5y agoI do laugh at people that say "Using the cloud means less downtime than your own server", then things like this come along :D
- HatchedLake721 5y agoIt’s still less downtime than your own server and you have hundreds of engineers 24/7 ready to diagnose and fix issues
- iso1210 5y agoReally, downtime of google so far this year: More than 0 minutes. Downtime of my own server: 0 minutes. Of course I'm not trying to run a massively scalable service coping with millions of customers, because I don't need that.
- mekkkkkk 5y agoYou think that's how statistics works? Obviously it's possible to keep your own server running with 0 downtime. But you are exposed to a much higher risk of severe downtimes longer than the cloud provider would likely be. Be it hardware failure, grid, ISP, whatever.
- iso1210 5y agoClaim was "It’s still less downtime than your own server" Claim disproven
- mekkkkkk 5y agoThat claim was a direct snippet from your original comment. So unless you meant that you laugh at people who are talking specifically about your personal server, I'd say you are being selectively literal.
- 5y ago
- hulitu 5y agoMaybe they are just upset because they got fined. It's common by big companies.
- southerntofu 5y agoOr maybe this is unrelated and it's just one of the many cloud outages contradicting their uptime promises. Welcome to the reality of cloud computing, where everything's foggy and the hallucinogenic fumes made us believe downtime was a thing of the past.
- gabkins 5y agoI do wonder how many companies and businesses got affected by the outage.
- natch 5y agoEU to AI: Here's a €2.4 billion fine for you. AI to EU: OK, human.
- blauditore 5y agoITT: "My rasperry has been up for 2 years with almost no downtime, what is up with these suckers?"
- Nextgrid 5y agoIt's valid criticism for projects that don't need the scale or features that cloud-based solutions offer. A single machine won't give you the same level of scaling and management features, but also won't have hundreds of distributed moving parts that could break and take your service down as a side-effect.
- Spivak 5y agoI feel like you are selling the cloud short for SMBs. The majority of my career has been spent working for shops that have their own datacenters or colo. It's so freaking cheap to run a huge service it's not even funny. And it's a dream to have so much compute and storage floating around that you can "waste" it without even thinking -- it's already paid for! However, when your small the economics flip. You want to run as little as possible as cheap as possible. And a load balancer pointing at a two asgs in different availability zones is by far the cheapest way to get something production ready on the internet. You can go from your garage to a 100M business and probably never need to graduate from this architecture.
- remram 5y ago"For a Linux user, you can already build a Google yourself quite trivially with wget -R and grep."
- ac50hz 5y agoI guess the network admin who skipped the classes on Network Fundamentals, has moved on from their recent placement in OVH, to Facebook, then to Google. Next stop AWS, firewall maintenance division?!
- yosito 5y agoDegoogled European here. Genuinely didn't even notice. Google can fuck off.
- deleted 5y ago[deleted]
- bob229 5y agoGood then cancer machine is down for a while