11 ms·
Google Cloud Global Loadbalancer Outage
- cdiddy2 8y agoYup, discord impacted for me
- xb95 8y agoYes, yes we are.
- cdiddy2 8y agotime to spread resources across clouds? or would that be too expensive?
- RobinUS2 8y agothis one appears to be just global LBs, so a load balancer in AWS/Azure that hits the actual backends over a VPN or something would have worked, but that's just this case
- jkaplowitz 8y agoEven Google Cloud's non-global load balancers, which live within a single region, wouldn't have hit this particular outage.
- merb 8y agojep we had non-global lb's in europe-west-3 and i could not sense any outage.
- theclaw 8y agoApparently snapchat, spotify and some game servers affected too.
- Puer 8y agoDefinitely noticed the issues on SnapChat.
- parthdesai 8y agodiscord as well
- velox_io 8y agoThought my TV was playing up (Youtube)
- pgrote 8y agoIs there an easy way to see what cloud based resources sites are using? For instance, PushBullet is down now and this might explain why. Several other sites affected, too.
- anderspetersson 8y agocurl -I www.domain.com This might give you a clue, PushBullet seems to run behind CloudFlare though.
- CydeWeys 8y agoIf the web front-end or any external web-accessible APIs are being served directly via App Engine, you can see it in the HTTP response headers. Look for x-google-appengine-
- RobinUS2 8y agogenerally lookup the IPs (dig <hostname>) and then attempt a host/whois on that IP. For PushBullet it's hidden by CloudFlare so hard to see easily, you should try and find an exposed endpoint (which they haven't if they've done well).
- p1mrx 8y agoMy Chrome extension (IPvFoo) displays all the subresource IP addresses in a table: https://chrome.google.com/webstore/detail/ipvfoo/ecanpcehffngcegjmadlcijfolapggal https://chrome.google.com/webstore/detail/ipvfoo/ecanpcehffn... (There's a right-click option to look up each address on bgp.he.net, but that doesn't happen automatically, for privacy.)
- mcqueenjordan 8y agohttps://stackshare.io/ https://stackshare.io/ is decent for crowd-sourced info on what the stack for a given company/product is comprised of. Not guaranteed to be accurate, so take it with a grain of salt, of course.
- drexlspivey 8y agoI got a Could not connect to POP server 'pop.gmail.com:995' (SSL=ssl): SSL connect attempt failed because of handshake problems
- stanmancan 8y agoThis has been causing issues for us for the last couple of hours. We migrated a web application over to GAE last month. It's been rock solid and this is the first issue we've had. Happy to know we can sit back and let them sort it out.
- WCityMike 8y agoI suspect this is what caused the technical difficulties with HQ Trivia.
- iowahansen 8y agoGrrr. So much for global redundancy. What is going to be faster? Updating DNS records with TTL 3600 to point to a single data center or Google fixing their problem. We host DNS at AWS, but servers in GCP. Should we use AWS's automatic DNS failover feature to cover for such a case?
- iowahansen 8y agoTraffic is coming back. Looks like Google fixed their load balancer problem within 28 minutes.
- tango24 8y agoHmm, this comment says it’s been happening for hours (below). Maybe their status page isn’t accurate https://news.ycombinator.com/item?id=17552693 https://news.ycombinator.com/item?id=17552693
- partiallypro 8y agoI'm sure it was a cascading event, similar to the one Amazon had yesterday on their own site. Started small until it snowballed and effected everyone.
- jagthebeetle 8y agoI'd wait for further details from the status page, but as a GCP employee (for whatever that claim's worth on the internet), I'm not seeing evidence of an issue earlier than 12:15 PDT.
- colmmacc 8y agoAWS engineer here, I was lead for Route 53. We generally use 60 second TTLs, and as low as 10 seconds is very common. There's a lot of myth out there about upstream DNS resolvers not honoring low TTLs, but we find that it's very reliable. We actually see faster convergence times with DNS failover than using BGP/IP Anycast. That's probably because DNS TTLs decrement concurrently on every resolver with the record, but BGP advertisements have to propagate serially network-by-network. The way DNS failover works is that the health checks are integrated directly with the Route 53 name servers. In fact every name server is checking the latest healthiness status every single time it gets a query. Those statuses are basically a bitset, being updated /all/ of the time. The system doesn't "care" or "know" how many health status change each time, it's not delta-based. That's made it very very reliable over the years. We use it ourselves for everything. Of course the downside of low TTLs is more queries, and we charge by the query unless you ALIAS to an ELB, S3, or CloudFront (then the cost of the queries is on us).
- LiquidFlux 8y agoOur services have just become available again, initial downtime starting at 20:25 GMT, uptime returning at 20:56 GMT EDIT: App Engine and Kubernetes environments in our EU region appeared to go down, Compute Engine was okay.
- intsunny 8y agoWhy can't Google just use UTC timestamps? Or at least include them alongside their "US/Pacific" timestamps. I don't want to remember if "US/Pacific" currently has daylight savings or not. Its a very strange decision especially considering that GCP has numerous regions outside of "US/Pacific".
- _wmd 8y agoJust another case of their idiotic culture leaking into their products. An early "design" decision led to all their production kit being set to US/Pacific, triggering frequent DST bugs, and last I heard (many years ago) it was still the case. Coordinating a tz change over a network of that size is probably infeasible, so may as well push the pain on to customers
- stingraycharles 8y agoWhat’s the source of this? Is sounds intruiging, I wonder what led them to that decision.
- _wmd 8y agoJust ask any current/former employee, it's practically folklore by now (i.e. knowledge older than 6 months)
- puzzle 8y agoTen years ago it was called jokingly GST: not Gulf Standard Time, but Google Standard Time. I don't remember outages, but the monitoring graphs when switching to DST, where plots go back in time, did look very funky. Something like this: https://imgur.com/a/ZRtt9 https://imgur.com/a/ZRtt9 The leap second troubles in 2005 (GFS?) that led to NTP smearing were more memorable.
- reaperducer 8y agoGoogle Standard Time Wow. Just... wow. Thanks for that. That phrase brought back a flood of old memories I'd forgotten.
- hnarn 8y agoIt's interesting and scary, mostly scary, that we now are almost at the point where Google going down equals the Internet going down.
- fortylove 8y agoTo be honest, I hadn't even noticed. What popular sites went down?
- dewey 8y agoSnapchat, Spotify, Discord,...
- fortylove 8y agoThanks for the information. I see now why I hadn't noticed. I don't Snapchat anymore, I switched to Google Music, and I haven't adopted Discord. I guess I'm not in with the cool crowd anymore!
- hnarn 8y agoIf you check the other comments you'll see that Spotify, Snapchat, Discord etc. were all affected. We're not talking about "sites" but any application or site built on Google's infrastructure, even partly.
- twistedpair 8y agoPokemon Go
- Operyl 8y agoNot "every application," just applications making use of the GLB or services built upon it.
- dvh 8y agoI'm not sure if it is related but Google developer console (for Android apps) stopped displaying total number of installs today
- manigandham 8y agoThe global load balancer is one of the best offerings from GCP but we are always concerned about the single point of failure it causes. Unfortunately this isn't the first time it's broken and it's starting to look like a bad choice. We use a CDN in front so were able to switch traffic around but it seems it's better to do the load balancing ourselves too instead of using GLB.
- dsl 8y agoAny load balancer is always a single point of failure. This is why lots of folks go multi-cloud. You should also be looking at multi-CDN.
- manigandham 8y agoCDNs are single vendors but usually have a resilient and distributed system without a SPOF within, meaning traffic can always route through even with major network outages. This has been a core part of their design for a long time and I can't remember the last time an entire CDN has been down. GLB is also distributed but clearly is a single "service" within GCP with weaknesses that can take down the entire thing, probably because of the complexity and integration involved in its architecture. It's fast and convenient but the reliability just isn't there yet.
- p0rkbelly 8y agoI have yet to read solid case studies of real multi-cloud at scale. E.g. Active-Active load-balanced between multiple providers. Plenty of companies use multiple clouds, but, it tends to be a line of business decision. Team A likes AWS, Team B like Azure etc.
- manigandham 8y agoAgreed, multi-cloud is rather costly and means only using infrastructure services like VMs and storage.
- user5994461 8y agoIt's fairly common. No need to call it multi cloud, companies have had multiple datacenters in multiple countries for a long time. The only challenge is that you need global geographic load balancers and that means F5 and at least a million dollar. Also, you will find out later that some dependencies were only running in a single location and services failed with the datacenter.
- EZ-E 8y agoStill having issues as of 20:20 UTC - Firebase Database messages not beeing delivered to client. It's been a full hour so far. The delivery rate also had a minor drop earlier around 03:00AM to 05:00AM but no incident had been listed.
- qaq 8y agoIs there a single cloud that can match uptime of a single top tier DC?
- jasonvorhe 8y agoUhm, no, because a cloud service spans over several data centers and features a lot more moving parts than just a solid data center. Let's turn this around: could a typical data center + server uptime + service uptime equal that of a major cloud provider?
- qaq 8y agoIn my experience yes but too few data points.
- avip 8y agoSomething no one has mentioned yet, could it be that the engineering force at Google is no longer what it used to be?
- jasonvorhe 8y agoBecause of one data point? Seriously?
- rrampage 8y agoI think OP was jovially referring to a similar comment regarding Amazon because of their recent Prime day outage.
- throwaway923842 8y agoI sure wish we hosted on AWS today. Work sucked big time.
- beilabs 8y agoCould have sworn I was going crazy last night; was attempting to write a scraper to download attachments / tickets from pivotal tracker for a yearly accounting report. Everything was working fine, then suddenly not. At around 2am I gave up and just put it down to the 'gods' telling me to go to bed. This was the first article I saw after I woke up, somewhat refreshed. /feeling-vindicated
- sidcool 8y agoWow, 2 AM, some persistence. But as someone who's very sensitive to sleep deprivation (I get violently ill if I don't sleep well for more than 2 days in a row), I insist not making it a habit. Especially if you are in your 30s, it has bad after effects. PM for more info.
- voltagex_ 8y agoCan't PM with no contact info in your profile - email is not public - this is HackerNews.