22 ms·

Google Cloud Is Down

1395 points by markoa 7y ago

https://status.cloud.google.com https://status.cloud.google.com https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003 Status page reports all green, however the outage is affecting YouTube, Snapchat, and thousands of other users.

dang 7y ago
Related: https://news.ycombinator.com/item?id=20077275 https://news.ycombinator.com/item?id=20077275 https://news.ycombinator.com/item?id=20077293 https://news.ycombinator.com/item?id=20077293
PerfectElement 7y ago
Just 2 weeks after I migrated a DB cluster from Azure to Google Cloud thinking things would get better.
- duality 7y ago
  FWIW they might still be.
- AaronFriel 7y ago
  Just remember that "better" can be entirely based on when you pick the starting and ending points of your graph ;)
  - XCSme 7y ago
    It's just like the stock market.
- user5994461 7y ago
  It's the yearly outage. You will be fine for next year. Better than the monthly outage from Azure.
leowoo91 7y ago
0 issues at compute, reporting for europe-west3-b,
ksajadi 7y ago
GCP status page is worthless as it's always happy and green when production systems are down and then they might acknowledge something an hour later
- JimboOmega 7y ago
  Just like AWS, then. "Some users are experiencing increased error rates" = "Everything has been down for hours"
  - bsimpson 7y ago
    "Everything is fine, unless you're Carl. There's a massage outage, but only at Carl's house. Sorry, Carl."
    - notatoad 7y ago
      I'm also experiencing a massage outage. Please send masseuse.
      - bsimpson 7y ago
        Goddammit some (most?) days I can't type. "Massive"
        quickthrower2 7y ago
        I got the missive. Thanks.
  - djsumdog 7y ago
    I remember when S3 was down and the status was green because the updates for the status page with pushed via S3.
    - chmhsm 7y ago
      That's not just ironic, that's stupid. How do you count on S3 to update S3 status? Isn't that a huge design fault?
arach 7y ago
Confirming issues on our end. I'm able to load up my console but when I go to Kubernetes Engine, I don't see my clusters. I'm monitoring closely on twitter
walshemj 7y ago
just had rolld20 in the USA blow out a game I wonder if it is effected
- zoidq 7y ago
  Looks to be working in the UK
  - walshemj 7y ago
    Network peering probs in Canada - looks like an unrelated problem
RappingBoomer 7y ago
youtube streaming is also down
horyzen 7y ago
Yeah I was having trouble accessing my Gsuite apps, had a couple of 502s, which led me to check HN. While it doesn't give me 502 now, it's abnormally slow.
nkassis 7y ago
The GCE console also affected, couldn't send a support ticket just getting errors.
darkof 7y ago
That feeling when you open https://console.cloud.google.com https://console.cloud.google.com and see that you don't have your Kubernetes clusters and CloudSQL databases, but CTA to create first.
- vpontis 7y ago
  Gosh, this was so scary... I thought someone had hacked in and deleted everything... I hope they come back. This is still pretty scary
  - stnmtn 7y ago
    Same, my Manager called my and said "everything is down". So I wander over to my Firebase console, and there's no database loading. Thank god for twitter, and people also saying that they have the same issue or I would have for sure though we've been hacked. I hope this is a good wake up call for everyone. I know that I'm going to think more about how we do backups and fail-safes
  - hanniabu 7y ago
    And here I thought I was having a bad day with Google Play not loading
  - jjeaff 7y ago
    Same. I was thinking, oh, my db cluster must be having trouble recovering. Couldn't get any response through kubectl. Logged in to the cloud console and it looks all brand new, like I have no clusters setup at all. Of course, this is 2 weeks after switching everything over from AWS.
  - dpau 7y ago
    my vm instances are all still there, can even log in via SSH in the compute engine tab. looks like they got a reboot 15 min ago. just restarted some processes but lost my progress on about 12hrs of computing time, i'm guessing it's going to be hard to get a refund..
    - 7y ago
codys 7y ago
Other google services are also affected: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status https://i.imgur.com/pcqwwA4.png https://i.imgur.com/pcqwwA4.png
- mixedbit 7y ago
  Good that Google+ is up again
- deleted 7y ago
  [deleted]
digaozao 7y ago
I cant see any gke cluster in Brazil, or any VM.
- miller_joe 7y ago
  I'm seeing that with northamerica-northeast1. I can't access anything over the network in that region and most of the GKE clusters and VMs in that region aren't listed in the console
stuck_in_the_ma 7y ago
Google Play is also experiencing massive issues.
snisarenko 7y ago
The status page took a while to show issues. My app was down, and Twitter knew google cloud was down before the official status page.
deleted 7y ago
[deleted]
vinayan3 7y ago
Looks like only GCE is down according to the status page now. I'm able to access my console for instances and GKE clusters.
- snisarenko 7y ago
  My site runs on Google App Engine and its down as well.
- sladey 7y ago
  I happened to be initializing a GKE pool upgrade just as this occurred. The upgrade is now stuck according to the console. The interesting thing is that a couple of minutes before everything went wrong, kubectl returned a "error: You must be logged in to the server (Unauthorized)" error
- miller_joe 7y ago
  They initially opened an incident against GCE (https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003) then opened one against Networking (https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...). The networking incident looks like the one to follow for updates now.
murat124 7y ago
GCP has been down since 11:50am and they acked it 35 mins later. They're great at leaving their customers in the dark.
- copperx 7y ago
  Not much different from AWS, from what I've heard.
  - numbsafari 7y ago
    Definitely the case. Neither are super great at this. One issue is that issues that may 100% impact individual clients may only impact a vanishingly small amount of their overall service load. That mismatch between customer and provider experience is one of the ugly aspects of public cloud providers.
    - kazen44 7y ago
      Also, it's one which gets hugely understated when people "move to the cloud". especially if you use your bussiness for B2B services. Stuff like this could make you loose your bussiness, especially if some entity like google doesn't communicate and as a result, you do not have a answer for your own customers. Medium sized private cloud providers are a lot better at this, considering the communication lines are a lot shorter.
      - zepolen 7y ago
        On the flip side - a customer is more willing to be understanding if 'Google is down' instead of 'our server is down'.
    - discodave 7y ago
      That's why AWS is all about their Personal Health Dashboard (PHD). They can post specific issues for your account in there. Also, they get to keep the public page looking nice and green to show to executives of prospective customers.
  - 7y ago
xerxes901 7y ago
Couldn't load the support console to "me too" this one either!
macintux 7y ago
And thus was ruined hundreds or thousands of pleasant Sunday afternoons. I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.
- xerxes901 7y ago
  Spare a thought for the pleasant Australian early Monday mornings too! Always a rude awakening...
- jagtesh 7y ago
  Multi-cloud for those times when you really need that level of availability and can afford it.
  - jjeaff 7y ago
    It's not even about being able to afford it. Some things just don't lend themselves to hot failover. If your data throughput is high, it may not be feasible or possible to stream a redundant copy to a data center outside the network.
    - EugeneOZ 7y ago
      All parts of the system should be copied (if you decided to build multi-cloud system), not just some of them.
- newsbinator 7y ago
  It's the Queen's birthday, a Monday off here in New Zealand... ... but not for everybody now.
  - Scoundreller 7y ago
    So what happens when the crown changes? They change the holiday? Immediately? For the next year? Sounds like a bit of a nightmare.
mandatory 7y ago
Yep, I can no longer see my Cloud SQL database - it's as if I've never created one at all. Really hoping this is just an issue displaying it and that Google hasn't punted my infrastructure and backups.
- vpontis 7y ago
  Praying isn't working. Now, I'll try sobbing :(
  - Havoc 7y ago
    Systematic problem solving. I like it
  - dodobirdlord 7y ago
    Reports claim that it's a network congestion issue primarily in the northeast United States. It seems doubtful that any data has been lost, but you probably can't see resources due to requests not getting through. I hope that hearing this helps you to feel better.
- flamingcow 7y ago
  [I am the Cloud SQL tech lead] This is a networking issue, and your data is safe. Cloud SQL stores instance metadata regionally, so it shares a failure domain with the data it describes. When the region is down or inaccessible, instances are missing from the list results, but that doesn't say anything about the instance availability from within region.
tr33house 7y ago
Took me a while to track latency issues to GCP. Wasn't expecting it. This also seems to affect some GAE instances and some of their products like google photos. At least according to my observations
- different_sort 7y ago
  I see this as well.
bilal4hmed 7y ago
gitlab is slow too
- amenod 7y ago
  Slow is understatement... some pages on gitlab.com take minutes to load, and jobs take tens of minutes to start. EDIT: It's been like that since at least 12h ago though. Not sure if it's connected to Google Cloud?
  - dsumenkovic 7y ago
    Yes, it was definitely related to GCP services. GitLab is no longer seeing errors and Google Cloud has resolved the issue as of 23:00 UTC yesterday. Any further information can be found on the issue at https://gitlab.com/gitlab-com/gl-infra/production/issues/862 https://gitlab.com/gitlab-com/gl-infra/production/issues/862
    - amenod 7y ago
      The timeline seems a bit short, but ok. :) (the problems with runners and the UI started at least at 2019-06-02 7:48 UTC, though they were hit-and-miss at the time) Still, happy this is solved and we can use the (fantastic otherwise) service again!
klon 7y ago
Anyone using both AWS and GCP that can form an opinion on availability of both? As a GCP customer I am not very happy with theirs.
- pm90 7y ago
  GCP is incredibly bad at communicating when there are problems with their systems. Just terrible. Its only when our apps start to break that we notice something is down, then look at the green dashboard which is even more infuriating.
  - timdorr 7y ago
    AWS is often the same way. No one seems to be good at communicating outage details.
    - TallGuyShort 7y ago
      I suspect there's a correlation between outages that are easy to detect and communicate and outages that automation can recover from so easily that you hardly notice.
      - deleted 7y ago
        [deleted]
    - obeattie 7y ago
      I really don’t get this. There’s a huge number of complaints about poor communication from companies like Google and AWS during every outage. Yet they remain seemingly indifferent to how much customer trust they are losing, and the competitive edge the first one to get this right could gain.
      - atmosx 7y ago
        If 20% of Google Cloud's customers leave after this outage because of poor communication they'll prioritise accordingly and apply all that nice SRE theories to their infra. But this isn't happening, because <various reasons>, so... who cares?
landon32 7y ago
u.s. west: all our cloud compute is inaccessible rn.... our API is down, can't ssh into the servers, and also can't see them on the dashboard.
colinbartlett 7y ago
Google Cloud is the number 4 most monitored status page on StatusGator and Google Apps is number 12. In addition, at least 20 other services we monitor seemingly depend on Google Cloud because they all posted issues as soon as Google went down. It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up.
- hazeii 7y ago
  ...and only the paranoid survive?
  - MrMorden 7y ago
    Just because you're paranoid, it doesn't mean they're not out to get you.
- FPGAhacker 7y ago
  I guess we know what steam uses (the store at least).
  - xNevo 7y ago
    No issues for me. Maybe they have a failover mechanism?
    - FPGAhacker 7y ago
      yeah, maybe it was coincidence. seems to be back up for me as well: https://steamstat.us https://steamstat.us
  - colinbartlett 7y ago
    I don't know about Steam, but I know Apple must use Google Cloud: https://www.apple.com/support/systemstatus/ https://www.apple.com/support/systemstatus/
    - 7y ago
tus87 7y ago
So that's why I can't login to YouTube this morning...
different_sort 7y ago
I was playing around this afternoon with appengine, and thought I broke one my projects when I started getting 502 back. There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me. The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.
deleted 7y ago
[deleted]
derekhh 7y ago
> We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific. I'm not seeing anything at 12:47.
- derekhh 7y ago
  https://status.cloud.google.com/incident/cloud-networking/19008 https://status.cloud.google.com/incident/cloud-networking/19... 404 - Impressive
  - ryacko 7y ago
    Cloud status dashboards seem to be hosted on the same cloud, which doesn't say much about redundancy.
    - chupasaurus 7y ago
      AWS changed internals of Service Health dashboard after they couldn't update it when S3 went down in us-east-1 (https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/) edit: wording
      - ryacko 7y ago
        Someone had to design the status page, and failed to anticipate the issue about how the status page depends on the systems it is reporting on. Unlikely the rest of AWS, a cached web page does not require much complexity.
- timdorr 7y ago
  They are updating the root cause issue, which is networking, here: https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... Next update is in about 25 minutes.
rwbhn 7y ago
gmail also down/super slow atm for me (East Coast, USA)
pgoodjohn 7y ago
Everything looking normal on our GKE / CloudSQL stuff (eu-west1)
- wichert 7y ago
  gcloud tells me: WARNING: The following zones did not respond: us-west2, us-west2-a, southamerica-east1-c, us-west2-b, southamerica-east1, us-east4-b, us-east4, us-east4-a, northamerica-northeast1-c, northamerica-northeast1-b, us-west2-c, southamerica-east1-b, northamerica-northeast1, southamerica-east1-a, northamerica-northeast1-a, us-east4-c. List results may be incomplete. Luckily for us eu-west1 seems to be working normally.
echelon 7y ago
Nest is down for me right now. It's amazing how far-reaching outages can be these days.
- unicornmama 7y ago
  There are dozens of us.
- andrewprock 7y ago
  Code reuse is a wonderful thing, until it's not.
  - hn23 7y ago
    Code reuse? Couple if years ago some started to call servers "cloud" but the cloud is just that: computers managed by someone else. If the whole service layer relies on Three big companies, there is a problem..
squarefoot 7y ago
And Gmail too doesn't feel very well today. [21:55:19] POP< +OK send PASS [21:55:19] POP> PASS ******** [21:55:21] POP< +OK Welcome. [21:55:21] POP> STAT [21:55:21] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later.
- ToniCipriani 7y ago
  Pretty much Gsuite is out: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status
  - megous 7y ago
    At least G+ is working. /s
- hazeii 7y ago
  IMAP as well - for some considerable time now.
  - agentile 7y ago
    Yes experiencing IMAP auth/connectivity issues here as well, along with outgoing SMTP
benbristow 7y ago
Wondered why Snapchat was being weird today. Thought it was my pi-hole setup blocking something from working, but nope, it's Google!
brown9-2 7y ago
When talking about GCE being down please also mention what regions you are talking about
- chupasaurus 7y ago
  In this case it's a luck if any are working correctly, a problem is global with some exceptions.
  - brown9-2 7y ago
    seems to be some comments here of some regions functioning ok, although perhaps it’s not 100% in all regions
- sgammon 7y ago
  us-central1 us-west1 us-west2 is what I’ve heard so far. east seems to be OK, and Europe too
  - digaozao 7y ago
    South America is down for me. But youtube works OK, I can watch Google IO and they talking about all Google greatness while their outage impact second class clients on their cloud.
javabean_ 7y ago
vimeo.com is down.
boulos 7y ago
Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!). We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages. There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. You'll have to wait for someone actually involved in the incident to say more.
- odiroot 7y ago
  > including unfortunately the tooling we usually use to communicate across the company about outages. There's some irony in that.
  - boulos 7y ago
    Edit: and I agree! I’m not in SRE so I don’t bother with all the backup modes (direct IRC channel, phone lines, “pagers” with backup numbers). I don’t think the networking SRE folks are as impacted in their direct communication, but they are (obviously) not able to get the word out as easily. Still, it seems reasonable to me to use tooling for most outages that relies on “the network is fine overall”, to optimize for the common case. Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. The Networking outage is the root cause.
    - marksomnian 7y ago
      > the status dashboard now correctly highlights that multiple things are impacted because Networking. this column of green checkmarks begs to differ: https://i.imgur.com/2TPD9e9.png https://i.imgur.com/2TPD9e9.png
      - boulos 7y ago
        The banner at the top. Sorry if that wasn’t clear.
deleted 7y ago
[deleted]
wenbin 7y ago
btw, Google Analytics realtime is down as well.
- bbayer 7y ago
  I wasn't aware that outage and had small heartattack when I saw huge drop of visitors. I think other metrics are also affected.
- mileycyrusXOXO 7y ago
  Thanks, I noticed I didn't have any users after pushing an update and thought I broke something.
sdan 7y ago
Github contribution graphs are also gone
sadris 7y ago
So is Youtube.
deleted 7y ago
[deleted]
mtarnovan 7y ago
Is shopify on google cloud? i noticed they are having issues too
- brown9-2 7y ago
  yes
  - captn3m0 7y ago
    I'd reported a networking issue to Shopify yesterday, which they said was resolved upstream (GCP?). Could it be?
- drchiu 7y ago
  Yes it’s affecting their system as well.
- crispyporkbites 7y ago
  Sunday evening is a pretty big time for ecommerce. Losing 1000s of dollars of sales here :-(
40acres 7y ago
Can't wait for the postmortem!
- murat124 7y ago
  My money is on config push.
  - bpye 7y ago
    I would question why they are doing a rollout on a weekend?
    - dodobirdlord 7y ago
      A config push on a weekend seems pretty unlikely. Given that it's apparently a network congestion issue and showed up on a weekend, my guess is that it's probably a bizarre networking hardware failure like the one that took out CenturyLink[1] last December. [1] https://news.ycombinator.com/item?id=18789071 https://news.ycombinator.com/item?id=18789071
seibelj 7y ago
Not sure if related, but I was going to a BBQ yesterday and myself and 3 other people got lost because Google Maps app glitched out, directing us to the wrong places. If you search twitter for #googlemaps tons of people have the same issue. Surprised no one has posted about it.
w_s_l 7y ago
You know this reminds me of a bad taste that Google Sales team left when I asked for some of my billing that I was unaware of running after following a quickstart guide. AWS refunded me in the first reply on the same day! GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me. So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. I have called my bank to do a chargeback and put a hold on all future billing with GCP. I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice.... I've yet to receive a reply from Google and its been a week now. I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.
- WC3w6pXxgGd 7y ago
  Anything created in-house at Google (GCP) is typically created by technically-proficient devs, those devs then leave the project to start something new and maintenance is left to interns and new hires. Google customer service basically doesn't care and also has no tools at their disposal to fix any issues anyway. The infinite money spout that is Google Ads has created a situation in which devs are at Google just to have fun - there really is no incentive to maintain anything because the money will flow regardless of quality. Source: I interned at Google.
  - marcinzm 7y ago
    Isn't it also that promotions at Google are based on creating new products/projects rather than maintaining existing ones? So engineers have a negative incentive to maintain things since it costs them promotions.
    - xkcd-sucks 7y ago
      That explains the proliferation of chat services and why they all get actively worse over time until they're replaced
Exuma 7y ago
https://downdetector.com/ https://downdetector.com/ Pretty much every service is down
- lachs 7y ago
  It's frightening how many services rely on gcloud and which impact this downtime has
markoa 7y ago
We're hosting an open global Zoom call for all engineers affected by the outage, join us at https://zoom.us/j/793450725 https://zoom.us/j/793450725
- avocado4 7y ago
  One click on this link and it instantly starts streaming your webcam footage to everyone in the chat room.
  - deleted 7y ago
    [deleted]
  - Sebguer 7y ago
    This is a local zoom setting, you can change it.
  - chrisseaton 7y ago
    Use 'Turn off my video when joining meeting' in Zoom.
  - briandear 7y ago
    Put some clothes on.
    - deleted 7y ago
      [deleted]
  - revvx 7y ago
    This is weird, I had Zoom installed a long time ago but uninstalled according to their instructions [1]. I'm a macOS user. As soon as I clicked that link, the client downloaded a PKG file, installed itself and launched itself without asking me if I wanted to share my camera or audio. I uninstalled according to their instructions again, searched for all "zoom" files in my disk and rebooted. This leads me to believe that following their uninstall instructions is insufficient, and there are hidden files left on my computer. Sorry in advance for the off topic message [1] https://support.zoom.us/hc/en-us/articles/201362983-How-to-uninstall-Zoom-on-a-Mac- https://support.zoom.us/hc/en-us/articles/201362983-How-to-u...
sdan 7y ago
Weird for Twitter to still be up and fully functioning. I thought they migrated everything to GCP this/last year?
- sorenn111 7y ago
  Not the main functionality of the service, just lots of data analysis tooling. nothing that end users would notice
  - sdan 7y ago
    Interesting. Thought I had read some posts of them migrating their data, but you could definitely be right.
f_martinez 7y ago
We are on region us-east1 and our systems are still up. Specifically, we are on us-east1-b.
jshprentz 7y ago
The two Google Cloud networking incidents are: Incident #19008 began at 2019-06-02 12:48. https://status.cloud.google.com/incident/cloud-networking/19008 https://status.cloud.google.com/incident/cloud-networking/19... Incident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... Times are US/Pacific
- pm90 7y ago
  They both seem to say the same thing....
- boulos 7y ago
  dang et al.: I’d make 19009 the URL for this (GCE Just reported first, but this is an outage / networking issue in the Eastern US).
  - AnssiH 7y ago
    #19008 mentioned by parent is not the GCE one, there are two networking ones: #19008 and #19009. The #19008 (networking) says there will be an update by "13:30 US/Pacific", but as of 17:05 there is no update. Similarly, #19003 (GCE) says there will be an update by "16:00 US/Pacific", but no update as of 17:05. All the latest updates seem to only be in the third incident #19009 (networking).
- cheeze 7y ago
  Looks like they are having trouble updating their statuses. 19008 was supposed to be updated over an hour ago. Meanwhile, 19009 has the same comment posted three times. I'm guessing internal tools are barely working at best.
  - londons_explore 7y ago
    That is normal for Google status pages. They don't want to admit fault or place blame because there can be legal and commercial ramifications, so they can only say canned responses.
jamisonbryant 7y ago
I've noticed problems on GDrive (GSuite) and YouTube as well. Connected?
- mindcrime 7y ago
  FWIW, Youtube is fine for me, but I'm seeing intermittent errors saving updates to a document in GSuite. I had thought the latter error was a problem with the wifi where I am, until I saw this. Now I'm not so sure. HN is loading fine on the same wifi...
ikeboy 7y ago
So that's why YouTube was being weird. I thought it was an extension problem or something.
sarim 7y ago
Funny how as soon as I realized that Gmail and Google Sheet aren’t working properly I rushed to HN to figure out what’s going on. I love this community!
sgammon 7y ago
So far the Ko list: GCE, GKE, BQ, Pub/Sub, GAE asia-south1 us-west1 us-central1 us-west2
- captn3m0 7y ago
  Our GKE stuff in asia-south1 (Mumbai) is up, and the GKE console works fine here. (Bangalore).
maz1b 7y ago
Ironically, I moved all of my objects off GCS today.
javabean_ 7y ago
could this be the result of another BGP hack ? cyberwarfare ? I am just speculating here big time.
pishpash 7y ago
Let's see if perfect leetcode skills will save the day. /s
CSDude 7y ago
With Google Cloud incidents, most of the time whole regions fail, and with AWS generally only a region fails. Of course there would be exceptions, but Google Cloud does not make me feel safe as an outsider (and a user of multi-region AWS)
gist 7y ago
Right now this is at the 'good at least it's not mine to fix or worry about' kind of like 'and the reason I choose IBM' [1]. I can just sit back and wait for gmail to work correctly. Now at the point it starts to last what I would consider a long time well then I will have things to worry about. One thing with gmail though. When it's down it's similar to a snow storm if you only do business in a city. Everyone is impacted and everyone understands a missed deadline is unavoidable. [1] For those not old enough to know what I mean read this: https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalcomputer/words/ https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalc...
cobaimelan 7y ago
seems like only us down
d2xdy2 7y ago
Was certainly an interesting alert when my Cloud Functions started reporting downtime. Among the many things that dip in and out on what seems like a monthly basis, I’ve not seen them just drop out in quite a while. Hopefully they get things sorted out. I can’t really imagine what it looks like internally when this level of outage is going on, but I want to think everyone is fairly collected
deleted 7y ago
[deleted]
javabean_ 7y ago
Nice overview : https://downdetector.com/ https://downdetector.com/
- XCSme 7y ago
  I see a lot of Nest complaints, isn't there any security issue if Nest goes down?
  - pmlnr 7y ago
    I see people complain about Nest and the only thing I can think of is "what on earth where you thinking to have a door or thermostat that doesn't function without internet?!"
    - mrb 7y ago
      Nest thermostats can function without Internet. You can walk up to it and adjust the temperature just like any old thermostat.
    - disillusioned 7y ago
      I live in a house with a Nest thermostat and Nest Secure and Nest x Yale door locks. The AC is on just fine, and the door unlocked just fine. (The door lock doesn't require an internet connection, unless you've enabled privacy mode for some reason.)
      - pmlnr 7y ago
        I thought there were versions that don't work offline; my bad if this is not the case.
    - sofaofthedamned 7y ago
      Internet isn't the issue, it's the endpoint (Google) It's reasonably easy to run a router/gateway that has a 4G backup to get the ping out. Whether the ping works...
  - 7y ago
ccarpenterg 7y ago
Google Colaboratory is back. At least I can access my github's notebooks and public notebooks from google drive.
cjohansson 7y ago
On days like these I’m glad I don’t use any of the affected services
biohax2015 7y ago
Does anyone know if this is a regional or global outage? I can see my GKE clusters in one region but not in another, so I am guessing it's the former. Looks like we'll need a cluster in each region going forward...
- boulos 7y ago
  To help others: which two regions?
  - biohax2015 7y ago
    us-central1 is up us-east4 is down
    - hoonto 7y ago
      us-west2 is down
- XCSme 7y ago
  I can not access vimeo from EU, not sure if related to Google though.
peterwwillis 7y ago
The last big outage, iirc, was Google didn't test their rollback procedures for router upgrades. I'll be very interested to hear if it's yet another change control problem that caused this outage.
amasad 7y ago
Anyone experiencing issues with GCS? Seems highly intermittent and dependent on the location the request is coming from (maybe that's because it's a networking bug). The status page says GCS is fine but that's highly unlikely.
ejanus 7y ago
I just can't reach google apps on my HTC m9 since yesterday. I am in West Africa. My Whatsapp crashed too and I lost all my previous threads. Is my issue related to Cloud being down?
captn3m0 7y ago
What I've realized from this: Google doesn't have an official status page for GCP. There are a few unofficial ones, but nothing official that I could find.
- cheeze 7y ago
  Like this? https://status.cloud.google.com/ https://status.cloud.google.com/
  - captn3m0 7y ago
    I meant Twitter Status Page (in my defense it was 3 am). The only ones are: @gcpstatus and @gcp_incidents, both unofficial
codemati 7y ago
https://tour.golang.org https://tour.golang.org is also down
outside1234 7y ago
It seems crazy to me that Google Cloud can have this level of instability but I, on the other hand, can never remember google.com going down. Why are they operating one with a different networking infrastructure from the other?
- chupasaurus 7y ago
  AFAIK google.com is running independently on a different infrastructure, but some of other services are relying on GCP hence the problem affects them too.
- sakisv 7y ago
  They did have a major outage for a few minutes on 2013: https://www.theregister.co.uk/2013/08/17/google_outage/ https://www.theregister.co.uk/2013/08/17/google_outage/
  - escapecharacter 7y ago
    So this is now the longest outage ever, right?
- savrajsingh 7y ago
  Sounds like a good design — but in the early days of google cloud — 2008 — you’d get the google homepage on a bad request
- vl 7y ago
  It's completely different mode of failure - when part of search is down somewhere, it's just routed to the same service in other location. If part of the GCP is down anywhere, some customers are affected. Since original Google infrastructure was developed specifically for first kind of services, cloud org still has problems adopting it to its needs.
chpmrc 7y ago
This might be all in my head but I've been experiencing really bad latency for like an hour, while browsing, and then I read this.
m0zg 7y ago
Prediction: the final postmortem will say "someone pushed a bad config", just like most of the previous postmortems (and most of the internal postmortems as well, for Borg-based services). This is the cause of most other outages in other cloud providers as well. A really hard to solve problem.
- krackers 7y ago
  Multiple regions seem to be affected though. Wouldn't it make more sense to start out config pushes with a single region before expanding it to avoid these types of wide outages?
  - m0zg 7y ago
    It's a networking outage. Google is a well known user of SDN. Networks, by virtue of connecting things, necessarily affect more than one region.
bluefox 7y ago
You look up. A single ray of light has made its way to the earth. Some day, you hope, the sky will be clear again.
typenil 7y ago
Was trying to set up SSL on a GKE cluster today. Guess I'll have to wait for tomorrow if I want to be able to tell my mistakes apart from Google's.
wenbin 7y ago
forbes.com is down also?
AdmiralAsshat 7y ago
And Thunderbird suddenly kept throwing prompts for my Gmail passwords, even after I sign back in. I hope it's related.
- captn3m0 7y ago
  It says "could not reach imap.gmail.com" for me.
titzer 7y ago
This is your yearly reminder to resist centralization of the internet.
- quickthrower2 7y ago
  Yes, I haven't read the Bitcoin is down or BitTorrent is down status yet.
- jasonvorhe 7y ago
  Because Internet Exchanges and carriers never had network issues.
  - eitland 7y ago
    Because then at least you and your colleagues has a chance to get some work done.
el_benhameen 7y ago
That explains why my google home thing thinks it’s sub-zero even though I’m warm in shorts and a t-shirt. https://pasteboard.co/IhBsyrsO.jpg https://pasteboard.co/IhBsyrsO.jpg
lewis1028282 7y ago
So just got a new phone and trying to resync my 2FA for AWS but I can't because Gmail is down. Ffs.
netmonk 7y ago
There is a loop in the spanning tree.....
finphil 7y ago
Working here but slow (I'm based in Central Canada).
deleted 7y ago
[deleted]
unilynx 7y ago
G suite was broken for me 20 minutes ago (in Europe) but is working now. Perhaps things are starting to come back?
based2 7y ago
https://www.reddit.com/r/sysadmin/comments/bw1gye/most_google_gsuite_services_gmail_calendar_etc/ https://www.reddit.com/r/sysadmin/comments/bw1gye/most_googl... https://www.zdnet.com/article/some-internet-outages-predicted-for-the-coming-month-as-768k-day-approaches/ https://www.zdnet.com/article/some-internet-outages-predicte... 768k Day
- _Marak_ 7y ago
  This should be voted higher up. According to https://twitter.com/bgp4_table https://twitter.com/bgp4_table, we have just exceeded 768k Border Gateway Protocol routing entries, which may be causing some routers to malfunction.
  - dreamer_soul 7y ago
    Isn't it weird that it's happening now even though that number was surpassed nearly a month and half ago?
    - dyu 7y ago
      Different locations see different counts because of aggregation/de-aggregation.
- juanuys 7y ago
  Will this affect more than just Google? I haven't seen any outages from other cloud providers.
  - namibj 7y ago
    packet.net was hit. Specifically, also their San Jose DC. Internet only. It took less than an hour to recover. More than 20 minutes. I didn't ping it continuously, and I can say that the traceroute got stuck in Frankfuhrt (Where my ISP and their ISP first (as seen from me) meet). I was actually surprised, as they tend to have excellent networking. Now I'm not nearly as distrusting as I was initially, knowing it was likely their ISP getting screwed by routing table overflow.
tntn 7y ago
There goes 3 nines for June and for Q2. I guess everyone gets a 10% discount for the month? https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla
- gundmc 7y ago
  A couple more hours and everyone will get 25% off for June.
  - twistedpair 7y ago
    Does that apply to the rest of June? Might be a good month to rebuild all your models ;)
    - quickthrower2 7y ago
      The vultures are circling.
- CamelCaseName 7y ago
  Ironically, the SLA page returns a 502 error.
- londons_explore 7y ago
  The discount seems way too small. I would pay a premium for a cloud provider happy to give 100 percent discount for the month for 10 minutes downtime, and 100 percent discount for the year for an hour's downtime.
  - gundmc 7y ago
    Any cloud provider offering those terms would go out of business VERY quickly. Outages happen, all providers are incentivized to minimize the frequency and severity of disruptions - not just from the financial hit of breaching SLA (which for something like this will be significant), but for the reputational damage which can be even more impactful.
baalimago 7y ago
Playing "is google on this website or not?" just became so much easier, simply see if website works or not. Scary stuff. What happens when Murphy's law decides to crash things even more?
ramon 7y ago
No wonder I was trying Webcodesk right now and it's not working it's all firebase, yay!
ramon 7y ago
It seems that it`s focused on the database stuff like firebase and firestore.
- dodobirdlord 7y ago
  Apparently (from further up the thread) it's a network congestion issue causing extremely high rates of packet loss. I imagine pretty much anything that's homed in the effected regions will be degraded or unaccessible.
sneak 7y ago
A demo of “too big to fail [via antitrust]”?
noncoml 7y ago
There go the nine nines for this year? Is it more like four nines or so now?
skizm 7y ago
Ah, wonder if this is affecting Google's SSO. It was super slow when I was trying it just now.
squarefoot 7y ago
I appears that logging into the webmail solves at least POP mail problems. I tried my mail client and failed, then attempted a login to the webmail which worked. Gmail then asked me to confirm my recovery address and cellphone, which I did, and finally loaded the inbox page. I immediately attempted a connection through the POP client and this time it worked. It might be something security related if it triggers a mandatory identity confirmation. edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error. [23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later. The problem seems much more complex.
pg_is_a_butt 7y ago
None of my stuff is down.
filvdg 7y ago
Level(3) one of the biggest backbones has issues too, might be related https://downdetector.com/status/level3 https://downdetector.com/status/level3
Yrlec 7y ago
Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: https://cloud.google.com/storage/sla https://cloud.google.com/storage/sla. So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.
- crazygringo 7y ago
  Ironically I can't read that page because, since it's Google-hosted, I'm getting an HTTP 500 error... but which means at least that service is SLA-covered... Cloud services live and die by their reputation, so I'd be shocked if Google ever tried to get out of following an SLA contract based on a technicality like that. It would be business suicide, so it doesn't seem like something to be too worried about?
- deleted 7y ago
  [deleted]
moon2 7y ago
Its funny how searching for "Google Down" on Google and filtering for results within 1 hour yields nothing.
stvkoch 7y ago
My gmail it's down!
StaticRedux 7y ago
This must be why GitLab is giving me shit. They recently made the switch. Wonder if there is any second guessing going on over there right now.
- Uninen 7y ago
  They have a incident report page up at: https://docs.google.com/document/d/1RM3QnuJ4FPH10J3UrJS0T26d-mn_Dd11-jmZweHQVV8/edit# https://docs.google.com/document/d/1RM3QnuJ4FPH10J3UrJS0T26d...
  - StaticRedux 7y ago
    Using Google Docs to report problems with Google, including Google Docs. Irony at it's finest.
    - Uninen 7y ago
      They do have an internal ticket as well, but Gitlab.com is also hosted on Google, so... https://gitlab.com/gitlab-com/gl-infra/production/issues/862 https://gitlab.com/gitlab-com/gl-infra/production/issues/862
- kerng 7y ago
  I was just thinking the same. One of their reasons (besides being paid a lot by Google) was stability.
- dsumenkovic 7y ago
  You can find some more data about GitLab availability after the move to GCP here https://about.gitlab.com/2018/10/11/gitlab-com-stability-post-gcp-migration/ https://about.gitlab.com/2018/10/11/gitlab-com-stability-pos.... As we're trying to stay transparent as always, we'll definitely let everyone know if are going/thinking about the change.
eisvogel 7y ago
gmail is down in Australia...
atopuzov 7y ago
Hug your on-call engineer.
cedricium 7y ago
Majority of G Suite services are suffering service outages: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status
tlogan 7y ago
I also noticed that Google search stop indexing news articles. So I searched for "gmail down" on bing and I got some results [1]. But searching on Google for "gmail down" does not return any results [2]. [1] https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT [2] https://www.google.com/search?q=gmail+down&source=lnms&tbm=nws https://www.google.com/search?q=gmail+down&source=lnms&tbm=n...
griffinkelly 7y ago
Looks like Google Analytics isnt reporting stats either?
_xerxes_ 7y ago
Nest is down too, not surprising given they are part of Google. What I don't understand is why I can't still control my devices over my local network. Why does the system even require access to Google servers?
- deleted 7y ago
  [deleted]
- fredthomsen 7y ago
  That made me rage the other day when I had an internet outage. I am honestly fed up with Nest esp after this new Google Nest branding, and just disconnected my thermostats from the internet. I rarely changed it from my scheduled settings via the app anyway.
  - jansan 7y ago
    Makes me wonder What your thermostats are doing on the internet in the first place.
- good_guy 7y ago
  becasue who gives a shit ? you already bought it.
- kibwen 7y ago
  Let's make the following the tech mantra of the next decade: "the Internet Of Things can never possibly work until the LAN Of Things does".
  - nektro 7y ago
    the IoT trend really needs to stop
    - xhgdvjky 7y ago
      LANoT?
- 7y ago
deleted 7y ago
[deleted]
fdgwhite 7y ago
I tried multiple times to setup a Google Wifi router today. Wifi would work but the app said it was offline. Perhaps I am not insane or incompetent after all
ecabuk 7y ago
My builds are failing because it cannot download the chromium. > Error: Download failed: server returned code 502. URL: https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/662092/chrome-linux.zip https://storage.googleapis.com/chromium-browser-snapshots/Li...
the-dude 7y ago
Huawei just flexing its muscles. Nothing to see here, move along.
robdachshund 7y ago
Looks like this is why Shopify servers are down as well. Apparently last year they moved everything to GCP. I don't know why anyone uses them. They have proven to be highly unreliable and have horrible support. While I'm at it, I'm still bitter about gitlab switching to gcp from azure. They were trying to save face after the github acquisition and just moved on to an even more dubious company for their servers.
pitaj 7y ago
Looks like my Gmail is back, but I don't have any emails from while it was down. Yikes. Edit: just got one email from the downtime, so perhaps my initial conclusion was incorrect
- Stryder 7y ago
  Whoa, if confirmed this would be really bad. Downtime happens, but data loss is much worse.
  - BillinghamJ 7y ago
    This is normal handling for SMTP - the sending server just tries again later.
chx 7y ago
These things happen. That's OK. Here's what's not OK: > We are investigating an issue with Google Compute Engine. We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific. The next update is at 12:59. Just ... no.
CallumRobertson 7y ago
Snapchat is fixed but snap maps is still disabled
CallumRobertson 7y ago
Snapchat is back up but snap maps is down
colo12 7y ago
Is it related to TianAnMen square massacre 6/4/1989 30th anniversary?
nmstoker 7y ago
Seems to have coincided almost exactly with my Chromecast stopping displaying my photos (in ambient mode).
fourty2 7y ago
google services also f´ed up here in middle europe; cannot reach anything google related from hamburg, germany.
londons_explore 7y ago
It seems the AdWords anti-spam system is down, which means anyone can put a billion dollar bid on every keyword and get their ads showing on every Google search for every query. Systems that fail 'open'...
- good_guy 7y ago
  lol, google "engineers" are overrated.
remote_phone 7y ago
Phase 2 of mitigation was completed a few minutes ago and GCE is expecting improvement shortly.
nopriorarrests 7y ago
https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003 >We will provide an update at 16:00 US/Pacific. it's 16:22 and no updates were posted. a bit unprofessional..
bbayer 7y ago
I think nobody mentioned but Adsense also is not updating for couple of hours.
joaomacp 7y ago
I was finishing a university assignment with the deadline 90mins away. I wanted to upload a video of the project to YouTube and add a link to it in the report. YouTube takes a long time to process the video, and then says it's unavailable. I go to Vimeo: it's down. I upload the video to Dropbox, and copy its link to the report. But my report was a Google doc. And when I tried to export it as PDF (which I had not done yet) it couldn't do it. I never hated google more. Eventually the video went through to YouTube, and I could export the PDF on the third try, but this really made me conscious of my dependance on Google.
misrab 7y ago
Anything to do with China?
bahmboo 7y ago
No, not really.
rum3 7y ago
Keep on centralizing the internet with your stupid clouds and this is what happens.
AngeloAnolin 7y ago
Does Google also have some sort of listing on which consumer apps are particularly affected (i.e. Gmail, Hangouts, Docs, Sheets, etc). The cloud components may be directly affected but for consumers, there's nothing which will provide info on what consumer facing services are getting some issues.
tcarn 7y ago
Once again HN proves to be the best status monitor.
ecthiender 7y ago
In India, I could access Youtube, Gmail (web) and Google cloud console and GKE and Compute Engine instances in south-east asia region.
bprasanna 7y ago
https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003 Does status update mean only the status. Won't there be words like "sorry", "apologize" & "inconvenience"! Only PR is responsible for those words?
- jwandborg 7y ago
  Apologies are unnecessary on status pages. You can be sure that they are sorry.
sandra101 7y ago
hello everyone , i dont need to talk much about , this you need to be careful with any hacker you give your work to , the trustworthy, most reliable hacker you need for repairing your credit score is Neo , you just need to mail him now and make your problem solve on neogonzalezhacks at gmail dot come or text him on 5139001324 , dont think about it , just mail him he will respond immediately , tell him from james Guinn , he help me with my credit score within 2 weeks , my credit score was very low and helped me repair it to 820, dont hesistate to mail him now , just mail him and rejoice .happy for you for doing that ..
aasssdddd 7y ago
w

1395 points by markoa 7y ago

dang 7y ago

Related: https://news.ycombinator.com/item?id=20077275 https://news.ycombinator.com/item?id=20077275 https://news.ycombinator.com/item?id=20077293 https://news.ycombinator.com/item?id=20077293

PerfectElement 7y ago

Just 2 weeks after I migrated a DB cluster from Azure to Google Cloud thinking things would get better.

duality 7y ago
FWIW they might still be.
AaronFriel 7y ago
Just remember that "better" can be entirely based on when you pick the starting and ending points of your graph ;)
- XCSme 7y ago
  It's just like the stock market.
user5994461 7y ago
It's the yearly outage. You will be fine for next year. Better than the monthly outage from Azure.

leowoo91 7y ago

0 issues at compute, reporting for europe-west3-b,

ksajadi 7y ago

GCP status page is worthless as it's always happy and green when production systems are down and then they might acknowledge something an hour later

JimboOmega 7y ago
Just like AWS, then. "Some users are experiencing increased error rates" = "Everything has been down for hours"
- bsimpson 7y ago
  "Everything is fine, unless you're Carl. There's a massage outage, but only at Carl's house. Sorry, Carl."
  - notatoad 7y ago
    I'm also experiencing a massage outage. Please send masseuse.
    - bsimpson 7y ago
      Goddammit some (most?) days I can't type. "Massive"
      - quickthrower2 7y ago
        I got the missive. Thanks.
- djsumdog 7y ago
  I remember when S3 was down and the status was green because the updates for the status page with pushed via S3.
  - chmhsm 7y ago
    That's not just ironic, that's stupid. How do you count on S3 to update S3 status? Isn't that a huge design fault?

arach 7y ago

Confirming issues on our end. I'm able to load up my console but when I go to Kubernetes Engine, I don't see my clusters. I'm monitoring closely on twitter

walshemj 7y ago

just had rolld20 in the USA blow out a game I wonder if it is effected

zoidq 7y ago
Looks to be working in the UK
- walshemj 7y ago
  Network peering probs in Canada - looks like an unrelated problem

RappingBoomer 7y ago

youtube streaming is also down

horyzen 7y ago

Yeah I was having trouble accessing my Gsuite apps, had a couple of 502s, which led me to check HN. While it doesn't give me 502 now, it's abnormally slow.

nkassis 7y ago

The GCE console also affected, couldn't send a support ticket just getting errors.

darkof 7y ago

That feeling when you open https://console.cloud.google.com https://console.cloud.google.com and see that you don't have your Kubernetes clusters and CloudSQL databases, but CTA to create first.

vpontis 7y ago
Gosh, this was so scary... I thought someone had hacked in and deleted everything... I hope they come back. This is still pretty scary
- stnmtn 7y ago
  Same, my Manager called my and said "everything is down". So I wander over to my Firebase console, and there's no database loading. Thank god for twitter, and people also saying that they have the same issue or I would have for sure though we've been hacked. I hope this is a good wake up call for everyone. I know that I'm going to think more about how we do backups and fail-safes
- hanniabu 7y ago
  And here I thought I was having a bad day with Google Play not loading
- jjeaff 7y ago
  Same. I was thinking, oh, my db cluster must be having trouble recovering. Couldn't get any response through kubectl. Logged in to the cloud console and it looks all brand new, like I have no clusters setup at all. Of course, this is 2 weeks after switching everything over from AWS.
- dpau 7y ago
  my vm instances are all still there, can even log in via SSH in the compute engine tab. looks like they got a reboot 15 min ago. just restarted some processes but lost my progress on about 12hrs of computing time, i'm guessing it's going to be hard to get a refund..
  - 7y ago

codys 7y ago

Other google services are also affected: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status https://i.imgur.com/pcqwwA4.png https://i.imgur.com/pcqwwA4.png

mixedbit 7y ago
Good that Google+ is up again
deleted 7y ago
[deleted]

digaozao 7y ago

I cant see any gke cluster in Brazil, or any VM.

miller_joe 7y ago
I'm seeing that with northamerica-northeast1. I can't access anything over the network in that region and most of the GKE clusters and VMs in that region aren't listed in the console

stuck_in_the_ma 7y ago

Google Play is also experiencing massive issues.

snisarenko 7y ago

The status page took a while to show issues. My app was down, and Twitter knew google cloud was down before the official status page.

deleted 7y ago

[deleted]

vinayan3 7y ago

Looks like only GCE is down according to the status page now. I'm able to access my console for instances and GKE clusters.

snisarenko 7y ago
My site runs on Google App Engine and its down as well.
sladey 7y ago
I happened to be initializing a GKE pool upgrade just as this occurred. The upgrade is now stuck according to the console. The interesting thing is that a couple of minutes before everything went wrong, kubectl returned a "error: You must be logged in to the server (Unauthorized)" error
miller_joe 7y ago
They initially opened an incident against GCE (https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003) then opened one against Networking (https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...). The networking incident looks like the one to follow for updates now.

murat124 7y ago

GCP has been down since 11:50am and they acked it 35 mins later. They're great at leaving their customers in the dark.

copperx 7y ago
Not much different from AWS, from what I've heard.
- numbsafari 7y ago
  Definitely the case. Neither are super great at this. One issue is that issues that may 100% impact individual clients may only impact a vanishingly small amount of their overall service load. That mismatch between customer and provider experience is one of the ugly aspects of public cloud providers.
  - kazen44 7y ago
    Also, it's one which gets hugely understated when people "move to the cloud". especially if you use your bussiness for B2B services. Stuff like this could make you loose your bussiness, especially if some entity like google doesn't communicate and as a result, you do not have a answer for your own customers. Medium sized private cloud providers are a lot better at this, considering the communication lines are a lot shorter.
    - zepolen 7y ago
      On the flip side - a customer is more willing to be understanding if 'Google is down' instead of 'our server is down'.
  - discodave 7y ago
    That's why AWS is all about their Personal Health Dashboard (PHD). They can post specific issues for your account in there. Also, they get to keep the public page looking nice and green to show to executives of prospective customers.
- 7y ago

xerxes901 7y ago

Couldn't load the support console to "me too" this one either!

macintux 7y ago

And thus was ruined hundreds or thousands of pleasant Sunday afternoons. I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.

xerxes901 7y ago
Spare a thought for the pleasant Australian early Monday mornings too! Always a rude awakening...
jagtesh 7y ago
Multi-cloud for those times when you really need that level of availability and can afford it.
- jjeaff 7y ago
  It's not even about being able to afford it. Some things just don't lend themselves to hot failover. If your data throughput is high, it may not be feasible or possible to stream a redundant copy to a data center outside the network.
  - EugeneOZ 7y ago
    All parts of the system should be copied (if you decided to build multi-cloud system), not just some of them.
newsbinator 7y ago
It's the Queen's birthday, a Monday off here in New Zealand... ... but not for everybody now.
- Scoundreller 7y ago
  So what happens when the crown changes? They change the holiday? Immediately? For the next year? Sounds like a bit of a nightmare.

mandatory 7y ago

Yep, I can no longer see my Cloud SQL database - it's as if I've never created one at all. Really hoping this is just an issue displaying it and that Google hasn't punted my infrastructure and backups.

vpontis 7y ago
Praying isn't working. Now, I'll try sobbing :(
- Havoc 7y ago
  Systematic problem solving. I like it
- dodobirdlord 7y ago
  Reports claim that it's a network congestion issue primarily in the northeast United States. It seems doubtful that any data has been lost, but you probably can't see resources due to requests not getting through. I hope that hearing this helps you to feel better.
flamingcow 7y ago
[I am the Cloud SQL tech lead] This is a networking issue, and your data is safe. Cloud SQL stores instance metadata regionally, so it shares a failure domain with the data it describes. When the region is down or inaccessible, instances are missing from the list results, but that doesn't say anything about the instance availability from within region.

tr33house 7y ago

Took me a while to track latency issues to GCP. Wasn't expecting it. This also seems to affect some GAE instances and some of their products like google photos. At least according to my observations

different_sort 7y ago
I see this as well.

bilal4hmed 7y ago

gitlab is slow too

amenod 7y ago
Slow is understatement... some pages on gitlab.com take minutes to load, and jobs take tens of minutes to start. EDIT: It's been like that since at least 12h ago though. Not sure if it's connected to Google Cloud?
- dsumenkovic 7y ago
  Yes, it was definitely related to GCP services. GitLab is no longer seeing errors and Google Cloud has resolved the issue as of 23:00 UTC yesterday. Any further information can be found on the issue at https://gitlab.com/gitlab-com/gl-infra/production/issues/862 https://gitlab.com/gitlab-com/gl-infra/production/issues/862
  - amenod 7y ago
    The timeline seems a bit short, but ok. :) (the problems with runners and the UI started at least at 2019-06-02 7:48 UTC, though they were hit-and-miss at the time) Still, happy this is solved and we can use the (fantastic otherwise) service again!

klon 7y ago

Anyone using both AWS and GCP that can form an opinion on availability of both? As a GCP customer I am not very happy with theirs.

pm90 7y ago
GCP is incredibly bad at communicating when there are problems with their systems. Just terrible. Its only when our apps start to break that we notice something is down, then look at the green dashboard which is even more infuriating.
- timdorr 7y ago
  AWS is often the same way. No one seems to be good at communicating outage details.
  - TallGuyShort 7y ago
    I suspect there's a correlation between outages that are easy to detect and communicate and outages that automation can recover from so easily that you hardly notice.
    - deleted 7y ago
      [deleted]
  - obeattie 7y ago
    I really don’t get this. There’s a huge number of complaints about poor communication from companies like Google and AWS during every outage. Yet they remain seemingly indifferent to how much customer trust they are losing, and the competitive edge the first one to get this right could gain.
    - atmosx 7y ago
      If 20% of Google Cloud's customers leave after this outage because of poor communication they'll prioritise accordingly and apply all that nice SRE theories to their infra. But this isn't happening, because <various reasons>, so... who cares?

landon32 7y ago

u.s. west: all our cloud compute is inaccessible rn.... our API is down, can't ssh into the servers, and also can't see them on the dashboard.

colinbartlett 7y ago

Google Cloud is the number 4 most monitored status page on StatusGator and Google Apps is number 12. In addition, at least 20 other services we monitor seemingly depend on Google Cloud because they all posted issues as soon as Google went down. It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up.

hazeii 7y ago
...and only the paranoid survive?
- MrMorden 7y ago
  Just because you're paranoid, it doesn't mean they're not out to get you.
FPGAhacker 7y ago
I guess we know what steam uses (the store at least).
- xNevo 7y ago
  No issues for me. Maybe they have a failover mechanism?
  - FPGAhacker 7y ago
    yeah, maybe it was coincidence. seems to be back up for me as well: https://steamstat.us https://steamstat.us
- colinbartlett 7y ago
  I don't know about Steam, but I know Apple must use Google Cloud: https://www.apple.com/support/systemstatus/ https://www.apple.com/support/systemstatus/
  - 7y ago

tus87 7y ago

So that's why I can't login to YouTube this morning...

different_sort 7y ago

I was playing around this afternoon with appengine, and thought I broke one my projects when I started getting 502 back. There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me. The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.

deleted 7y ago

[deleted]

derekhh 7y ago

> We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific. I'm not seeing anything at 12:47.

derekhh 7y ago
https://status.cloud.google.com/incident/cloud-networking/19008 https://status.cloud.google.com/incident/cloud-networking/19... 404 - Impressive
- ryacko 7y ago
  Cloud status dashboards seem to be hosted on the same cloud, which doesn't say much about redundancy.
  - chupasaurus 7y ago
    AWS changed internals of Service Health dashboard after they couldn't update it when S3 went down in us-east-1 (https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/) edit: wording
    - ryacko 7y ago
      Someone had to design the status page, and failed to anticipate the issue about how the status page depends on the systems it is reporting on. Unlikely the rest of AWS, a cached web page does not require much complexity.
timdorr 7y ago
They are updating the root cause issue, which is networking, here: https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... Next update is in about 25 minutes.

rwbhn 7y ago

gmail also down/super slow atm for me (East Coast, USA)

pgoodjohn 7y ago

Everything looking normal on our GKE / CloudSQL stuff (eu-west1)

wichert 7y ago
gcloud tells me: WARNING: The following zones did not respond: us-west2, us-west2-a, southamerica-east1-c, us-west2-b, southamerica-east1, us-east4-b, us-east4, us-east4-a, northamerica-northeast1-c, northamerica-northeast1-b, us-west2-c, southamerica-east1-b, northamerica-northeast1, southamerica-east1-a, northamerica-northeast1-a, us-east4-c. List results may be incomplete. Luckily for us eu-west1 seems to be working normally.

echelon 7y ago

Nest is down for me right now. It's amazing how far-reaching outages can be these days.

unicornmama 7y ago
There are dozens of us.
andrewprock 7y ago
Code reuse is a wonderful thing, until it's not.
- hn23 7y ago
  Code reuse? Couple if years ago some started to call servers "cloud" but the cloud is just that: computers managed by someone else. If the whole service layer relies on Three big companies, there is a problem..

squarefoot 7y ago

And Gmail too doesn't feel very well today. [21:55:19] POP< +OK send PASS [21:55:19] POP> PASS ******** [21:55:21] POP< +OK Welcome. [21:55:21] POP> STAT [21:55:21] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later.

ToniCipriani 7y ago
Pretty much Gsuite is out: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status
- megous 7y ago
  At least G+ is working. /s
hazeii 7y ago
IMAP as well - for some considerable time now.
- agentile 7y ago
  Yes experiencing IMAP auth/connectivity issues here as well, along with outgoing SMTP

benbristow 7y ago

Wondered why Snapchat was being weird today. Thought it was my pi-hole setup blocking something from working, but nope, it's Google!

brown9-2 7y ago

When talking about GCE being down please also mention what regions you are talking about

chupasaurus 7y ago
In this case it's a luck if any are working correctly, a problem is global with some exceptions.
- brown9-2 7y ago
  seems to be some comments here of some regions functioning ok, although perhaps it’s not 100% in all regions
sgammon 7y ago
us-central1 us-west1 us-west2 is what I’ve heard so far. east seems to be OK, and Europe too
- digaozao 7y ago
  South America is down for me. But youtube works OK, I can watch Google IO and they talking about all Google greatness while their outage impact second class clients on their cloud.

javabean_ 7y ago

vimeo.com is down.

boulos 7y ago

Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!). We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages. There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. You'll have to wait for someone actually involved in the incident to say more.

odiroot 7y ago
> including unfortunately the tooling we usually use to communicate across the company about outages. There's some irony in that.
- boulos 7y ago
  Edit: and I agree! I’m not in SRE so I don’t bother with all the backup modes (direct IRC channel, phone lines, “pagers” with backup numbers). I don’t think the networking SRE folks are as impacted in their direct communication, but they are (obviously) not able to get the word out as easily. Still, it seems reasonable to me to use tooling for most outages that relies on “the network is fine overall”, to optimize for the common case. Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. The Networking outage is the root cause.
  - marksomnian 7y ago
    > the status dashboard now correctly highlights that multiple things are impacted because Networking. this column of green checkmarks begs to differ: https://i.imgur.com/2TPD9e9.png https://i.imgur.com/2TPD9e9.png
    - boulos 7y ago
      The banner at the top. Sorry if that wasn’t clear.

deleted 7y ago

[deleted]

wenbin 7y ago

btw, Google Analytics realtime is down as well.

bbayer 7y ago
I wasn't aware that outage and had small heartattack when I saw huge drop of visitors. I think other metrics are also affected.
mileycyrusXOXO 7y ago
Thanks, I noticed I didn't have any users after pushing an update and thought I broke something.

sdan 7y ago

Github contribution graphs are also gone

sadris 7y ago

So is Youtube.

deleted 7y ago

[deleted]

mtarnovan 7y ago

Is shopify on google cloud? i noticed they are having issues too

brown9-2 7y ago
yes
- captn3m0 7y ago
  I'd reported a networking issue to Shopify yesterday, which they said was resolved upstream (GCP?). Could it be?
drchiu 7y ago
Yes it’s affecting their system as well.
crispyporkbites 7y ago
Sunday evening is a pretty big time for ecommerce. Losing 1000s of dollars of sales here :-(

40acres 7y ago

Can't wait for the postmortem!

murat124 7y ago
My money is on config push.
- bpye 7y ago
  I would question why they are doing a rollout on a weekend?
  - dodobirdlord 7y ago
    A config push on a weekend seems pretty unlikely. Given that it's apparently a network congestion issue and showed up on a weekend, my guess is that it's probably a bizarre networking hardware failure like the one that took out CenturyLink[1] last December. [1] https://news.ycombinator.com/item?id=18789071 https://news.ycombinator.com/item?id=18789071

seibelj 7y ago

Not sure if related, but I was going to a BBQ yesterday and myself and 3 other people got lost because Google Maps app glitched out, directing us to the wrong places. If you search twitter for #googlemaps tons of people have the same issue. Surprised no one has posted about it.

w_s_l 7y ago

You know this reminds me of a bad taste that Google Sales team left when I asked for some of my billing that I was unaware of running after following a quickstart guide. AWS refunded me in the first reply on the same day! GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me. So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. I have called my bank to do a chargeback and put a hold on all future billing with GCP. I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice.... I've yet to receive a reply from Google and its been a week now. I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.

WC3w6pXxgGd 7y ago
Anything created in-house at Google (GCP) is typically created by technically-proficient devs, those devs then leave the project to start something new and maintenance is left to interns and new hires. Google customer service basically doesn't care and also has no tools at their disposal to fix any issues anyway. The infinite money spout that is Google Ads has created a situation in which devs are at Google just to have fun - there really is no incentive to maintain anything because the money will flow regardless of quality. Source: I interned at Google.
- marcinzm 7y ago
  Isn't it also that promotions at Google are based on creating new products/projects rather than maintaining existing ones? So engineers have a negative incentive to maintain things since it costs them promotions.
  - xkcd-sucks 7y ago
    That explains the proliferation of chat services and why they all get actively worse over time until they're replaced

Exuma 7y ago

https://downdetector.com/ https://downdetector.com/ Pretty much every service is down

lachs 7y ago
It's frightening how many services rely on gcloud and which impact this downtime has

markoa 7y ago

We're hosting an open global Zoom call for all engineers affected by the outage, join us at https://zoom.us/j/793450725 https://zoom.us/j/793450725

avocado4 7y ago
One click on this link and it instantly starts streaming your webcam footage to everyone in the chat room.
- deleted 7y ago
  [deleted]
- Sebguer 7y ago
  This is a local zoom setting, you can change it.
- chrisseaton 7y ago
  Use 'Turn off my video when joining meeting' in Zoom.
- briandear 7y ago
  Put some clothes on.
  - deleted 7y ago
    [deleted]
- revvx 7y ago
  This is weird, I had Zoom installed a long time ago but uninstalled according to their instructions [1]. I'm a macOS user. As soon as I clicked that link, the client downloaded a PKG file, installed itself and launched itself without asking me if I wanted to share my camera or audio. I uninstalled according to their instructions again, searched for all "zoom" files in my disk and rebooted. This leads me to believe that following their uninstall instructions is insufficient, and there are hidden files left on my computer. Sorry in advance for the off topic message [1] https://support.zoom.us/hc/en-us/articles/201362983-How-to-uninstall-Zoom-on-a-Mac- https://support.zoom.us/hc/en-us/articles/201362983-How-to-u...

sdan 7y ago

Weird for Twitter to still be up and fully functioning. I thought they migrated everything to GCP this/last year?

sorenn111 7y ago
Not the main functionality of the service, just lots of data analysis tooling. nothing that end users would notice
- sdan 7y ago
  Interesting. Thought I had read some posts of them migrating their data, but you could definitely be right.

f_martinez 7y ago

We are on region us-east1 and our systems are still up. Specifically, we are on us-east1-b.

jshprentz 7y ago

The two Google Cloud networking incidents are: Incident #19008 began at 2019-06-02 12:48. https://status.cloud.google.com/incident/cloud-networking/19008 https://status.cloud.google.com/incident/cloud-networking/19... Incident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... Times are US/Pacific

pm90 7y ago
They both seem to say the same thing....
boulos 7y ago
dang et al.: I’d make 19009 the URL for this (GCE Just reported first, but this is an outage / networking issue in the Eastern US).
- AnssiH 7y ago
  #19008 mentioned by parent is not the GCE one, there are two networking ones: #19008 and #19009. The #19008 (networking) says there will be an update by "13:30 US/Pacific", but as of 17:05 there is no update. Similarly, #19003 (GCE) says there will be an update by "16:00 US/Pacific", but no update as of 17:05. All the latest updates seem to only be in the third incident #19009 (networking).
cheeze 7y ago
Looks like they are having trouble updating their statuses. 19008 was supposed to be updated over an hour ago. Meanwhile, 19009 has the same comment posted three times. I'm guessing internal tools are barely working at best.
- londons_explore 7y ago
  That is normal for Google status pages. They don't want to admit fault or place blame because there can be legal and commercial ramifications, so they can only say canned responses.

jamisonbryant 7y ago

I've noticed problems on GDrive (GSuite) and YouTube as well. Connected?

mindcrime 7y ago
FWIW, Youtube is fine for me, but I'm seeing intermittent errors saving updates to a document in GSuite. I had thought the latter error was a problem with the wifi where I am, until I saw this. Now I'm not so sure. HN is loading fine on the same wifi...

ikeboy 7y ago

So that's why YouTube was being weird. I thought it was an extension problem or something.

sarim 7y ago

Funny how as soon as I realized that Gmail and Google Sheet aren’t working properly I rushed to HN to figure out what’s going on. I love this community!

sgammon 7y ago

So far the Ko list: GCE, GKE, BQ, Pub/Sub, GAE asia-south1 us-west1 us-central1 us-west2

captn3m0 7y ago
Our GKE stuff in asia-south1 (Mumbai) is up, and the GKE console works fine here. (Bangalore).

maz1b 7y ago

Ironically, I moved all of my objects off GCS today.

javabean_ 7y ago

could this be the result of another BGP hack ? cyberwarfare ? I am just speculating here big time.

pishpash 7y ago

Let's see if perfect leetcode skills will save the day. /s

CSDude 7y ago

With Google Cloud incidents, most of the time whole regions fail, and with AWS generally only a region fails. Of course there would be exceptions, but Google Cloud does not make me feel safe as an outsider (and a user of multi-region AWS)

gist 7y ago

Right now this is at the 'good at least it's not mine to fix or worry about' kind of like 'and the reason I choose IBM' [1]. I can just sit back and wait for gmail to work correctly. Now at the point it starts to last what I would consider a long time well then I will have things to worry about. One thing with gmail though. When it's down it's similar to a snow storm if you only do business in a city. Everyone is impacted and everyone understands a missed deadline is unavoidable. [1] For those not old enough to know what I mean read this: https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalcomputer/words/ https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalc...

cobaimelan 7y ago

seems like only us down

d2xdy2 7y ago

Was certainly an interesting alert when my Cloud Functions started reporting downtime. Among the many things that dip in and out on what seems like a monthly basis, I’ve not seen them just drop out in quite a while. Hopefully they get things sorted out. I can’t really imagine what it looks like internally when this level of outage is going on, but I want to think everyone is fairly collected

deleted 7y ago

[deleted]

javabean_ 7y ago

Nice overview : https://downdetector.com/ https://downdetector.com/

XCSme 7y ago
I see a lot of Nest complaints, isn't there any security issue if Nest goes down?
- pmlnr 7y ago
  I see people complain about Nest and the only thing I can think of is "what on earth where you thinking to have a door or thermostat that doesn't function without internet?!"
  - mrb 7y ago
    Nest thermostats can function without Internet. You can walk up to it and adjust the temperature just like any old thermostat.
  - disillusioned 7y ago
    I live in a house with a Nest thermostat and Nest Secure and Nest x Yale door locks. The AC is on just fine, and the door unlocked just fine. (The door lock doesn't require an internet connection, unless you've enabled privacy mode for some reason.)
    - pmlnr 7y ago
      I thought there were versions that don't work offline; my bad if this is not the case.
  - sofaofthedamned 7y ago
    Internet isn't the issue, it's the endpoint (Google) It's reasonably easy to run a router/gateway that has a 4G backup to get the ping out. Whether the ping works...
- 7y ago

ccarpenterg 7y ago

Google Colaboratory is back. At least I can access my github's notebooks and public notebooks from google drive.

cjohansson 7y ago

On days like these I’m glad I don’t use any of the affected services

biohax2015 7y ago

Does anyone know if this is a regional or global outage? I can see my GKE clusters in one region but not in another, so I am guessing it's the former. Looks like we'll need a cluster in each region going forward...

boulos 7y ago
To help others: which two regions?
- biohax2015 7y ago
  us-central1 is up us-east4 is down
  - hoonto 7y ago
    us-west2 is down
XCSme 7y ago
I can not access vimeo from EU, not sure if related to Google though.

peterwwillis 7y ago

The last big outage, iirc, was Google didn't test their rollback procedures for router upgrades. I'll be very interested to hear if it's yet another change control problem that caused this outage.

amasad 7y ago

Anyone experiencing issues with GCS? Seems highly intermittent and dependent on the location the request is coming from (maybe that's because it's a networking bug). The status page says GCS is fine but that's highly unlikely.

ejanus 7y ago

I just can't reach google apps on my HTC m9 since yesterday. I am in West Africa. My Whatsapp crashed too and I lost all my previous threads. Is my issue related to Cloud being down?

captn3m0 7y ago

What I've realized from this: Google doesn't have an official status page for GCP. There are a few unofficial ones, but nothing official that I could find.

cheeze 7y ago
Like this? https://status.cloud.google.com/ https://status.cloud.google.com/
- captn3m0 7y ago
  I meant Twitter Status Page (in my defense it was 3 am). The only ones are: @gcpstatus and @gcp_incidents, both unofficial

codemati 7y ago

https://tour.golang.org https://tour.golang.org is also down

outside1234 7y ago

It seems crazy to me that Google Cloud can have this level of instability but I, on the other hand, can never remember google.com going down. Why are they operating one with a different networking infrastructure from the other?

chupasaurus 7y ago
AFAIK google.com is running independently on a different infrastructure, but some of other services are relying on GCP hence the problem affects them too.
sakisv 7y ago
They did have a major outage for a few minutes on 2013: https://www.theregister.co.uk/2013/08/17/google_outage/ https://www.theregister.co.uk/2013/08/17/google_outage/
- escapecharacter 7y ago
  So this is now the longest outage ever, right?
savrajsingh 7y ago
Sounds like a good design — but in the early days of google cloud — 2008 — you’d get the google homepage on a bad request
vl 7y ago
It's completely different mode of failure - when part of search is down somewhere, it's just routed to the same service in other location. If part of the GCP is down anywhere, some customers are affected. Since original Google infrastructure was developed specifically for first kind of services, cloud org still has problems adopting it to its needs.

chpmrc 7y ago

This might be all in my head but I've been experiencing really bad latency for like an hour, while browsing, and then I read this.

m0zg 7y ago

Prediction: the final postmortem will say "someone pushed a bad config", just like most of the previous postmortems (and most of the internal postmortems as well, for Borg-based services). This is the cause of most other outages in other cloud providers as well. A really hard to solve problem.

krackers 7y ago
Multiple regions seem to be affected though. Wouldn't it make more sense to start out config pushes with a single region before expanding it to avoid these types of wide outages?
- m0zg 7y ago
  It's a networking outage. Google is a well known user of SDN. Networks, by virtue of connecting things, necessarily affect more than one region.

bluefox 7y ago

You look up. A single ray of light has made its way to the earth. Some day, you hope, the sky will be clear again.

typenil 7y ago

Was trying to set up SSL on a GKE cluster today. Guess I'll have to wait for tomorrow if I want to be able to tell my mistakes apart from Google's.

wenbin 7y ago

forbes.com is down also?

AdmiralAsshat 7y ago

And Thunderbird suddenly kept throwing prompts for my Gmail passwords, even after I sign back in. I hope it's related.

captn3m0 7y ago
It says "could not reach imap.gmail.com" for me.

titzer 7y ago

This is your yearly reminder to resist centralization of the internet.

quickthrower2 7y ago
Yes, I haven't read the Bitcoin is down or BitTorrent is down status yet.
jasonvorhe 7y ago
Because Internet Exchanges and carriers never had network issues.
- eitland 7y ago
  Because then at least you and your colleagues has a chance to get some work done.

el_benhameen 7y ago

That explains why my google home thing thinks it’s sub-zero even though I’m warm in shorts and a t-shirt. https://pasteboard.co/IhBsyrsO.jpg https://pasteboard.co/IhBsyrsO.jpg

lewis1028282 7y ago

So just got a new phone and trying to resync my 2FA for AWS but I can't because Gmail is down. Ffs.

netmonk 7y ago

There is a loop in the spanning tree.....

finphil 7y ago

Working here but slow (I'm based in Central Canada).

deleted 7y ago

[deleted]

unilynx 7y ago

G suite was broken for me 20 minutes ago (in Europe) but is working now. Perhaps things are starting to come back?

based2 7y ago

https://www.reddit.com/r/sysadmin/comments/bw1gye/most_google_gsuite_services_gmail_calendar_etc/ https://www.reddit.com/r/sysadmin/comments/bw1gye/most_googl... https://www.zdnet.com/article/some-internet-outages-predicted-for-the-coming-month-as-768k-day-approaches/ https://www.zdnet.com/article/some-internet-outages-predicte... 768k Day

_Marak_ 7y ago
This should be voted higher up. According to https://twitter.com/bgp4_table https://twitter.com/bgp4_table, we have just exceeded 768k Border Gateway Protocol routing entries, which may be causing some routers to malfunction.
- dreamer_soul 7y ago
  Isn't it weird that it's happening now even though that number was surpassed nearly a month and half ago?
  - dyu 7y ago
    Different locations see different counts because of aggregation/de-aggregation.
juanuys 7y ago
Will this affect more than just Google? I haven't seen any outages from other cloud providers.
- namibj 7y ago
  packet.net was hit. Specifically, also their San Jose DC. Internet only. It took less than an hour to recover. More than 20 minutes. I didn't ping it continuously, and I can say that the traceroute got stuck in Frankfuhrt (Where my ISP and their ISP first (as seen from me) meet). I was actually surprised, as they tend to have excellent networking. Now I'm not nearly as distrusting as I was initially, knowing it was likely their ISP getting screwed by routing table overflow.

tntn 7y ago

There goes 3 nines for June and for Q2. I guess everyone gets a 10% discount for the month? https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla

gundmc 7y ago
A couple more hours and everyone will get 25% off for June.
- twistedpair 7y ago
  Does that apply to the rest of June? Might be a good month to rebuild all your models ;)
  - quickthrower2 7y ago
    The vultures are circling.
CamelCaseName 7y ago
Ironically, the SLA page returns a 502 error.
londons_explore 7y ago
The discount seems way too small. I would pay a premium for a cloud provider happy to give 100 percent discount for the month for 10 minutes downtime, and 100 percent discount for the year for an hour's downtime.
- gundmc 7y ago
  Any cloud provider offering those terms would go out of business VERY quickly. Outages happen, all providers are incentivized to minimize the frequency and severity of disruptions - not just from the financial hit of breaching SLA (which for something like this will be significant), but for the reputational damage which can be even more impactful.

baalimago 7y ago

Playing "is google on this website or not?" just became so much easier, simply see if website works or not. Scary stuff. What happens when Murphy's law decides to crash things even more?

ramon 7y ago

No wonder I was trying Webcodesk right now and it's not working it's all firebase, yay!

ramon 7y ago

It seems that it`s focused on the database stuff like firebase and firestore.

dodobirdlord 7y ago
Apparently (from further up the thread) it's a network congestion issue causing extremely high rates of packet loss. I imagine pretty much anything that's homed in the effected regions will be degraded or unaccessible.

sneak 7y ago

A demo of “too big to fail [via antitrust]”?

noncoml 7y ago

There go the nine nines for this year? Is it more like four nines or so now?

skizm 7y ago

Ah, wonder if this is affecting Google's SSO. It was super slow when I was trying it just now.

squarefoot 7y ago

I appears that logging into the webmail solves at least POP mail problems. I tried my mail client and failed, then attempted a login to the webmail which worked. Gmail then asked me to confirm my recovery address and cellphone, which I did, and finally loaded the inbox page. I immediately attempted a connection through the POP client and this time it worked. It might be something security related if it triggers a mandatory identity confirmation. edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error. [23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later. The problem seems much more complex.

pg_is_a_butt 7y ago

None of my stuff is down.

filvdg 7y ago

Level(3) one of the biggest backbones has issues too, might be related https://downdetector.com/status/level3 https://downdetector.com/status/level3

Yrlec 7y ago

Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: https://cloud.google.com/storage/sla https://cloud.google.com/storage/sla. So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.

crazygringo 7y ago
Ironically I can't read that page because, since it's Google-hosted, I'm getting an HTTP 500 error... but which means at least that service is SLA-covered... Cloud services live and die by their reputation, so I'd be shocked if Google ever tried to get out of following an SLA contract based on a technicality like that. It would be business suicide, so it doesn't seem like something to be too worried about?
deleted 7y ago
[deleted]

moon2 7y ago

Its funny how searching for "Google Down" on Google and filtering for results within 1 hour yields nothing.

stvkoch 7y ago

My gmail it's down!

StaticRedux 7y ago

This must be why GitLab is giving me shit. They recently made the switch. Wonder if there is any second guessing going on over there right now.

Uninen 7y ago
They have a incident report page up at: https://docs.google.com/document/d/1RM3QnuJ4FPH10J3UrJS0T26d-mn_Dd11-jmZweHQVV8/edit# https://docs.google.com/document/d/1RM3QnuJ4FPH10J3UrJS0T26d...
- StaticRedux 7y ago
  Using Google Docs to report problems with Google, including Google Docs. Irony at it's finest.
  - Uninen 7y ago
    They do have an internal ticket as well, but Gitlab.com is also hosted on Google, so... https://gitlab.com/gitlab-com/gl-infra/production/issues/862 https://gitlab.com/gitlab-com/gl-infra/production/issues/862
kerng 7y ago
I was just thinking the same. One of their reasons (besides being paid a lot by Google) was stability.
dsumenkovic 7y ago
You can find some more data about GitLab availability after the move to GCP here https://about.gitlab.com/2018/10/11/gitlab-com-stability-post-gcp-migration/ https://about.gitlab.com/2018/10/11/gitlab-com-stability-pos.... As we're trying to stay transparent as always, we'll definitely let everyone know if are going/thinking about the change.

eisvogel 7y ago

gmail is down in Australia...

atopuzov 7y ago

Hug your on-call engineer.

cedricium 7y ago

Majority of G Suite services are suffering service outages: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status

tlogan 7y ago

I also noticed that Google search stop indexing news articles. So I searched for "gmail down" on bing and I got some results [1]. But searching on Google for "gmail down" does not return any results [2]. [1] https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT [2] https://www.google.com/search?q=gmail+down&source=lnms&tbm=nws https://www.google.com/search?q=gmail+down&source=lnms&tbm=n...

griffinkelly 7y ago

Looks like Google Analytics isnt reporting stats either?

_xerxes_ 7y ago

Nest is down too, not surprising given they are part of Google. What I don't understand is why I can't still control my devices over my local network. Why does the system even require access to Google servers?

deleted 7y ago
[deleted]
fredthomsen 7y ago
That made me rage the other day when I had an internet outage. I am honestly fed up with Nest esp after this new Google Nest branding, and just disconnected my thermostats from the internet. I rarely changed it from my scheduled settings via the app anyway.
- jansan 7y ago
  Makes me wonder What your thermostats are doing on the internet in the first place.
good_guy 7y ago
becasue who gives a shit ? you already bought it.
kibwen 7y ago
Let's make the following the tech mantra of the next decade: "the Internet Of Things can never possibly work until the LAN Of Things does".
- nektro 7y ago
  the IoT trend really needs to stop
  - xhgdvjky 7y ago
    LANoT?
7y ago

deleted 7y ago

[deleted]

fdgwhite 7y ago

I tried multiple times to setup a Google Wifi router today. Wifi would work but the app said it was offline. Perhaps I am not insane or incompetent after all

ecabuk 7y ago

My builds are failing because it cannot download the chromium. > Error: Download failed: server returned code 502. URL: https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/662092/chrome-linux.zip https://storage.googleapis.com/chromium-browser-snapshots/Li...

the-dude 7y ago

Huawei just flexing its muscles. Nothing to see here, move along.

robdachshund 7y ago

Looks like this is why Shopify servers are down as well. Apparently last year they moved everything to GCP. I don't know why anyone uses them. They have proven to be highly unreliable and have horrible support. While I'm at it, I'm still bitter about gitlab switching to gcp from azure. They were trying to save face after the github acquisition and just moved on to an even more dubious company for their servers.

pitaj 7y ago

Looks like my Gmail is back, but I don't have any emails from while it was down. Yikes. Edit: just got one email from the downtime, so perhaps my initial conclusion was incorrect

Stryder 7y ago
Whoa, if confirmed this would be really bad. Downtime happens, but data loss is much worse.
- BillinghamJ 7y ago
  This is normal handling for SMTP - the sending server just tries again later.

chx 7y ago

These things happen. That's OK. Here's what's not OK: > We are investigating an issue with Google Compute Engine. We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific. The next update is at 12:59. Just ... no.

CallumRobertson 7y ago

Snapchat is fixed but snap maps is still disabled

CallumRobertson 7y ago

Snapchat is back up but snap maps is down

colo12 7y ago

Is it related to TianAnMen square massacre 6/4/1989 30th anniversary?

nmstoker 7y ago

Seems to have coincided almost exactly with my Chromecast stopping displaying my photos (in ambient mode).

fourty2 7y ago

google services also f´ed up here in middle europe; cannot reach anything google related from hamburg, germany.

londons_explore 7y ago

It seems the AdWords anti-spam system is down, which means anyone can put a billion dollar bid on every keyword and get their ads showing on every Google search for every query. Systems that fail 'open'...

good_guy 7y ago
lol, google "engineers" are overrated.

remote_phone 7y ago

Phase 2 of mitigation was completed a few minutes ago and GCE is expecting improvement shortly.

nopriorarrests 7y ago

https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003 >We will provide an update at 16:00 US/Pacific. it's 16:22 and no updates were posted. a bit unprofessional..

bbayer 7y ago

I think nobody mentioned but Adsense also is not updating for couple of hours.

joaomacp 7y ago

I was finishing a university assignment with the deadline 90mins away. I wanted to upload a video of the project to YouTube and add a link to it in the report. YouTube takes a long time to process the video, and then says it's unavailable. I go to Vimeo: it's down. I upload the video to Dropbox, and copy its link to the report. But my report was a Google doc. And when I tried to export it as PDF (which I had not done yet) it couldn't do it. I never hated google more. Eventually the video went through to YouTube, and I could export the PDF on the third try, but this really made me conscious of my dependance on Google.

misrab 7y ago

Anything to do with China?

bahmboo 7y ago

No, not really.

rum3 7y ago

Keep on centralizing the internet with your stupid clouds and this is what happens.

AngeloAnolin 7y ago

Does Google also have some sort of listing on which consumer apps are particularly affected (i.e. Gmail, Hangouts, Docs, Sheets, etc). The cloud components may be directly affected but for consumers, there's nothing which will provide info on what consumer facing services are getting some issues.

tcarn 7y ago

Once again HN proves to be the best status monitor.

ecthiender 7y ago

In India, I could access Youtube, Gmail (web) and Google cloud console and GKE and Compute Engine instances in south-east asia region.

bprasanna 7y ago

https://status.cloud.google.com/incident/compute/19003 https://status.cloud.google.com/incident/compute/19003 Does status update mean only the status. Won't there be words like "sorry", "apologize" & "inconvenience"! Only PR is responsible for those words?

jwandborg 7y ago
Apologies are unnecessary on status pages. You can be sure that they are sorry.

sandra101 7y ago

hello everyone , i dont need to talk much about , this you need to be careful with any hacker you give your work to , the trustworthy, most reliable hacker you need for repairing your credit score is Neo , you just need to mail him now and make your problem solve on neogonzalezhacks at gmail dot come or text him on 5139001324 , dont think about it , just mail him he will respond immediately , tell him from james Guinn , he help me with my credit score within 2 weeks , my credit score was very low and helped me repair it to 820, dont hesistate to mail him now , just mail him and rejoice .happy for you for doing that ..

aasssdddd 7y ago