12 ms·

GCP Outage

1468 points by thanhhaimai 1y ago

ddtaylor 1y ago
Smells like BGP since there are services people claim have nothing to do with GCP being affected. OpenRouter is down, Lovable is down, etc.
- DrBenCarson 1y ago
  npm as well
  - koito17 1y ago
    Initially attributed the unresponsiveness of `npm install` to npm (the CLI tool) in general. Tried using bun to install dependencies, saw the same result -- but with actual logs instead of a vague spinner -- and decided to check Hacker News. Getting 504 errors on anything from registry.npmjs.org that isn't cached on my machine.
    - yard2010 1y ago
      I just want to say that bun is a gift. It's just like npm, but backwards. So you imagine how perfect it is. I'm kidding, but really - bun is awesome. If you're using npm you can make the switch as it's mostly compatible.
- brown9-2 1y ago
  perhaps Lovable uses GCP somewhere in their stack?
- thallium205 1y ago
  AWS seems fine though. My bet is Cloudflare.
  - CSMastermind 1y ago
    AWS and Azure both had outages.
    - remram 1y ago
      Is that true? I see no direct report about that. downdetector says so, but it's crowdsourced so it tends to have fake positives.
      - CSMastermind 1y ago
        That's fair, I haven't seen any posts from the companies themselves.
    - deleted 1y ago
      [deleted]
coryvirok 1y ago
Shameless plug for https://rollbar.com https://rollbar.com Good luck out there!
CyrMeta 1y ago
if all services at down at once, no one is thinking or mentioning a potential attack on US cloud providers ? (China or Russia) Maybe ?
pancomplex 1y ago
thank god hn is hosted on a single bare metal server, free of all this bloat.
- cddotdotslash 1y ago
  I can recall plenty of times HN has been down.
  - rldjbpin 1y ago
    it has already been facing some issues in the past couple weeks. e.g. the "front" page was broken randomly for past dates (like https://news.ycombinator.com/front?day=2025-05-30 https://news.ycombinator.com/front?day=2025-05-30 or https://news.ycombinator.com/front?p=2 https://news.ycombinator.com/front?p=2) as per their api documentation [1], it might be linked to firebase, which might explain this? [1] https://github.com/HackerNews/API https://github.com/HackerNews/API
    - detaro 1y ago
      the API has afaik nothing to do with HN itself, its a read-only copy for API users, not what backs the actual site.
thanhhaimai 1y ago
The status page is green, but there are outages reported: https://downdetector.com/status/google-cloud/ https://downdetector.com/status/google-cloud/
- deleted 1y ago
  [deleted]
- ransom1538 1y ago
  Why can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing. We are truly in gods hands. $ prod Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists
  - jeanlucas 1y ago
    Because there are contracts related to uptime :)
    - rustc 1y ago
      Does any service even say they're "down" anymore? All I see is "elevated error rates".
      - colechristensen 1y ago
        4 to 6 hours after the flames are visible from orbit and management has finally given up on the 37th quick fix you do get that red X But really not until after it's been on CNN a while.
      - deleted 1y ago
        [deleted]
    - rixthefox 1y ago
      Those contracts will be monitoring their service availability on their own. If Google can't be honest you can bet your bottom dollar the companies paying for that SLA are going to hold them accountable if they report the outage properly or not.
matdehaast 1y ago
Not just GCP, most of Googles services are out of action
- milesward 1y ago
  I'm on a meet, in cal, editing a dozen docs, in GCP, pushing commits and launching containers; it's not clear yet what exactly is going on but it's certainly intermittent and sparse, at least so far
  - parpfish 1y ago
    stop it. you're overloading their system by doing three things at once. let the rest of us have a turn.
sourthyme 1y ago
Maybe cloudflare?
- nickporter 1y ago
  having issues with cloudflare as well
- aetherson 1y ago
  Cloudflare status page reports an issue: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
aetherson 1y ago
We're experiencing intermittent slowness and timeouts on our GCP everything.
dorkitude 1y ago
Same here. Even the page to submit support requests is down. Cloud console does nothing. They should host their support services on AWS and vice-versa.
- milesward 1y ago
  I just logged into several of my GCP accts, everything popped up, multiple home regions.. I wonder what % of folks are feeling this right now.
aetherson 1y ago
Does anyone know if it's region-specific? We're experiencing it and are in us-west-1.
- data-ottawa 1y ago
  Us-central-1 as well
- deleted 1y ago
  [deleted]
- vbb 1y ago
  europe (netherlands) region as well
- izolate 1y ago
  us-east-1 too
- johannes5117 1y ago
  Frankfurt seems to be down as well
- mccoyc 1y ago
  Can confirm us-east1 (and possibly us-south1) are having VPC host reachability problems.
- NeT8235 1y ago
  south korea as well
- tridao 1y ago
  it's due to IAM and global
ea016 1y ago
Google Cloud Storage seems to be down or very slow
digest 1y ago
love how their status page is green with no issues detected!
- pbmango 1y ago
  https://www.canva.com/design/DAGqKquGD-c/xtRObgH1r_4RoulPAyscgA/view?utm_content=DAGqKquGD-c&utm_campaign=designshare&utm_medium=link2&utm_source=uniquelinks&utlId=h74dd0f5b14 https://www.canva.com/design/DAGqKquGD-c/xtRObgH1r_4RoulPAys...
hambro 1y ago
@dang could you merge this and https://news.ycombinator.com/item?id=44260669 https://news.ycombinator.com/item?id=44260669?
- toomuchtodo 1y ago
  No notifications for mentions, have to email the mods at the hn@ email address.
  - cwillu 1y ago
    Do we know if email is still working? kidding-but-not-really-because-gmail…
  - hambro 1y ago
    I think I was a bit optimistic in the response time from mods. This thread won the popularity contest quite well... Thanks for letting me know about emailing the mods, refreshingly explicit to send email.
niij 1y ago
Experiencing 504s in Google Meet. Google Cloud Console won't load.
ransom1538 1y ago
Yeah their status page is all green nothing to see here (but all production systems are down).
alexcroox 1y ago
Cloudflare KV is also having an outage. I wonder who is reliant on who here.
- hackermondev 1y ago
  seriously doubt Google Cloud is relying on Cloudflare KV lol
- dlewis1788 1y ago
  Looks like more than KV is having an issue. Just tried to load dash.cloudflare.com and no bueno.
conroy 1y ago
We're in us-west-1 and seeing issues across Cloud Run, Cloud SQL, Cloud Storage and Compute Engine.
eterm 1y ago
Cloudflare speedtest is down too, I assume because of this?
- clairegraham 1y ago
  Our site depends on Workers and KV and it's very broken right now. Can't login to the Cloudflare Dashboard either.
- voxadam 1y ago
  Works for me in Portland on Quantum Fiber.
- gfs 1y ago
  Appears to be a separate incident: https://news.ycombinator.com/item?id=44261064 https://news.ycombinator.com/item?id=44261064
  - reassess_blind 1y ago
    Two big cloud provider outages at the same time? Has to be related surely.
siliconc0w 1y ago
Gemini API isn't working for me :/
oalessandr 1y ago
Having issues with services in cloud run as well
zacharynewton 1y ago
Ahhh, explains why some of my apps are going crazy... Couldn't read a message from my kids pre-school Thankfully we use AWS at work for everything critical
atonse 1y ago
Getting a lot of errors for Claude Sonnet 4 (Cursor) and Gemini Pro. Nooooo I'm going to have to use my brain again and write 100% of my code like a caveman from December 2024.
- bicx 1y ago
  I was in the middle of testing Cloud Storage file uploads, so I guess this is a good time to go for a walk.
  - deleted 1y ago
    [deleted]
  - matsemann 1y ago
    A good excuse for adding error handling, which otherwise is often overlooked, heh.
- orangebread 1y ago
  lmao i refuse to write code by hand anymore too. WHAT IS THIS
- sujayakar 1y ago
  switch to auto mode and it should still work!
  - ashu1461 1y ago
    GPT is working in agent mode, which kind of confirms that claude is hosted on google and GPT probably on MSFT servers / self hosted.
    - kenhwang 1y ago
      If you want a stronger confirmation about Claude being hosted on GCP, this is about as authoritative as it gets: https://www.anthropic.com/news/anthropic-partners-with-google-cloud https://www.anthropic.com/news/anthropic-partners-with-googl...
gigatexal 1y ago
YouTube is also very flakey.
deleted 1y ago
[deleted]
Brystephor 1y ago
some core GCP cloud services are down. might be a good time for GCP dependent people to go for a walk, do some stretches, and check back in a couple hours.
mlb_hn 1y ago
Our GCP is down
- milesward 1y ago
  What region?
  - ashu1461 1y ago
    I think multiple regions are down. asia-south, us-east atleast are impacted.
    - a_void_sky 1y ago
      asia-south is working for me
  - deleted 1y ago
    [deleted]
sergiotapia 1y ago
xAI having problems, Supabase down, Discord can't upload images to share in chat. Seems like a major backbone outage.
- saltcod 1y ago
  We're investigating right now. Looks like a potential issue with Cloudflare.
  - atsaloli 1y ago
    https://www.cloudflarestatus.com/incidents/25r9t0vz99rp https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
  - faizanrupani 1y ago
    You're right, https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, and claude outage.
    - cyberflame 1y ago
      Cloudflare uses GCP as a provider - it's something more upstream
theflyinghorse 1y ago
identitytoolkit.googleapis.com is 503-ing on us, my whole customer success team is locked out from our platform
madjam002 1y ago
Google Maps not loading, thought it was my 4g, go to see if my connection works by loading Hacker News, GCP Outage XD
evtothedev 1y ago
Yarn package registry also appears to be down.
- tom1337 1y ago
  npm is, registry.yarnpkg.com is only a CNAME to npm
kfarr 1y ago
Yes Firebase auth is down and affecting many apps, on Discord and Slack groups tons of others are corroborating. A bit disappointing that there is no post on the status page for nearly 30 mins: https://status.firebase.google.com/ https://status.firebase.google.com/
- kentlyons 1y ago
  It just updated. Maybe affected by their own outage!
  - ashu1461 1y ago
    Just proves how shady the status page and sla stuff is
    - dgellow 1y ago
      or how difficult it actually is to do that type of thing at scale
    - rco8786 1y ago
      Google is 10 minutes late updating their status page. "So shady" It's really, really hard to make a status page realtime.
      - ashu1461 1y ago
        What makes you think it’s hard? We have AI generating songs and writing code, but setting up basic health checks is too much?
        jug 1y ago
        An AI generated status page would be the epitome of 2025.
        urbandw311er 1y ago
        What makes you think it’s easy?
_kush 1y ago
Supabase is also down
- vpuna 1y ago
  Yes my project on Supabase is down as well.
andrewmcwatters 1y ago
Ah darn it. My Spotify DJ just stopped working.
deleted 1y ago
[deleted]
capital_guy 1y ago
BigQuery is completely dead
b0a04gl 1y ago
[dead]
TN1ck 1y ago
Cloudbuild completely down for us. Getting "Visibility check was unavailable" errors.
quectophoton 1y ago
Twitch was broken too: https://status.twitch.com/incidents/b79nyp1yhxql https://status.twitch.com/incidents/b79nyp1yhxql EDIT: Updated link to point to the specific incident.
- rplnt 1y ago
  Is Amazon running Twitch on Google Cloud (at least partially)?
  - quectophoton 1y ago
    I don't know, at this point I don't know who uses what. This is maybe unrelated but even BunnyCDN has an incident from a few hours ago (https://status.bunny.net/incidents/6g27lbtp67m4 https://status.bunny.net/incidents/6g27lbtp67m4). Seeing how everything seems to be broken everywhere, I'm very much looking forward to the post-mortem.
rcfox 1y ago
I'm able to login to the GCP dashboard, but it isn't able to find any of my projects.
- deleted 1y ago
  [deleted]
keizo 1y ago
Yup, intermittent db connection issues and cloud storage problems.
deleted 1y ago
[deleted]
admissionsguy 1y ago
Wish there existed a decentralized network connecting computers around the world
- redman25 1y ago
  Crazy, they could call it the "internet" or something like that... kind of rolls off the tongue.
tsouth 1y ago
Everyone is down. Cloudflare has problems too. All auth providers broken.
deleted 1y ago
[deleted]
traeregan 1y ago
For us Cloud SQL instances are toast but App Engine Standard instances are still serving requests. Google Cloud console is borked too, mostly just erroring out.
ekojs 1y ago
Super duper frustrating having the status page being green. Why can't Google do this properly?
- supportengineer 1y ago
  Those responsible have been sacked.
  - 18172828286177 1y ago
    Those responsible for sacking the people who have just been sacked, have been sacked.
imzadi 1y ago
Can't reach my nest thermometer, but their status page says it's fine lol
- charliemeyer 1y ago
  the real concerns in life
- andrelaszlo 1y ago
  This is pretty crazy :D How did it affect you?
  - imzadi 1y ago
    I almost died
    - andrelaszlo 1y ago
      Shock, overheating, hypothermia, or a combination of all three?
tmiku 1y ago
Looks like I'm about to start learning which of my time-killing websites are hosted on GCP - The Ringer is down, and since Spotify owns them and is a major GCP customer, it looks like they've been hit by this. CRAZY that the GCP status page is still green.
ashu1461 1y ago
When you deploy code generated by Gemini :D
whitedurna 1y ago
i think it'll be disaster.
whalesalad 1y ago
Meet is also down for me right now. Cannot attend any video calls.
braunshedd 1y ago
Our GCP workloads are unavailable across several US regions. The GCP console is intermittently unavailable for most pages. Crossing my fingers for a quick resolution.
morgandoane 1y ago
Storage, CloudRun, Firebase...... All down....
- dana321 1y ago
  Auth, GCP, Windsurf,Augment Code,Udio, the list is endless. Facebook, Reddit and Hacker News is still up, but thats about it
vpuna 1y ago
is supabase on GCP ? My Supabase projects are down.
- duckarmada 1y ago
  Supabase is on AWS, but this is looking like an upstream Cloudflare issue. https://status.supabase.com/ https://status.supabase.com/
devmor 1y ago
Not just GCP. AWS and Cloudflare too. Did someone screw up BGP again?
- remram 1y ago
  Source? We didn't see anything wrong with AWS here.
timcavel 1y ago
[dead]
deleted 1y ago
[deleted]
atgmal 1y ago
[dead]
akash8400 1y ago
GKE workloads are also affected.
alkalaine 1y ago
[flagged]
- supportengineer 1y ago
  Take an early lunch
akash8400 1y ago
[dead]
alkalaine 1y ago
[flagged]
elmenosdramatic 1y ago
[flagged]
elmenosdramatic 1y ago
[flagged]
meltyness 1y ago
Can't upload discord attachments from mobile.
Axsuul 1y ago
Does anyone know if instance-to-instance networking has been affected? My Redis instance has been throwing a lot of connection errors.
- markbnj 1y ago
  We're not seeing any connectivity issues between pods and vms in our vpc, but your mileage may vary.
  - Axsuul 1y ago
    Thanks
18172828286177 1y ago
YouTube was down for me for some time
dbacar 1y ago
kaggle not responding correctly, is it related?
faizanrupani 1y ago
https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, claude outage, firbase outage https://status.firebase.google.com/ https://status.firebase.google.com/
- andrelaszlo 1y ago
  How would Cloudflare's outage cause a GCP outage? I'm sure it's not entirely impossible, but sounds backwards to me. Sure - a lot of the internet relies on Cloudflare, but I'd be very surprised if GCP had a direct dependency on Cloudflare, for a lot of reasons. Maybe I misunderstood your comment?
paulddraper 1y ago
"No major incidents" as of 11:37 PDT. https://status.cloud.google.com/ https://status.cloud.google.com/ File that in the status pages worth ~0 category.
jschroeder 1y ago
Status page is showing green because GCP admins can't login to change it ;)
tiagod 1y ago
Getting Gateway timeouts on docker hub. Maybe related? I can pull images. Example: https://hub.docker.com/layers/library/eclipse-mosquitto/latest/images/sha256-2909e32eb3728f6e8d67e878dae5ec22c00c5ae30ebca630af119b63c200fae2 https://hub.docker.com/layers/library/eclipse-mosquitto/late...
deleted 1y ago
[deleted]
ZiyadFarhan 1y ago
this aint looking good yall
sleepybrett 1y ago
npm registry happen to be hosted on gcp, because that seems to be down as well.
supportengineer 1y ago
Interesting how I landed here. I was having trouble with Nest. Then I went to Down Detector. I noticed many sites having a simultaneous uptick. Then I came to HN, and found this link at the top of the front page.
- geekamongus 1y ago
  I usually just go here first.
- ryanscio 1y ago
  Same here with npm https://status.npmjs.org/incidents/dn5mcp85737y https://status.npmjs.org/incidents/dn5mcp85737y
waythenewsgoes 1y ago
Status pages at cloud providers aren't usually based in reality -- usually requires VP level political games to actually get them changed especially for serious outages.
rvnx 1y ago
It looks like that it is a central service @ Google called Chemist that is down. "Chemist checks the project status, activation status, abuse status, billing status, service status, location restrictions, VPC Service Controls, SuperQuota, and other policies." -> This would totally explain the error messages "visibility check (of the API) failed" and "cannot load policy" and the wide amount of services affected. cf. https://cloud.google.com/service-infrastructure/docs/service-management/reference/rpc/google.api#control https://cloud.google.com/service-infrastructure/docs/service... EDIT: Google says "(Google Cloud) is down due to Identity and Access Management Service Issue"
- VWWHFSfQ 1y ago
  There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.
  - rvnx 1y ago
    Absolutely possible. Though there is something curious: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.". So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.
    - whatevertrevor 1y ago
      Doesn't cloudflare have its own infrastructure, it's wild to me that both these things are down presumably together with this size of a blast radius.
      - cyberpunk 1y ago
        You'd think so wouldn't you? DownDetector also reports azure and oracle cloud, I can't see then also being dependant on GCP... I guess down detector isn't a full source of truth though. https://ocistatus.oraclecloud.com/#/ https://ocistatus.oraclecloud.com/#/ https://azure.status.microsoft/en-gb/status https://azure.status.microsoft/en-gb/status Both green
AlgebraFox 1y ago
[flagged]
chief_jeef 1y ago
Firebase status page has acknowledged it as a "global issue". https://status.firebase.google.com/ https://status.firebase.google.com/ A contact in google mentioned to me that some bad update to Google Cloud Storage service has caused some cascading issues affecting multiple GCP services.
0xffany 1y ago
Everything appears to be down as of 18:43 UTC... https://downdetector.com/ https://downdetector.com/
- AlienRobot 1y ago
  Wait, it's all Google?
  - plateng000 1y ago
    "always has been"
  - deepsun 1y ago
    Google was the first to report probably.
  - bananapub 1y ago
    all cloud
- patapong 1y ago
  Perhaps their detection logic is running on Google cloud /s
  - throitallaway 1y ago
    I believe Downdetector displays user reports.
    - brentm 1y ago
      Yea I am pretty sure that if you're checking if a service is down your essentially casting a vote that indicates that service is down.
      - lysace 1y ago
        Kind of a missed opportunity for Ookla - who's running both downdetector.com and speedtest.net. The have software running in most ISPs around the world: https://help.speedtest.net/hc/en-us/articles/360039164793-How-can-I-host-a-Speedtest-server https://help.speedtest.net/hc/en-us/articles/360039164793-Ho... (OTOH, it's not always trivial to define/detect an outage.)
deadbabe 1y ago
If LLMs are down work grinds to a halt until they return. Just the new era now.
deleted 1y ago
[deleted]
kachapopopow 1y ago
I just realized that the reason the status isn't updated is cause they can't access it lol.
- paulddraper 1y ago
  Don't host status pages (or their dependencies) on your own infra lol. Seems obvious.
  - deathanatos 1y ago
    It should be obvious because both AWS and Azure have done this in the past and shown what a bad idea it is…
- Axsuul 1y ago
  How do you know that?
fidotron 1y ago
It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2eo4R29 https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e... "Firebase Data Connect unavailable due to a known Google Cloud global outage" While the Google Cloud status page https://status.cloud.google.com/ https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it. Edit to add: within 10 minutes of this post Google updated their status page. More curiously the Firebase page I linked to has been edited to remove mention of Google Cloud in the status and now says "Firebase Data Connect is currently experiencing a service disruption. Please check back for status. ".
- cyberflame 1y ago
  Services are recovering in some locations it seems - Discord is healing
- alexcroox 1y ago
  CF too: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
- kjuulh 1y ago
  Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.
  - sss111 1y ago
    I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!
    - paulddraper 1y ago
      Most status pages are manual. At least some of the information has to be. The weird part is that it took them almost an full hour to update it.
deleted 1y ago
[deleted]
paxys 1y ago
Kinda funny that the top post on HN titled "GCP Outage" links to the Google Cloud status page which shows...no outage.
deleted 1y ago
[deleted]
kgwxd 1y ago
Sorry, after decades of being hard wired, I just installed a PCIe Wifi6 card on my desktop. Internet took a dive the second I got it connected. Must have done something wrong.
happy-camper 1y ago
Text messaging on Android is broken
0xCAP 1y ago
> No major incidents … Proceeds to show worldwide degraded service level alerts.
- jimt1234 1y ago
  Yep. Self-reporting status pages are pretty near worthless. At my former large company (not FAANG), we weren't allowed to update the status page until we got VP approval, which also required approval from both PR and Legal. It would take a lot more time and effort to get those approvals than to just fix the problem and move on.
  - iFred 1y ago
    SLA contracts, clawbacks, and performance obligations make these pages a bit of a minefield for CSPs. When I was at a top-tier CSP, we had the status page that was public, one that was for a trusted tier of customers, one built for a customer-by-customer basis, and one for internal engineering.
    - genewitch 1y ago
      When i worked at a top tier speakeasy, we had a book up front for the man, a book in the back for the boss, a book for the trusted accountants...
happy-camper 1y ago
Text messaging for android is broken as well
ipsum2 1y ago
Cloudflare is down too. From https://www.cloudflarestatus.com https://www.cloudflarestatus.com: Update - We are seeing a number of services suffer intermittent failures. We are continuing to investigate this and we will update this list as we assess the impact on a per-service level. Impacted services: Access WARP Durable Objects (SQLite backed Durable Objects only) Workers KV Realtime Workers AI Stream Parts of the Cloudflare dashboard Jun 12, 2025 - 18:48 UTC Edit: https://news.ycombinator.com/item?id=44261064 https://news.ycombinator.com/item?id=44261064
- paulddraper 1y ago
  Broken link? EDIT: Weird, definitely was just empty
  - ipsum2 1y ago
    Should work, but its also on the front page.
- 0xy 1y ago
  Seems like a major wtf if Cloudflare is using GCP as a key dependency.
  - a2128 1y ago
    Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it
    - lysace 1y ago
      That's what IRC is for. (Its Finnish inventor is incidentally working for Google in Stockholm, as per https://en.wikipedia.org/wiki/Jarkko_Oikarinen https://en.wikipedia.org/wiki/Jarkko_Oikarinen)
    - IX-103 1y ago
0xffany 1y ago
Does anyone know of a good dashboard to check for such BGP routing anomalies as (apparently) this one? I am currently digging around https://radar.cloudflare.com/routing https://radar.cloudflare.com/routing but it doesn't show which routes were actually leaked. I would love if anyone has any good tool recommendations!
- mrngm 1y ago
  My default go-to: https://bgp.tools/ https://bgp.tools/ Why would you think this outage is (internet) BGP related?
  - whalesalad 1y ago
    Cloudflare runs all their own bare metal servers. Seems odd that they would be impacted by Google cloud. Same can be said for all the other issues on downdetector. This points to a broad issue at the core internet which could certainly be related to BGP.
    - ilkkao 1y ago
      Cloudflare is now saying: "Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency." I really hope CF explains this apparent Google dependecy in detail in their post mortem.
      - whalesalad 1y ago
        Imagine it's just a google spanner wrapper lmao
- SparkyMcUnicorn 1y ago
  I don't know if I've seen CF Radar before. That's pretty cool! Here are some others, although some seem to be experiencing issues due to the current outage I can only presume. - https://atlas.ripe.net/probes/public https://atlas.ripe.net/probes/public - https://www.ihr.live/en/global-report https://www.ihr.live/en/global-report - https://www.ihr.live/en/network https://www.ihr.live/en/network - https://bgp.he.net/ https://bgp.he.net/ - https://ioda.inetintel.cc.gatech.edu/dashboard/asn https://ioda.inetintel.cc.gatech.edu/dashboard/asn
stepri 1y ago
They have updated the status page finally https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4 https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
acureau 1y ago
Well this explains the issues I've been having with Spotify through the last hour.
artooro 1y ago
What's crazy is that RCS messaging is down as a result of this outage. It shows how poorly the technology or infrastructure was designed.
- dcchambers 1y ago
  Oh my god is that why my RCS chats were failing earlier?!?!
- wbl 1y ago
  That explains why I couldn't get the photo of my parents dog today.
- foota 1y ago
  Isn't RCS basically just instant messaging? I don't know why it's surprising that it would be down.
  - roywiggins 1y ago
    I'm not sure any single company could have an outage that would take out SMS globally, but RCS is presumably more centralized.
    - watusername 1y ago
      It used to be kind of distributed, but Google has been strong arming carriers to use their hosted Jibe service through a combination of proprietary extensions (e.g., E2E which is finally standard) and bypassing carrier control (if the carrier didn't provision RCS, Google Messages would use their own service iMessage-style). From the end user's perspective, if the carrier didn't use Jibe RCS, it simply wouldn't work well.
    - whynotminot 1y ago
      People liked to be utterly pissed at Apple for not supporting RCS. But there were reasons
    - toast0 1y ago
compscidr 1y ago
well this explains so much lol
ilovebabyyoda 1y ago
It looks like more than GCP: outages reported across the board including aws https://downdetector.com/ https://downdetector.com/
- koliber 1y ago
  About the only thing not down is down detector.
  - tonyhart7 1y ago
    god send omg, imagine down detector is down lmao anyone know what tech stack they use and where they host
ekojs 1y ago
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S... > Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue IAM issue huh. The post-mortem should be interesting at least.
- yard2010 1y ago
  Ha. With all this soviet style euphemism I rather read the onion instead.
  - bananapub 1y ago
    It’s not a euphemism - every outage, including the 99.9% that don’t end up on HN gets a postmortem document written about it, which is almost always a fascinating discussion of the technical, cultural and organisational situation that led to an unexpected bad thing happening. Even a few years ago senior management knew to stay the fuck out except for asking for more info.
deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
xan_ps007 1y ago
Where are the AI agents?
- dionys 1y ago
  Poor agents, finally taking a break
  - kyleee 1y ago
    The AI is over employed
Jayakumark 1y ago
Someone must have checked in AI Generated code :-)
cyrux004 1y ago
GPay which is a widely used payment service in India is down as well
- edm0nd 1y ago
  India is having a really bad day today
unsupp0rted 1y ago
The last few times this happened I wouldn't have thought "So this is the day AI takes over". But this time...
agawish 1y ago
GCP status page now reflect the issues, looks like Google Cloud Dataproc, Google Cloud Storage and Identity & Access Management
dineshsingh1 1y ago
https://downdetector.in/ https://downdetector.in/
SenpaiHurricane 1y ago
Now my api can not connect to PostreSQL... sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3
DonHopkins 1y ago
Damn you Bart Simpson! https://en.wikipedia.org/wiki/Bart_Gets_Famous https://en.wikipedia.org/wiki/Bart_Gets_Famous
dinvlad 1y ago
#HugOps
devmor 1y ago
Incident report published: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
james_m1231 1y ago
My firebase hosting and firestore db are back online, but GCP console and Google SQL instances are still having serious issues as of 7:00pm UTC.
phoenix98 1y ago
when its going to be fixed i am seeing now more and more services are down?
chupamela 1y ago
internal systems at google are currently broken.
phoenix__1998 1y ago
when its going to be fixed i am seeing now more and more services are down?
dinvlad 1y ago
Surprised no one else mentioned "it's always DNS" yet :-)
phoenix__1998 1y ago
when its going to be fixed , i am seeing now more and more services getting outage started with IAM ?
kjuulh 1y ago
I wonder what the damage ($) for having a good portion of the internet down for an hour or two ;)
throwcarsales 1y ago
My friends and I are even having trouble getting Rcs text messages to send.
CyrMeta 1y ago
if everything down at the same time - No one is mentioning an attack on us cloud services ? ( China or Russia ) Maybe ?
cyberflame 1y ago
They've now added this as a major incident - before it just was listed under overview
dpedu 1y ago
reCAPTCHA affected? I couldn't log into my local utilities website due to a reCAPTCHA error. Downdetector agrees, but I interpret that site as dubious.
- marifjeren 1y ago
  Yeah recaptcha is down intermittently
ashwinsundar 1y ago
Interesting that all Digital Ocean services are fine...
baq 1y ago
One of these days in which the young engineers learn the concept of 'counterparty risk'.
DonHopkins 1y ago
What is this Touchable Grass stuff I keep hearing of?
bsamson05 1y ago
So frustrating, but here's a link to track status of this outage: https://status.anthropic.com/incidents/kn7mvrgb0c8m https://status.anthropic.com/incidents/kn7mvrgb0c8m
CrimsonCape 1y ago
I'm having trouble getting any Street View imagery. Can anyone else confirm?
- NameError 1y ago
  Yep, street view is not working at all for me
estee_ecstacy 1y ago
Seems recovering now
estees_ecstacy 1y ago
Seems recovering now
estees_ecstacy 1y ago
seems recovering
jamesrwhite 1y ago
Seems like a wider issue at Google than just GCP, the Sheets and Chat APIs are also returning similar "Visibility check was unavailable" errors.
- yunwal 1y ago
  Presumably many Google products run on GCP
throwaway7783 1y ago
Even though BigQuery is not listed in affected services, we see errors connecting to it
- tecleandor 1y ago
  It's listed by regions :(
leoh 1y ago
If Google Chat is down per https://www.google.com/appsstatus/dashboard/ https://www.google.com/appsstatus/dashboard/, the ability for Google engineers to communicate among themselves impaired, despite SREs having IRC as a backup.
- iamdelirium 1y ago
  Google Chat wasn't down for me throughout the entire incident.
- miohtama 1y ago
  Someone actually uses Google Chat...?
  - 00deadbeef 1y ago
    Google has a chat product?
  - asadm 1y ago
    it's the best
    - ZiiS 1y ago
      Well given how many they have decommissioned...
    - clhodapp 1y ago
      Oh no, that's how you know it's nearing the point of being reaped and thrown in the graveyard!
      - IX-103 1y ago
        Don't worry, they're not following the "deprecate and cancel" playbook for that. They seem to be using the "copy a competitor poorly" one. The few features I liked about it, that distinguished it from Slack, disappeared in the latest update.
      - 1y ago
jim180 1y ago
Claude Code is down :( too lazy to do manual conversion from Cocoapods dependency to SwiftPM
rectang 1y ago
Haha, I don't ordinarily spend a lot of time in the Google Cloud Console but just now I was debugging a squirrely OAuth issue with reCAPTCHA failing to refresh several days running. I'm getting this weird page error, and I think, "Is this an issue with my organization? [futz futz futz] Hey wait is GCP actually down?" And it turns out to be the top discussion on HN. XD
dgellow 1y ago
Well, good luck to all googlers dealing with this, that's not fun :(
enahs-sf 1y ago
Would be comedy if one of the progenitors of this took Sundar’s buyout offer yesterday and let the world burn today.
AIorNot 1y ago
sheesh so many side-affected issues accross all systems, maybe big tech companies like google shouldn't have laid off all those engineers.. https://www.google.com/appsstatus/dashboard/incidents/Eab7zGLWSaUJyh9c9R9L https://www.google.com/appsstatus/dashboard/incidents/Eab7zG... but no tech bros, just keep following your ketamine addled edgelord when he did this with twitter..
dang 1y ago
Related ongoing thread: Ask HN: Is Firebase Down? - https://news.ycombinator.com/item?id=44260669 https://news.ycombinator.com/item?id=44260669
cyberflame 1y ago
Everything except us-central1 is back up - it's recovering now though
admissionsguy 1y ago
Google denies the outage. https://x.com/Google/status/1933246051512644069 https://x.com/Google/status/1933246051512644069
- baggy_trough 1y ago
  "clearing cache and cookies"? what is this, 1997?
  - bosmanos 1y ago
    lol
  - andrelaszlo 1y ago
    First, check that nobody else in your family is making a call on the phone line that your modem is connected to, then make sure to disable your Internet Explorer add-ons before trying again.
    - gred 1y ago
      https://x.com/ProductHunt/status/1626586036402003970 https://x.com/ProductHunt/status/1626586036402003970
- ElijahLynn 1y ago
  for those who boycott X: https://nitter.net/Google/status/1933246051512644069 https://nitter.net/Google/status/1933246051512644069
desktopninja 1y ago
Borg and K8s were fighting for resources, so Gemini decided to take out DNS. Now a sysAdmin has to step in. * just trying to add a little humour. pretty stressfull outage. grarr!!
lawrenceyan 1y ago
Solana is up ¯\_(ツ)_/¯
pier25 1y ago
"All locations except us-central1 have fully recovered. us-central1 is mostly recovered. We do not have an ETA for full recovery in us-central1."
- kubectl_h 1y ago
  An hour later and everything is a mess in central-1. They seemed to jump the gun on that one. Doesn't matter if some dinky service like "AutoML Vision" is working, if GCS isn't, then they shouldn't post an optimistic message.
cyberflame 1y ago
Root cause has been identified and it's being resolved/monitored now
_2d30 1y ago
This is at least why Claude is dead: https://status.anthropic.com/incidents/kn7mvrgb0c8m https://status.anthropic.com/incidents/kn7mvrgb0c8m Also spotify isn't working for me so I assume that's also related. These are my most important productivity resources! Sad!
estee_ecstacy 1y ago
sentry is down https://status.sentry.io/ https://status.sentry.io/
johnnyApplePRNG 1y ago
This appears to be continuing to cascade over an hour later... wow... more and more services mentioned as completely down on the outage page. Kind of nice to not be glued to AI chat prompts for a while to be honest.
sigmaball 1y ago
Guess they used Jules to code their services :)
alexcroox 1y ago
2 hour outage at this point
plerpler 1y ago
GCP Artifact registry still down... Not accepting image push and showing 500 status code
augbog 1y ago
Cloudflare Outage also just updated > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information
- deleted 1y ago
  [deleted]
- togume 1y ago
  Is GCP the third party?
itdependsnet 1y ago
Any chance this is the root being that so many different services are effected? https://github.com/kubernetes/kops/issues/17433 https://github.com/kubernetes/kops/issues/17433
- yunwal 1y ago
  I doubt gcloud would be affected by an aws-specific cni. Unless maybe enough AWS users have a GCP backup environment that they flipped on all at once, but it seems unlikely
  - itdependsnet 1y ago
    good point. I took that as simply the example that they had in front of them but a generic issue.
- jamie0 1y ago
  https://cloud.google.com/kubernetes-engine/docs/release-notes-rapid#June_12_2025 https://cloud.google.com/kubernetes-engine/docs/release-note... google did release an update to gcp k8s today, seemingly shortly before the outage
pikdum 1y ago
Was just about to do a demo, but Google Meet was down. Tried to use Jitsi as a fallback, but couldn't log in because Firebase was down too. Ended up using a Slack Huddle, lol.
zeke 1y ago
mapbox maps seemed to be down for a few minutes about an hour ago. I wonder if it is related.
ferchysmn 1y ago
Hola
jwatte 1y ago
Let's say a typical base service (network attached RAM or whatever) has 99.99% reliability. If you have a dependency on 100 of those, you're suddenly closer to 99% reliability. So you switch to higher-level dependencies, and only have 10 dependencies, for a 99.9% reliability. But! It turns out, those dependencies each have dependencies, so they're really already more like 99.9% at best, and you're back at 99% reliability. "good enough" is, indeed, just good enough to make it not worthwhile to rip out all the upstreams and roll your own everything from scratch, because the cost of the occasional outages is much lower than the cost of reinventing every single wheel, nut, bolt, axle, bearing, and grease formulation.
geocrasher 1y ago
https://soundcloud.com/ryan-flowers-916961339/the-internet-is-down https://soundcloud.com/ryan-flowers-916961339/the-internet-i...
max4c 1y ago
[flagged]
quyleanh 1y ago
Look like affect to Cloudflare as well [1] Update - Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. Jun 12, 2025 - 19:57 UTC 1: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
asim 1y ago
Just our bi-yearly reminder of our over reliance on cloud providers for literally everything. Can't say there's an answer beyond trying to build more independent tech but we know how that goes.
- ocdtrekkie 1y ago
  Hilariously, I did not know about any outages today during the workday because we discourage cloud service usage and nobody complained about anything breaking. :)
- ManBeardPc 1y ago
  Yet migration to the cloud continues, driven by people arguing that doing it yourself is too complicated and expensive. Let’s see how long until one outage takes down the global economy for multiple days or weeks.
  - JohnMakin 1y ago
    "The cloud" isn't the problem nor is migration to it. The problem is single points of failure in a vast, super-connected, global network. To demonstrate my point, say someone like cloudflare opted to get "off cloud" and run their own datacenters. Half the web would still go down if they had some issue in their datacenters. If anything, the economy of scale and resiliency of a huge cloud network is far beyond what any single operator can ensure for their own service. If this wasn't the case, cloud services couldn't be as profitable as they are. It isn't the panacea people seem to think it is. One critical service going down, regardless of whether it's cloud hosted or not, has ripple effects in the broader network.
LZ_Khan 1y ago
Is this the new Y2k?
riknos314 1y ago
Actual incident link posted: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
makk 1y ago
And THAT, Smithers, is why we wear hardhats on the job.
kodisha 1y ago
> Waiting for downdetector.com to respond...
ManBeardPc 1y ago
The cloud enables you to scale. It allows us to distribute systems across multiple regions and data centers. Seems that this is true for outages as well. The PHP application I wrote as a student running on a single self-hosted server had a higher uptime than any of the cloud providers or redundant system I have seen so far. If you don’t need the cloud for scalability, do it yourself and save yourself the trouble and money. Most companies would be better off investing into some IT staff instead of giving away their systems in the hands of some proprietary and insanely complex cloud environment. You are becoming dependent on someone you don’t know, have no control over and can’t talk with directly. Also the single point of failure is just shifting: from your system to whatever system is managing the cloud. Guess one advantage is that you can shift the blame to someone else…
herpderperator 1y ago
When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?
- slt2021 1y ago
  so much for System Design interview and bs gatekeeping...
- jlhawn 1y ago
  The outage was global. For my team specifically, a global Identity and Access Management outage meant that our internal service accounts could not refresh their short-lived access tokens and so different parts of our infrastructure began to fail over the course of an hour or so, regardless of what region or zone they were in. Services were up, but they could not access critical GCP services because of auth-related issues which resulted in internal service errors for us. To give an example, our web servers connect to our GCP CloudSQL database via a Cloud SQL Auth Proxy (there's also a connection pooler in between but that also stayed up). The connection to the proxy was always available, but the proxy wasn't able to renew auth tokens it uses to tunnel to the database, regardless of where the webserver or database happened to be located. To mitigate this in the future we're planning to stop using the auth proxy and connect directly via mutual TLS but now it means we have to manage TLS certificates.
marcinignac 1y ago
Can't wait to see how charts are going to look like here on the project we have developed for Maintel https://variable.io/maintel-digital-landscape/ https://variable.io/maintel-digital-landscape/. It shows availability across multiple services as a landscape. Expecting to see a lot of spikes tomorrow.
biglyburrito 1y ago
I wonder how many SLAs Google blew out today with this outage.
michaelawatkins 1y ago
[dead]
knuppar 1y ago
Spotify was not loading, thought my 5G was bad, used YouTube Music instead without issues. Hmmm...
curtisszmania 1y ago
[dead]
anonfordays 1y ago
[flagged]

1468 points by thanhhaimai 1y ago

Smells like BGP since there are services people claim have nothing to do with GCP being affected. OpenRouter is down, Lovable is down, etc.

DrBenCarson 1y ago
npm as well
- koito17 1y ago
  Initially attributed the unresponsiveness of `npm install` to npm (the CLI tool) in general. Tried using bun to install dependencies, saw the same result -- but with actual logs instead of a vague spinner -- and decided to check Hacker News. Getting 504 errors on anything from registry.npmjs.org that isn't cached on my machine.
  - yard2010 1y ago
    I just want to say that bun is a gift. It's just like npm, but backwards. So you imagine how perfect it is. I'm kidding, but really - bun is awesome. If you're using npm you can make the switch as it's mostly compatible.
brown9-2 1y ago
perhaps Lovable uses GCP somewhere in their stack?
thallium205 1y ago
AWS seems fine though. My bet is Cloudflare.
- CSMastermind 1y ago
  AWS and Azure both had outages.
  - remram 1y ago
    Is that true? I see no direct report about that. downdetector says so, but it's crowdsourced so it tends to have fake positives.
    - CSMastermind 1y ago
      That's fair, I haven't seen any posts from the companies themselves.
  - deleted 1y ago
    [deleted]

coryvirok 1y ago

Shameless plug for https://rollbar.com https://rollbar.com Good luck out there!

CyrMeta 1y ago

if all services at down at once, no one is thinking or mentioning a potential attack on US cloud providers ? (China or Russia) Maybe ?

pancomplex 1y ago

thank god hn is hosted on a single bare metal server, free of all this bloat.

cddotdotslash 1y ago
I can recall plenty of times HN has been down.
- rldjbpin 1y ago
  it has already been facing some issues in the past couple weeks. e.g. the "front" page was broken randomly for past dates (like https://news.ycombinator.com/front?day=2025-05-30 https://news.ycombinator.com/front?day=2025-05-30 or https://news.ycombinator.com/front?p=2 https://news.ycombinator.com/front?p=2) as per their api documentation [1], it might be linked to firebase, which might explain this? [1] https://github.com/HackerNews/API https://github.com/HackerNews/API
  - detaro 1y ago
    the API has afaik nothing to do with HN itself, its a read-only copy for API users, not what backs the actual site.

thanhhaimai 1y ago

The status page is green, but there are outages reported: https://downdetector.com/status/google-cloud/ https://downdetector.com/status/google-cloud/

deleted 1y ago
[deleted]
ransom1538 1y ago
Why can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing. We are truly in gods hands. $ prod Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists
- jeanlucas 1y ago
  Because there are contracts related to uptime :)
  - rustc 1y ago
    Does any service even say they're "down" anymore? All I see is "elevated error rates".
    - colechristensen 1y ago
      4 to 6 hours after the flames are visible from orbit and management has finally given up on the 37th quick fix you do get that red X But really not until after it's been on CNN a while.
    - deleted 1y ago
      [deleted]
  - rixthefox 1y ago
    Those contracts will be monitoring their service availability on their own. If Google can't be honest you can bet your bottom dollar the companies paying for that SLA are going to hold them accountable if they report the outage properly or not.

matdehaast 1y ago

Not just GCP, most of Googles services are out of action

milesward 1y ago
I'm on a meet, in cal, editing a dozen docs, in GCP, pushing commits and launching containers; it's not clear yet what exactly is going on but it's certainly intermittent and sparse, at least so far
- parpfish 1y ago
  stop it. you're overloading their system by doing three things at once. let the rest of us have a turn.

sourthyme 1y ago

Maybe cloudflare?

nickporter 1y ago
having issues with cloudflare as well
aetherson 1y ago
Cloudflare status page reports an issue: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/

aetherson 1y ago

We're experiencing intermittent slowness and timeouts on our GCP everything.

dorkitude 1y ago

Same here. Even the page to submit support requests is down. Cloud console does nothing. They should host their support services on AWS and vice-versa.

milesward 1y ago
I just logged into several of my GCP accts, everything popped up, multiple home regions.. I wonder what % of folks are feeling this right now.

aetherson 1y ago

Does anyone know if it's region-specific? We're experiencing it and are in us-west-1.

data-ottawa 1y ago
Us-central-1 as well
deleted 1y ago
[deleted]
vbb 1y ago
europe (netherlands) region as well
izolate 1y ago
us-east-1 too
johannes5117 1y ago
Frankfurt seems to be down as well
mccoyc 1y ago
Can confirm us-east1 (and possibly us-south1) are having VPC host reachability problems.
NeT8235 1y ago
south korea as well
tridao 1y ago
it's due to IAM and global

ea016 1y ago

Google Cloud Storage seems to be down or very slow

digest 1y ago

love how their status page is green with no issues detected!

pbmango 1y ago
https://www.canva.com/design/DAGqKquGD-c/xtRObgH1r_4RoulPAyscgA/view?utm_content=DAGqKquGD-c&utm_campaign=designshare&utm_medium=link2&utm_source=uniquelinks&utlId=h74dd0f5b14 https://www.canva.com/design/DAGqKquGD-c/xtRObgH1r_4RoulPAys...

hambro 1y ago

@dang could you merge this and https://news.ycombinator.com/item?id=44260669 https://news.ycombinator.com/item?id=44260669?

toomuchtodo 1y ago
No notifications for mentions, have to email the mods at the hn@ email address.
- cwillu 1y ago
  Do we know if email is still working? kidding-but-not-really-because-gmail…
- hambro 1y ago
  I think I was a bit optimistic in the response time from mods. This thread won the popularity contest quite well... Thanks for letting me know about emailing the mods, refreshingly explicit to send email.

niij 1y ago

Experiencing 504s in Google Meet. Google Cloud Console won't load.

ransom1538 1y ago

Yeah their status page is all green nothing to see here (but all production systems are down).

alexcroox 1y ago

Cloudflare KV is also having an outage. I wonder who is reliant on who here.

hackermondev 1y ago
seriously doubt Google Cloud is relying on Cloudflare KV lol
dlewis1788 1y ago
Looks like more than KV is having an issue. Just tried to load dash.cloudflare.com and no bueno.

conroy 1y ago

We're in us-west-1 and seeing issues across Cloud Run, Cloud SQL, Cloud Storage and Compute Engine.

eterm 1y ago

Cloudflare speedtest is down too, I assume because of this?

clairegraham 1y ago
Our site depends on Workers and KV and it's very broken right now. Can't login to the Cloudflare Dashboard either.
voxadam 1y ago
Works for me in Portland on Quantum Fiber.
gfs 1y ago
Appears to be a separate incident: https://news.ycombinator.com/item?id=44261064 https://news.ycombinator.com/item?id=44261064
- reassess_blind 1y ago
  Two big cloud provider outages at the same time? Has to be related surely.

siliconc0w 1y ago

Gemini API isn't working for me :/

oalessandr 1y ago

Having issues with services in cloud run as well

zacharynewton 1y ago

Ahhh, explains why some of my apps are going crazy... Couldn't read a message from my kids pre-school Thankfully we use AWS at work for everything critical

atonse 1y ago

Getting a lot of errors for Claude Sonnet 4 (Cursor) and Gemini Pro. Nooooo I'm going to have to use my brain again and write 100% of my code like a caveman from December 2024.

bicx 1y ago
I was in the middle of testing Cloud Storage file uploads, so I guess this is a good time to go for a walk.
- deleted 1y ago
  [deleted]
- matsemann 1y ago
  A good excuse for adding error handling, which otherwise is often overlooked, heh.
orangebread 1y ago
lmao i refuse to write code by hand anymore too. WHAT IS THIS
sujayakar 1y ago
switch to auto mode and it should still work!
- ashu1461 1y ago
  GPT is working in agent mode, which kind of confirms that claude is hosted on google and GPT probably on MSFT servers / self hosted.
  - kenhwang 1y ago
    If you want a stronger confirmation about Claude being hosted on GCP, this is about as authoritative as it gets: https://www.anthropic.com/news/anthropic-partners-with-google-cloud https://www.anthropic.com/news/anthropic-partners-with-googl...

gigatexal 1y ago

YouTube is also very flakey.

deleted 1y ago

[deleted]

Brystephor 1y ago

some core GCP cloud services are down. might be a good time for GCP dependent people to go for a walk, do some stretches, and check back in a couple hours.

mlb_hn 1y ago

Our GCP is down

milesward 1y ago
What region?
- ashu1461 1y ago
  I think multiple regions are down. asia-south, us-east atleast are impacted.
  - a_void_sky 1y ago
    asia-south is working for me
- deleted 1y ago
  [deleted]

sergiotapia 1y ago

xAI having problems, Supabase down, Discord can't upload images to share in chat. Seems like a major backbone outage.

saltcod 1y ago
We're investigating right now. Looks like a potential issue with Cloudflare.
- atsaloli 1y ago
  https://www.cloudflarestatus.com/incidents/25r9t0vz99rp https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
- faizanrupani 1y ago
  You're right, https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, and claude outage.
  - cyberflame 1y ago
    Cloudflare uses GCP as a provider - it's something more upstream

theflyinghorse 1y ago

identitytoolkit.googleapis.com is 503-ing on us, my whole customer success team is locked out from our platform

madjam002 1y ago

Google Maps not loading, thought it was my 4g, go to see if my connection works by loading Hacker News, GCP Outage XD

evtothedev 1y ago

Yarn package registry also appears to be down.

tom1337 1y ago
npm is, registry.yarnpkg.com is only a CNAME to npm

kfarr 1y ago

Yes Firebase auth is down and affecting many apps, on Discord and Slack groups tons of others are corroborating. A bit disappointing that there is no post on the status page for nearly 30 mins: https://status.firebase.google.com/ https://status.firebase.google.com/

kentlyons 1y ago
It just updated. Maybe affected by their own outage!
- ashu1461 1y ago
  Just proves how shady the status page and sla stuff is
  - dgellow 1y ago
    or how difficult it actually is to do that type of thing at scale
  - rco8786 1y ago
    Google is 10 minutes late updating their status page. "So shady" It's really, really hard to make a status page realtime.
    - ashu1461 1y ago
      What makes you think it’s hard? We have AI generating songs and writing code, but setting up basic health checks is too much?
      - jug 1y ago
        An AI generated status page would be the epitome of 2025.
      - urbandw311er 1y ago
        What makes you think it’s easy?

_kush 1y ago

Supabase is also down

vpuna 1y ago
Yes my project on Supabase is down as well.

andrewmcwatters 1y ago

Ah darn it. My Spotify DJ just stopped working.

deleted 1y ago

[deleted]

capital_guy 1y ago

BigQuery is completely dead

b0a04gl 1y ago

[dead]

TN1ck 1y ago

Cloudbuild completely down for us. Getting "Visibility check was unavailable" errors.

quectophoton 1y ago

Twitch was broken too: https://status.twitch.com/incidents/b79nyp1yhxql https://status.twitch.com/incidents/b79nyp1yhxql EDIT: Updated link to point to the specific incident.

rplnt 1y ago
Is Amazon running Twitch on Google Cloud (at least partially)?
- quectophoton 1y ago
  I don't know, at this point I don't know who uses what. This is maybe unrelated but even BunnyCDN has an incident from a few hours ago (https://status.bunny.net/incidents/6g27lbtp67m4 https://status.bunny.net/incidents/6g27lbtp67m4). Seeing how everything seems to be broken everywhere, I'm very much looking forward to the post-mortem.

rcfox 1y ago

I'm able to login to the GCP dashboard, but it isn't able to find any of my projects.

deleted 1y ago
[deleted]

keizo 1y ago

Yup, intermittent db connection issues and cloud storage problems.

deleted 1y ago

[deleted]

admissionsguy 1y ago

Wish there existed a decentralized network connecting computers around the world

redman25 1y ago
Crazy, they could call it the "internet" or something like that... kind of rolls off the tongue.

tsouth 1y ago

Everyone is down. Cloudflare has problems too. All auth providers broken.

deleted 1y ago

[deleted]

traeregan 1y ago

For us Cloud SQL instances are toast but App Engine Standard instances are still serving requests. Google Cloud console is borked too, mostly just erroring out.

ekojs 1y ago

Super duper frustrating having the status page being green. Why can't Google do this properly?

supportengineer 1y ago
Those responsible have been sacked.
- 18172828286177 1y ago
  Those responsible for sacking the people who have just been sacked, have been sacked.

imzadi 1y ago

Can't reach my nest thermometer, but their status page says it's fine lol

charliemeyer 1y ago
the real concerns in life
andrelaszlo 1y ago
This is pretty crazy :D How did it affect you?
- imzadi 1y ago
  I almost died
  - andrelaszlo 1y ago
    Shock, overheating, hypothermia, or a combination of all three?

tmiku 1y ago

Looks like I'm about to start learning which of my time-killing websites are hosted on GCP - The Ringer is down, and since Spotify owns them and is a major GCP customer, it looks like they've been hit by this. CRAZY that the GCP status page is still green.

ashu1461 1y ago

When you deploy code generated by Gemini :D

whitedurna 1y ago

i think it'll be disaster.

whalesalad 1y ago

Meet is also down for me right now. Cannot attend any video calls.

braunshedd 1y ago

Our GCP workloads are unavailable across several US regions. The GCP console is intermittently unavailable for most pages. Crossing my fingers for a quick resolution.

morgandoane 1y ago

Storage, CloudRun, Firebase...... All down....

dana321 1y ago
Auth, GCP, Windsurf,Augment Code,Udio, the list is endless. Facebook, Reddit and Hacker News is still up, but thats about it

vpuna 1y ago

is supabase on GCP ? My Supabase projects are down.

duckarmada 1y ago
Supabase is on AWS, but this is looking like an upstream Cloudflare issue. https://status.supabase.com/ https://status.supabase.com/

devmor 1y ago

Not just GCP. AWS and Cloudflare too. Did someone screw up BGP again?

remram 1y ago
Source? We didn't see anything wrong with AWS here.

timcavel 1y ago

[dead]

deleted 1y ago

[deleted]

atgmal 1y ago

[dead]

akash8400 1y ago

GKE workloads are also affected.

alkalaine 1y ago

[flagged]

supportengineer 1y ago
Take an early lunch

akash8400 1y ago

[dead]

alkalaine 1y ago

[flagged]

elmenosdramatic 1y ago

[flagged]

elmenosdramatic 1y ago

[flagged]

meltyness 1y ago

Can't upload discord attachments from mobile.

Axsuul 1y ago

Does anyone know if instance-to-instance networking has been affected? My Redis instance has been throwing a lot of connection errors.

markbnj 1y ago
We're not seeing any connectivity issues between pods and vms in our vpc, but your mileage may vary.
- Axsuul 1y ago
  Thanks

18172828286177 1y ago

YouTube was down for me for some time

dbacar 1y ago

kaggle not responding correctly, is it related?

faizanrupani 1y ago

https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, claude outage, firbase outage https://status.firebase.google.com/ https://status.firebase.google.com/

andrelaszlo 1y ago
How would Cloudflare's outage cause a GCP outage? I'm sure it's not entirely impossible, but sounds backwards to me. Sure - a lot of the internet relies on Cloudflare, but I'd be very surprised if GCP had a direct dependency on Cloudflare, for a lot of reasons. Maybe I misunderstood your comment?

paulddraper 1y ago

"No major incidents" as of 11:37 PDT. https://status.cloud.google.com/ https://status.cloud.google.com/ File that in the status pages worth ~0 category.

jschroeder 1y ago

Status page is showing green because GCP admins can't login to change it ;)

tiagod 1y ago

Getting Gateway timeouts on docker hub. Maybe related? I can pull images. Example: https://hub.docker.com/layers/library/eclipse-mosquitto/latest/images/sha256-2909e32eb3728f6e8d67e878dae5ec22c00c5ae30ebca630af119b63c200fae2 https://hub.docker.com/layers/library/eclipse-mosquitto/late...

deleted 1y ago

[deleted]

ZiyadFarhan 1y ago

this aint looking good yall

sleepybrett 1y ago

npm registry happen to be hosted on gcp, because that seems to be down as well.

supportengineer 1y ago

Interesting how I landed here. I was having trouble with Nest. Then I went to Down Detector. I noticed many sites having a simultaneous uptick. Then I came to HN, and found this link at the top of the front page.

geekamongus 1y ago
I usually just go here first.
ryanscio 1y ago
Same here with npm https://status.npmjs.org/incidents/dn5mcp85737y https://status.npmjs.org/incidents/dn5mcp85737y

waythenewsgoes 1y ago

Status pages at cloud providers aren't usually based in reality -- usually requires VP level political games to actually get them changed especially for serious outages.

rvnx 1y ago

It looks like that it is a central service @ Google called Chemist that is down. "Chemist checks the project status, activation status, abuse status, billing status, service status, location restrictions, VPC Service Controls, SuperQuota, and other policies." -> This would totally explain the error messages "visibility check (of the API) failed" and "cannot load policy" and the wide amount of services affected. cf. https://cloud.google.com/service-infrastructure/docs/service-management/reference/rpc/google.api#control https://cloud.google.com/service-infrastructure/docs/service... EDIT: Google says "(Google Cloud) is down due to Identity and Access Management Service Issue"

VWWHFSfQ 1y ago
There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.
- rvnx 1y ago
  Absolutely possible. Though there is something curious: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.". So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.
  - whatevertrevor 1y ago
    Doesn't cloudflare have its own infrastructure, it's wild to me that both these things are down presumably together with this size of a blast radius.
    - cyberpunk 1y ago
      You'd think so wouldn't you? DownDetector also reports azure and oracle cloud, I can't see then also being dependant on GCP... I guess down detector isn't a full source of truth though. https://ocistatus.oraclecloud.com/#/ https://ocistatus.oraclecloud.com/#/ https://azure.status.microsoft/en-gb/status https://azure.status.microsoft/en-gb/status Both green

AlgebraFox 1y ago

[flagged]

chief_jeef 1y ago

Firebase status page has acknowledged it as a "global issue". https://status.firebase.google.com/ https://status.firebase.google.com/ A contact in google mentioned to me that some bad update to Google Cloud Storage service has caused some cascading issues affecting multiple GCP services.

0xffany 1y ago

Everything appears to be down as of 18:43 UTC... https://downdetector.com/ https://downdetector.com/

AlienRobot 1y ago
Wait, it's all Google?
- plateng000 1y ago
  "always has been"
- deepsun 1y ago
  Google was the first to report probably.
- bananapub 1y ago
  all cloud
patapong 1y ago
Perhaps their detection logic is running on Google cloud /s
- throitallaway 1y ago
  I believe Downdetector displays user reports.
  - brentm 1y ago
    Yea I am pretty sure that if you're checking if a service is down your essentially casting a vote that indicates that service is down.
    - lysace 1y ago
      Kind of a missed opportunity for Ookla - who's running both downdetector.com and speedtest.net. The have software running in most ISPs around the world: https://help.speedtest.net/hc/en-us/articles/360039164793-How-can-I-host-a-Speedtest-server https://help.speedtest.net/hc/en-us/articles/360039164793-Ho... (OTOH, it's not always trivial to define/detect an outage.)

deadbabe 1y ago

If LLMs are down work grinds to a halt until they return. Just the new era now.

deleted 1y ago

[deleted]

kachapopopow 1y ago

I just realized that the reason the status isn't updated is cause they can't access it lol.

paulddraper 1y ago
Don't host status pages (or their dependencies) on your own infra lol. Seems obvious.
- deathanatos 1y ago
  It should be obvious because both AWS and Azure have done this in the past and shown what a bad idea it is…
Axsuul 1y ago
How do you know that?

fidotron 1y ago

It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2eo4R29 https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e... "Firebase Data Connect unavailable due to a known Google Cloud global outage" While the Google Cloud status page https://status.cloud.google.com/ https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it. Edit to add: within 10 minutes of this post Google updated their status page. More curiously the Firebase page I linked to has been edited to remove mention of Google Cloud in the status and now says "Firebase Data Connect is currently experiencing a service disruption. Please check back for status. ".

cyberflame 1y ago
Services are recovering in some locations it seems - Discord is healing
alexcroox 1y ago
CF too: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
kjuulh 1y ago
Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.
- sss111 1y ago
  I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!
  - paulddraper 1y ago
    Most status pages are manual. At least some of the information has to be. The weird part is that it took them almost an full hour to update it.

deleted 1y ago

[deleted]

paxys 1y ago

Kinda funny that the top post on HN titled "GCP Outage" links to the Google Cloud status page which shows...no outage.

deleted 1y ago

[deleted]

kgwxd 1y ago

Sorry, after decades of being hard wired, I just installed a PCIe Wifi6 card on my desktop. Internet took a dive the second I got it connected. Must have done something wrong.

happy-camper 1y ago

Text messaging on Android is broken

0xCAP 1y ago

> No major incidents … Proceeds to show worldwide degraded service level alerts.

jimt1234 1y ago
Yep. Self-reporting status pages are pretty near worthless. At my former large company (not FAANG), we weren't allowed to update the status page until we got VP approval, which also required approval from both PR and Legal. It would take a lot more time and effort to get those approvals than to just fix the problem and move on.
- iFred 1y ago
  SLA contracts, clawbacks, and performance obligations make these pages a bit of a minefield for CSPs. When I was at a top-tier CSP, we had the status page that was public, one that was for a trusted tier of customers, one built for a customer-by-customer basis, and one for internal engineering.
  - genewitch 1y ago
    When i worked at a top tier speakeasy, we had a book up front for the man, a book in the back for the boss, a book for the trusted accountants...

happy-camper 1y ago

Text messaging for android is broken as well

ipsum2 1y ago

Cloudflare is down too. From https://www.cloudflarestatus.com https://www.cloudflarestatus.com: Update - We are seeing a number of services suffer intermittent failures. We are continuing to investigate this and we will update this list as we assess the impact on a per-service level. Impacted services: Access WARP Durable Objects (SQLite backed Durable Objects only) Workers KV Realtime Workers AI Stream Parts of the Cloudflare dashboard Jun 12, 2025 - 18:48 UTC Edit: https://news.ycombinator.com/item?id=44261064 https://news.ycombinator.com/item?id=44261064

paulddraper 1y ago
Broken link? EDIT: Weird, definitely was just empty
- ipsum2 1y ago
  Should work, but its also on the front page.
0xy 1y ago
Seems like a major wtf if Cloudflare is using GCP as a key dependency.
- a2128 1y ago
  Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it
  - lysace 1y ago
    That's what IRC is for. (Its Finnish inventor is incidentally working for Google in Stockholm, as per https://en.wikipedia.org/wiki/Jarkko_Oikarinen https://en.wikipedia.org/wiki/Jarkko_Oikarinen)
  - IX-103 1y ago

0xffany 1y ago

Does anyone know of a good dashboard to check for such BGP routing anomalies as (apparently) this one? I am currently digging around https://radar.cloudflare.com/routing https://radar.cloudflare.com/routing but it doesn't show which routes were actually leaked. I would love if anyone has any good tool recommendations!

mrngm 1y ago
My default go-to: https://bgp.tools/ https://bgp.tools/ Why would you think this outage is (internet) BGP related?
- whalesalad 1y ago
  Cloudflare runs all their own bare metal servers. Seems odd that they would be impacted by Google cloud. Same can be said for all the other issues on downdetector. This points to a broad issue at the core internet which could certainly be related to BGP.
  - ilkkao 1y ago
    Cloudflare is now saying: "Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency." I really hope CF explains this apparent Google dependecy in detail in their post mortem.
    - whalesalad 1y ago
      Imagine it's just a google spanner wrapper lmao
SparkyMcUnicorn 1y ago
I don't know if I've seen CF Radar before. That's pretty cool! Here are some others, although some seem to be experiencing issues due to the current outage I can only presume. - https://atlas.ripe.net/probes/public https://atlas.ripe.net/probes/public - https://www.ihr.live/en/global-report https://www.ihr.live/en/global-report - https://www.ihr.live/en/network https://www.ihr.live/en/network - https://bgp.he.net/ https://bgp.he.net/ - https://ioda.inetintel.cc.gatech.edu/dashboard/asn https://ioda.inetintel.cc.gatech.edu/dashboard/asn

stepri 1y ago

They have updated the status page finally https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4 https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

acureau 1y ago

Well this explains the issues I've been having with Spotify through the last hour.

artooro 1y ago

What's crazy is that RCS messaging is down as a result of this outage. It shows how poorly the technology or infrastructure was designed.

dcchambers 1y ago
Oh my god is that why my RCS chats were failing earlier?!?!
wbl 1y ago
That explains why I couldn't get the photo of my parents dog today.
foota 1y ago
Isn't RCS basically just instant messaging? I don't know why it's surprising that it would be down.
- roywiggins 1y ago
  I'm not sure any single company could have an outage that would take out SMS globally, but RCS is presumably more centralized.
  - watusername 1y ago
    It used to be kind of distributed, but Google has been strong arming carriers to use their hosted Jibe service through a combination of proprietary extensions (e.g., E2E which is finally standard) and bypassing carrier control (if the carrier didn't provision RCS, Google Messages would use their own service iMessage-style). From the end user's perspective, if the carrier didn't use Jibe RCS, it simply wouldn't work well.
  - whynotminot 1y ago
    People liked to be utterly pissed at Apple for not supporting RCS. But there were reasons
  - toast0 1y ago

compscidr 1y ago

well this explains so much lol

ilovebabyyoda 1y ago

It looks like more than GCP: outages reported across the board including aws https://downdetector.com/ https://downdetector.com/

koliber 1y ago
About the only thing not down is down detector.
- tonyhart7 1y ago
  god send omg, imagine down detector is down lmao anyone know what tech stack they use and where they host

ekojs 1y ago

https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S... > Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue IAM issue huh. The post-mortem should be interesting at least.

yard2010 1y ago
Ha. With all this soviet style euphemism I rather read the onion instead.
- bananapub 1y ago
  It’s not a euphemism - every outage, including the 99.9% that don’t end up on HN gets a postmortem document written about it, which is almost always a fascinating discussion of the technical, cultural and organisational situation that led to an unexpected bad thing happening. Even a few years ago senior management knew to stay the fuck out except for asking for more info.

deleted 1y ago

[deleted]

deleted 1y ago

[deleted]

xan_ps007 1y ago

Where are the AI agents?

dionys 1y ago
Poor agents, finally taking a break
- kyleee 1y ago
  The AI is over employed

Jayakumark 1y ago

Someone must have checked in AI Generated code :-)

cyrux004 1y ago

GPay which is a widely used payment service in India is down as well

edm0nd 1y ago
India is having a really bad day today

unsupp0rted 1y ago

The last few times this happened I wouldn't have thought "So this is the day AI takes over". But this time...

agawish 1y ago

GCP status page now reflect the issues, looks like Google Cloud Dataproc, Google Cloud Storage and Identity & Access Management

dineshsingh1 1y ago

https://downdetector.in/ https://downdetector.in/

SenpaiHurricane 1y ago

Now my api can not connect to PostreSQL... sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3

DonHopkins 1y ago

Damn you Bart Simpson! https://en.wikipedia.org/wiki/Bart_Gets_Famous https://en.wikipedia.org/wiki/Bart_Gets_Famous

dinvlad 1y ago

#HugOps

devmor 1y ago

Incident report published: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

james_m1231 1y ago

My firebase hosting and firestore db are back online, but GCP console and Google SQL instances are still having serious issues as of 7:00pm UTC.

phoenix98 1y ago

when its going to be fixed i am seeing now more and more services are down?

chupamela 1y ago

internal systems at google are currently broken.

phoenix__1998 1y ago

when its going to be fixed i am seeing now more and more services are down?

dinvlad 1y ago

Surprised no one else mentioned "it's always DNS" yet :-)

phoenix__1998 1y ago

when its going to be fixed , i am seeing now more and more services getting outage started with IAM ?

kjuulh 1y ago

I wonder what the damage ($) for having a good portion of the internet down for an hour or two ;)

throwcarsales 1y ago

My friends and I are even having trouble getting Rcs text messages to send.

CyrMeta 1y ago

if everything down at the same time - No one is mentioning an attack on us cloud services ? ( China or Russia ) Maybe ?

cyberflame 1y ago

They've now added this as a major incident - before it just was listed under overview

dpedu 1y ago

reCAPTCHA affected? I couldn't log into my local utilities website due to a reCAPTCHA error. Downdetector agrees, but I interpret that site as dubious.

marifjeren 1y ago
Yeah recaptcha is down intermittently

ashwinsundar 1y ago

Interesting that all Digital Ocean services are fine...

baq 1y ago

One of these days in which the young engineers learn the concept of 'counterparty risk'.

DonHopkins 1y ago

What is this Touchable Grass stuff I keep hearing of?

bsamson05 1y ago

So frustrating, but here's a link to track status of this outage: https://status.anthropic.com/incidents/kn7mvrgb0c8m https://status.anthropic.com/incidents/kn7mvrgb0c8m

CrimsonCape 1y ago

I'm having trouble getting any Street View imagery. Can anyone else confirm?

NameError 1y ago
Yep, street view is not working at all for me

estee_ecstacy 1y ago

Seems recovering now

estees_ecstacy 1y ago

Seems recovering now

estees_ecstacy 1y ago

seems recovering

jamesrwhite 1y ago

Seems like a wider issue at Google than just GCP, the Sheets and Chat APIs are also returning similar "Visibility check was unavailable" errors.

yunwal 1y ago
Presumably many Google products run on GCP

throwaway7783 1y ago

Even though BigQuery is not listed in affected services, we see errors connecting to it

tecleandor 1y ago
It's listed by regions :(

leoh 1y ago

If Google Chat is down per https://www.google.com/appsstatus/dashboard/ https://www.google.com/appsstatus/dashboard/, the ability for Google engineers to communicate among themselves impaired, despite SREs having IRC as a backup.

iamdelirium 1y ago
Google Chat wasn't down for me throughout the entire incident.
miohtama 1y ago
Someone actually uses Google Chat...?
- 00deadbeef 1y ago
  Google has a chat product?
- asadm 1y ago
  it's the best
  - ZiiS 1y ago
    Well given how many they have decommissioned...
  - clhodapp 1y ago
    Oh no, that's how you know it's nearing the point of being reaped and thrown in the graveyard!
    - IX-103 1y ago
      Don't worry, they're not following the "deprecate and cancel" playbook for that. They seem to be using the "copy a competitor poorly" one. The few features I liked about it, that distinguished it from Slack, disappeared in the latest update.
    - 1y ago

jim180 1y ago

Claude Code is down :( too lazy to do manual conversion from Cocoapods dependency to SwiftPM

rectang 1y ago

Haha, I don't ordinarily spend a lot of time in the Google Cloud Console but just now I was debugging a squirrely OAuth issue with reCAPTCHA failing to refresh several days running. I'm getting this weird page error, and I think, "Is this an issue with my organization? [futz futz futz] Hey wait is GCP actually down?" And it turns out to be the top discussion on HN. XD

dgellow 1y ago

Well, good luck to all googlers dealing with this, that's not fun :(

enahs-sf 1y ago

Would be comedy if one of the progenitors of this took Sundar’s buyout offer yesterday and let the world burn today.

AIorNot 1y ago

sheesh so many side-affected issues accross all systems, maybe big tech companies like google shouldn't have laid off all those engineers.. https://www.google.com/appsstatus/dashboard/incidents/Eab7zGLWSaUJyh9c9R9L https://www.google.com/appsstatus/dashboard/incidents/Eab7zG... but no tech bros, just keep following your ketamine addled edgelord when he did this with twitter..

dang 1y ago

Related ongoing thread: Ask HN: Is Firebase Down? - https://news.ycombinator.com/item?id=44260669 https://news.ycombinator.com/item?id=44260669

cyberflame 1y ago

Everything except us-central1 is back up - it's recovering now though

admissionsguy 1y ago

Google denies the outage. https://x.com/Google/status/1933246051512644069 https://x.com/Google/status/1933246051512644069

baggy_trough 1y ago
"clearing cache and cookies"? what is this, 1997?
- bosmanos 1y ago
  lol
- andrelaszlo 1y ago
  First, check that nobody else in your family is making a call on the phone line that your modem is connected to, then make sure to disable your Internet Explorer add-ons before trying again.
  - gred 1y ago
    https://x.com/ProductHunt/status/1626586036402003970 https://x.com/ProductHunt/status/1626586036402003970
ElijahLynn 1y ago
for those who boycott X: https://nitter.net/Google/status/1933246051512644069 https://nitter.net/Google/status/1933246051512644069

desktopninja 1y ago

Borg and K8s were fighting for resources, so Gemini decided to take out DNS. Now a sysAdmin has to step in. * just trying to add a little humour. pretty stressfull outage. grarr!!

lawrenceyan 1y ago

Solana is up ¯\_(ツ)_/¯

pier25 1y ago

"All locations except us-central1 have fully recovered. us-central1 is mostly recovered. We do not have an ETA for full recovery in us-central1."

kubectl_h 1y ago
An hour later and everything is a mess in central-1. They seemed to jump the gun on that one. Doesn't matter if some dinky service like "AutoML Vision" is working, if GCS isn't, then they shouldn't post an optimistic message.

cyberflame 1y ago

Root cause has been identified and it's being resolved/monitored now

_2d30 1y ago

This is at least why Claude is dead: https://status.anthropic.com/incidents/kn7mvrgb0c8m https://status.anthropic.com/incidents/kn7mvrgb0c8m Also spotify isn't working for me so I assume that's also related. These are my most important productivity resources! Sad!

estee_ecstacy 1y ago

sentry is down https://status.sentry.io/ https://status.sentry.io/

johnnyApplePRNG 1y ago

This appears to be continuing to cascade over an hour later... wow... more and more services mentioned as completely down on the outage page. Kind of nice to not be glued to AI chat prompts for a while to be honest.

sigmaball 1y ago

Guess they used Jules to code their services :)

alexcroox 1y ago

2 hour outage at this point

plerpler 1y ago

GCP Artifact registry still down... Not accepting image push and showing 500 status code

augbog 1y ago

Cloudflare Outage also just updated > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information

deleted 1y ago
[deleted]
togume 1y ago
Is GCP the third party?

itdependsnet 1y ago

Any chance this is the root being that so many different services are effected? https://github.com/kubernetes/kops/issues/17433 https://github.com/kubernetes/kops/issues/17433

yunwal 1y ago
I doubt gcloud would be affected by an aws-specific cni. Unless maybe enough AWS users have a GCP backup environment that they flipped on all at once, but it seems unlikely
- itdependsnet 1y ago
  good point. I took that as simply the example that they had in front of them but a generic issue.
jamie0 1y ago
https://cloud.google.com/kubernetes-engine/docs/release-notes-rapid#June_12_2025 https://cloud.google.com/kubernetes-engine/docs/release-note... google did release an update to gcp k8s today, seemingly shortly before the outage

pikdum 1y ago

Was just about to do a demo, but Google Meet was down. Tried to use Jitsi as a fallback, but couldn't log in because Firebase was down too. Ended up using a Slack Huddle, lol.

zeke 1y ago

mapbox maps seemed to be down for a few minutes about an hour ago. I wonder if it is related.

ferchysmn 1y ago

Hola

jwatte 1y ago

Let's say a typical base service (network attached RAM or whatever) has 99.99% reliability. If you have a dependency on 100 of those, you're suddenly closer to 99% reliability. So you switch to higher-level dependencies, and only have 10 dependencies, for a 99.9% reliability. But! It turns out, those dependencies each have dependencies, so they're really already more like 99.9% at best, and you're back at 99% reliability. "good enough" is, indeed, just good enough to make it not worthwhile to rip out all the upstreams and roll your own everything from scratch, because the cost of the occasional outages is much lower than the cost of reinventing every single wheel, nut, bolt, axle, bearing, and grease formulation.

geocrasher 1y ago

https://soundcloud.com/ryan-flowers-916961339/the-internet-is-down https://soundcloud.com/ryan-flowers-916961339/the-internet-i...

max4c 1y ago

[flagged]

quyleanh 1y ago

Look like affect to Cloudflare as well [1] Update - Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. Jun 12, 2025 - 19:57 UTC 1: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/

asim 1y ago

Just our bi-yearly reminder of our over reliance on cloud providers for literally everything. Can't say there's an answer beyond trying to build more independent tech but we know how that goes.

ocdtrekkie 1y ago
Hilariously, I did not know about any outages today during the workday because we discourage cloud service usage and nobody complained about anything breaking. :)
ManBeardPc 1y ago
Yet migration to the cloud continues, driven by people arguing that doing it yourself is too complicated and expensive. Let’s see how long until one outage takes down the global economy for multiple days or weeks.
- JohnMakin 1y ago
  "The cloud" isn't the problem nor is migration to it. The problem is single points of failure in a vast, super-connected, global network. To demonstrate my point, say someone like cloudflare opted to get "off cloud" and run their own datacenters. Half the web would still go down if they had some issue in their datacenters. If anything, the economy of scale and resiliency of a huge cloud network is far beyond what any single operator can ensure for their own service. If this wasn't the case, cloud services couldn't be as profitable as they are. It isn't the panacea people seem to think it is. One critical service going down, regardless of whether it's cloud hosted or not, has ripple effects in the broader network.

LZ_Khan 1y ago

Is this the new Y2k?

riknos314 1y ago

Actual incident link posted: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

makk 1y ago

And THAT, Smithers, is why we wear hardhats on the job.

kodisha 1y ago

> Waiting for downdetector.com to respond...

ManBeardPc 1y ago

The cloud enables you to scale. It allows us to distribute systems across multiple regions and data centers. Seems that this is true for outages as well. The PHP application I wrote as a student running on a single self-hosted server had a higher uptime than any of the cloud providers or redundant system I have seen so far. If you don’t need the cloud for scalability, do it yourself and save yourself the trouble and money. Most companies would be better off investing into some IT staff instead of giving away their systems in the hands of some proprietary and insanely complex cloud environment. You are becoming dependent on someone you don’t know, have no control over and can’t talk with directly. Also the single point of failure is just shifting: from your system to whatever system is managing the cloud. Guess one advantage is that you can shift the blame to someone else…

herpderperator 1y ago

When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?

slt2021 1y ago
so much for System Design interview and bs gatekeeping...
jlhawn 1y ago
The outage was global. For my team specifically, a global Identity and Access Management outage meant that our internal service accounts could not refresh their short-lived access tokens and so different parts of our infrastructure began to fail over the course of an hour or so, regardless of what region or zone they were in. Services were up, but they could not access critical GCP services because of auth-related issues which resulted in internal service errors for us. To give an example, our web servers connect to our GCP CloudSQL database via a Cloud SQL Auth Proxy (there's also a connection pooler in between but that also stayed up). The connection to the proxy was always available, but the proxy wasn't able to renew auth tokens it uses to tunnel to the database, regardless of where the webserver or database happened to be located. To mitigate this in the future we're planning to stop using the auth proxy and connect directly via mutual TLS but now it means we have to manage TLS certificates.

marcinignac 1y ago

Can't wait to see how charts are going to look like here on the project we have developed for Maintel https://variable.io/maintel-digital-landscape/ https://variable.io/maintel-digital-landscape/. It shows availability across multiple services as a landscape. Expecting to see a lot of spikes tomorrow.

biglyburrito 1y ago

I wonder how many SLAs Google blew out today with this outage.

michaelawatkins 1y ago

[dead]

knuppar 1y ago

Spotify was not loading, thought my 5G was bad, used YouTube Music instead without issues. Hmmm...

curtisszmania 1y ago

[dead]

anonfordays 1y ago

[flagged]