6 ms·
The status page is green, but there are outages reported: https://downdetector.com/status/google-cloud/ https://downdetector.com/status/google-cloud/
by thanhhaimai 1y ago
The status page is green, but there are outages reported:
https://downdetector.com/status/google-cloud/ https://downdetector.com/status/google-cloud/
- deleted 1y ago[deleted]
- ransom1538 1y agoWhy can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing. We are truly in gods hands. $ prod Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists
- jeanlucas 1y agoBecause there are contracts related to uptime :)
- rustc 1y agoDoes any service even say they're "down" anymore? All I see is "elevated error rates".
- colechristensen 1y ago4 to 6 hours after the flames are visible from orbit and management has finally given up on the 37th quick fix you do get that red X But really not until after it's been on CNN a while.
- deleted 1y ago[deleted]
- rixthefox 1y agoThose contracts will be monitoring their service availability on their own. If Google can't be honest you can bet your bottom dollar the companies paying for that SLA are going to hold them accountable if they report the outage properly or not.
- datadrivenangel 1y agoThe real point of SLAs is to give you a reason to break contracts. If a vendor doesn't meet their contractual promises, that gives you a lot of room to get out contracts
- 9rx 1y agoThe program that updates the status page is hosted on Google Cloud.
- ashu1461 1y agoSo even then, it should have been able to correctly report the status, it somehow shows that the status page is not automated and any change there needs to go through someone manual.
- 9rx 1y agoA program that updates the status page failing does not imply that the status page is manually edited. It is not like you would generate a status page on every request.
- ashu1461 1y agoHow do we know that the program is failing ? How hard is it for the frontend to detect if the last update to the status page was made a while ago and that itself implies there is an error and should be reported ?
- 9rx 1y agoWe don’t. But why would the frontend have processing logic when all you need is to serve a static HTML document? Even if it did, what would you do with that information? Throw up a screen with: Call us for service information at 1-HAHA-JUST-KIDDING It’s not like it really matters if it’s accurate anyway.
- rapus95 1y agothe services ARE healthy, status page is correct. The backbone which links YOU to the service isn't healthy. Take a look at cloudflare, they are already working on it
- ikiris 1y ago
- oxymoron 1y agoBecause a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding](https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/ https://aws.amazon.com/builders-library/workload-isolation-u...). So sometimes a service is completely down for some customers and fully unaffected for other customers.
- Eduard 1y ago> Because a lot of the time, not everyone is impacted then such pages should report a partial failure. Indeed the GCP outage page lists an orange "One or more regions affected" marker, but all services show the green "Available" marker, which apparently is not true.
- deepsun 1y agoThere's always a partial outage in large systems, some very small percentage. All clouds should report all red then.
- hnuser123456 1y ago"there is a 5% chance your instance is down" is still a partial outage. A green check should only mean everything (about that service) is working for everyone (in that region) as intended. Downdetector reports started spiking over an hour ago but there still isn't a single status that isn't a green checkmark on the status page.
- deleted 1y ago[deleted]
- rozap 1y agoPlease, won't somebody think of the KPIs.
- kingstnap 1y agoBecause they have unrealistic targets so they make up fake uptime numbers. 99.999% would mean not even having an hour of downtime in 10 years. I remember reddit being down for like a whole day or so and they claimed 99.5% in that month.
- wbl 1y agoMa Bell hit that decently often.
- Dylan16807 1y agoRunning a much simpler system with much more independent nodes. It's a lot easier to keep packets flowing than to keep non-self-contained servers serving.
- Uehreka 1y agoIs that even knowable? Like, I know they called it “The Astonishing, Unfailing, Bell System” but if they had an outage somewhere did they actually have an infrastructure of “canary phones” and such to tell in real time? (As in, they’d know even if service was restored in an hour) Not trying to snark, I legit got nerdsniped by this comment.
- wbl 1y agoThey absolutely did. Note that the reliability estimates exclude the last mine because trees falling and the like but they had a lot of self repair, reporting, and management facilities. Engineering and Operations in the Bell System is pretty great for this.
- voytec 1y ago> Why can't companies be honest with being down SLA agreements.
- organsnyder 1y agoAny customer with enough leverage to negotiate meaningful SLA agreements will also have the leverage to insist that uptime is not derived from the absence of incidents on public-facing status pages.
- remram 1y agoService level agreements agreements?
- rapus95 1y agoif half the internet is down, which it apparently is, it's usually not the service in question, but some backbone service like cloudflare. And as internal health monitoring doesn't route to the outside through the backbone to get back in, it won't pick it up. Which is good in some sense, as it means that we can see if it's on the path TO the service or the service itself.
- supportengineer 1y agoNobody gets a promotion, that's why.
- DrBenCarson 1y agoWhichever product person is in charge of the status page should be ashamed How could you possibly trust them with your critical workloads? They don't even tell you whether or not their services work (despite obviously knowing)
- FireBeyond 1y agoYeah, my company of hundreds of people working remotely are having 90%+ failures connecting to Google Meetings - joining a meeting just results in a 504.
- carter-0 1y ago[dead]
- bicx 1y agoAlmost everything on the downdetector home page is listed as having downtime...
- 0xCAP 1y agoAt this point I don’t know if I must assume people are trolling or the entire internet is down.
- roughly 1y agoSo’s Azure? https://downdetector.com/status/windows-azure/ https://downdetector.com/status/windows-azure/ This is where we get to learn about the one common system all of our “distributed cloud” systems rely on, isn’t it?
- deathanatos 1y agoMy gut says all clouds spike when one goes down from people misreporting issues. But I suppose there's always "something something BGP" but that feels less likely.
- ransom1538 1y agoIt's the entire internet. Check oracle cloud, etc etc. The ENTIRE INTERNET.
- nerdsniper 1y agoWhy even have a status page? Someone reported that their org of >100,000 users can't use Google Meet. If corps aren't going to update their status page, might as well just not have one. https://www.google.com/appsstatus/dashboard/ https://www.google.com/appsstatus/dashboard/ https://status.cloud.google.com/index.html https://status.cloud.google.com/index.html Edit: The GCP status page got updated <1 minute after I posted this, showing affected services are Cloud Data Fusion, Cloud Memorystore, Cloud Shell, Cloud Workstations, Google Cloud Bigtable, Google Cloud Console, Google Cloud Dataproc, Google Cloud Storage, Identity and Access Management, Identity Platform, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Vertex AI Search
- supportengineer 1y agoWho gets a promotion from a working status board?
- SOLAR_FIELDS 1y agoThere's no situation where the corporation controls the status page where you can trust the status page to have accurate information. None. The incentives will never be aligned in this regard. It's just too tempting and easy for the corp to control the narrative when they maintain their own status page. The only accurate status pages are provided by third party service checkers.
- the8472 1y ago> The incentives will never be aligned in this regard. Well, yes, incentives, do big customers with wads of cash have an incentive to demand accurate reporting from their suppliers so they can react better rather than trying to identify issues? If there's systematic underreporting, then apparently not. Though in this case they did update their page.
- staplers 1y agoIf there's systematic underreporting, then apparently not. You answered your own question.
- milesward 1y agoIt's updated now, shows the impact to console, dataproc, GCS, IAM and Identity Platform: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4 https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
- jorts 1y agoHere's the incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
- deathanatos 1y agoIt was nearly an hour into our company's internal incident channel on this for GCP to finally declare that yes, in fact, things on fire. … I get that PR-types probably want to massage the message, but going radio dark is not good PR.
- deleted 1y ago[deleted]