8 ms·
Sentry.io outage
- orose_ 5y agoIt's somewhat ironic that a monitoring service has shit the bed. For some reason at work we're using Webpack, and, for some reason we are also using Sentry Webpack plugin that is causing our whole CI to fail. This is fine.
- bovermyer 5y agoMaybe it's just me, but I feel like monitoring software failures should not cause application failures.
- rtpg 5y agoIt’s likely that the failure is around Sentry’s “signal a release and upload your source code out of band to us” feature. In theory things shouldn’t break but if your CI process involves signaling a third party, it can fail from that (hopefully you can temporarily disable it or something of the sort)
- the_gipsy 5y agoYes the CI should fail, or else you would be deploying something that can't match your errors to your source code (via the sourcemaps which is what presumably gets uploaded to sentry on the CI run).
- luismedel 5y agoTake in advance that I don't know Sentry, nor webpack et al but, in my mind, I'd like to have an opportunity to continue my build process with a big, red warning. I prefer a 98% accurate error reporting better than a 100% one I can't push to production. Or maybe I'm not understanding well the value Sentry offers, of course.
- the_gipsy 5y agoIt can be desired to continue on failure, the tradeoff is that there is then a release out there with lacking error-sourcecode mapping. I think if your release cycle is longer, then you will prefer not to deploy without sourcemaps.
- np_tedious 5y agoI kind of agree. Their source map thing to get better traces is a nicety, but few would find it release blocking to forgo it Edit due to other comment: this assumes CI/CD. If you were deploying only weekly then this might be good reason to delay
- CSDude 5y agoThis is a choice, usually hidden until you stumble upon it. Is Sentry being available, (actually, the ability to upload sourcemaps) that important for you to fail a CI or you can just fire up a warning somewhere?
- nickincardone 5y agoI just dealt with this in my laravel app and it was preventing the application from working. I was able to resolve by redeploying app with "SENTRY_LARAVEL_DSN=null" in the .env file
- ruuds 5y agoI'm curious if using Relay (https://docs.sentry.io/product/relay/ https://docs.sentry.io/product/relay/) would prevent these problems?
- nicoburns 5y agoObviously it's never good when a service has an outage, but it looks like Sentry's status page reports the actual status which makes a nice change.
- that_guy_iain 5y agoPlus, it's literally the first time I've seen them having an outage over the years so it seems like they're generally doing well on staying up.
- jrockway 5y agoI think this was their Big One: https://blog.sentry.io/2015/07/23/transaction-id-wraparound-in-postgres https://blog.sentry.io/2015/07/23/transaction-id-wraparound-...
- numbsafari 5y ago... from 7 years ago. They've been pretty solid. I've been really happy with Sentry. No muss, no fuss as a vendor. I wish they would buy/bundle a status monitoring tool.
- krono 5y agoIf even their own operational telemetry were misrepresented, one might have started wondering about the accuracy of what they're reporting to their customers. So, I agree, but it's not like they really had a choice :)
- s_dev 5y agoIt's well known that the AWS status page doesn't reflect the actual uptime. Anytime an massive outage is repored here on HN the AWS status page shows all green. Anytime this debate comes up there are two camps of thought -- the AWS camp where "measuring service degradation" is one of the most insanely complex problems that we lesser devs will never understand -- and the other camp is StatusGator a cheap service that seems to be easily able to tell you when one of your services you use is having trouble - including AWS.
- dijit 5y agoSentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.
- Klathmon 5y agoHas anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?
- carrja99 5y agoWe run the on-prem version, but looking to transition to SaaS soon. While one can point at the current outage and say "good thing we run it ourselves!" it is a chore to keep updated and running smoothly. That said, it runs untouched for long stretches of time no problem.
- franch 5y agoI'm using it for a small project and it works fine. Self hosting is documented here: https://develop.sentry.dev/self-hosted/ https://develop.sentry.dev/self-hosted/ Upgrading is a little bit tricky: in my (limited) experience the upgrading procedure for the self-hosted version sometimes fails and requires to take the application offline to perform the upgrade (but it is very possible that I may be doing something wrong). It is really an invaluable tool.
- scblzn 5y agoI do, it works great with their docker-compose install - processing a few billion events per day here. Installation and subsequent minor/major upgrades without any flaw so far, plus they have extensive documentation for the self hosted version which is to note.
- MrHamdulay 5y agoThe SaaS offering came later. I used to run it on-prem a few years ago before the SaaS showed up and it was very low maintenance. More of a fire and forget situation once it was set up.
- fbrncci 5y agoWhy do these outages come at just the right time, when you are showing friends or colleagues sentry; in relation to preventing outages.
- mekster 5y agoBecause someone is doing what you're doing at every moment?
- letitbeirie 5y agoA few years ago I was showing someone how many calls a request to Spotify makes and I noticed they were getting 429s from Sentry. An amusing little insight into what is surely a thorn in some Spotify engineer's side.
- golergka 5y agoWait, is it an honest status page? Not only red, but with timely updates at the top, too?
- matxd 5y agoLooks like Rollbar (their competitor) is also struggling with some problems right now. https://status.rollbar.com/ https://status.rollbar.com/
- jhiesey 5y agoThe domain sentry.io resolves (for me at least) to a Google Cloud ip address in the us-central1 region, which is having an outage as well: https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCMqV https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCM...
- jvmiert 5y agoSeems you are right: https://twitter.com/mitsuhiko/status/1522550576734691328 https://twitter.com/mitsuhiko/status/1522550576734691328
- candiddevmike 5y agoI am seeing some GKE control plane weirdness this morning too. API server endpoints going unreachable intermittently.
- nik736 5y agoAre they using a single zone though? Only a single zone seems to be down.
- dijit 5y agoI think it can be unfair to characterise single zone failures as being an failure to adequately deploy or architect. There's many opportunities for failure even if only a single zone goes away; most (if not nearly all) database solutions elect leaders for example, and "brown-outs" (as in, not total failures) can lead to the leader maintaining leadership status, or at least messing with quorum. other situations can exist where the migration out of a zone leads to hardware becoming unavailable for consumption for other people, after all, the cloud is not magic and if peoples workloads auto shift to the surrounding (unaffected) zones then it will impact peoples ability to do the same migration as all the free hardware could be used up. I can think of dozens of examples honestly where even if you had built everything multi-zonal you could be down due to a single zone; for instance if some unknown subsystem was zonal (like IAM?) or you use regionally available persistent disks and now they suddenly perform extremely bad with writes because they can't sync to the unavailable datacenter. I believe multi-zone is less possible than we would like it to be, there are many cases where you can commit no error but still be completely at the mercy of a single zone going away.
- ngshiheng 5y agoOne of the SaaS with the best free tier out there. Been using their service for years now
- javier2 5y agoI am actually amazed this does not happen more often.
- frogperson 5y agoI wonder if they are on GCP which is also down this morning durning the same time frame.
- emptysea 5y agoYeah they use GCP for most of their stuff. Not sure if their POP ingest servers are on it though.
- deleted 5y ago[deleted]
- raunometsa 5y agoI wake up to discover that my site https://remotehunt.com https://remotehunt.com is super slow. First thing I do is visit HN to see what's up and I instantly see that Sentry is down. I'm using Sentry to monitor logs and it now makes sense. Ok, so I remove Sentry from Laravel's error handler but nothing changes. And it's weird because sometimes it works, sometimes not. I tweak some things on Cloudflare (turning on Under Attack mode etc). I still have this feeling that maybe it's related to Sentry: I'm using it and HN says it's down. So I go and remove the composer package as well, just in case. And it worked. It took like an hour... I didn't use Sentry anyways :) What did I learn: do not depend on external services if you really don't need it.
- philip1209 5y agoSentry's outage documentation implies that some underlying platform is the cause. Perhaps there's a major data center having issues?
- raunometsa 5y agoMaybe yes. It was very weird: even without sending any error messages to Sentry, it was still slow just by having the composer package installed. Some Sentry DNS thing? (https://docs.sentry.io/platforms/php/guides/laravel/#configure https://docs.sentry.io/platforms/php/guides/laravel/#configu...)
- obituary_latte 5y agoAre you using Google cloud? https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCMqV?src=VESPA https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCM...
- hadrien01 5y agoShouldn't Sentry be called asynchronously?
- darknavi 5y agoCan still bog down job queues and cause odd hiccups.
- dodgerdan 5y agoI’ve used them for a number of years, never so much as a blip. Looks like they’re doing a good job managing this outage. They’re also super transparent with their bugs in their change logs. I recommend and love Sentry.
- dawnerd 5y agoAnd they let you self host for free, although it is quite the process.
- zeeg 5y agoService should be nearly restored at this point. We apologize for the significant interrupt to everyones days. As most of you all figured out at this point the issue was with a regional failure in GCP (central, which is our primary zone). We're going to be exploring what we can do to lessen a single zone failure, and more importantly, reduce/remove any impact of the storage services being unavailable. <3
- zeeg 5y agoFor folks reading this - and we'll solve this at a global scale - check your `SENTRY_DSN` setting and if its not using something like `oXXXXX.ingest.sentry.io` you should consider rolling out the updated value (located in your project settings). Our master domain routing layer is a more complicated failure point, and moving to the ingest based domains pushes you to our edge layer which _should_ be extremely reliable as well as much reduced global latency.
- ncmncm 5y agoWhat is this sentry.io thing for anyway? Should I block its JS code?
- aaronbrethorst 5y agoException tracking. I use it to monitor exceptions in my Rails app.
- mekster 5y agoError reporting. I wish you don't.
- bluepnume 5y agoSentry went down? My only question is - how did they find out?
- CSDude 5y ago#HugOps from ex-Opsgenie SRE. I know the pressure you are on as a mission critical software that people depend on you with their alerts, exceptions etc. I left more than 6 months ago and still recovering from the responsibility toll. You have a great history of availability and would like to say congrats.