4 ms·
> Isn't listening for spikes in complaints about outages a great way to detect them? When there's a major ISP outage, people report problems with all the major
by ceejayoz 3y ago
> Isn't listening for spikes in complaints about outages a great way to detect them?
When there's a major ISP outage, people report problems with all the major sites. When Facebook's down, people report problems with any site that has "Login with Facebook" as an option.
It's almost never actually an outage impacting all of FAANG at once.
- ImPostingOnHN 3y ago> When Facebook's down, people report problems with any site that has "Login with Facebook" as an option. If users log into your site with Facebook, then the login functionality of your site effectively is down when "Login with Facebook" is down. From the user's perspective, your subcontractors, including authentication subcontractors, are a problem for you to deal with and never show them. From your perspective, you could have architected your site in a way that logging in doesn't "go down" when Facebook login is down. If the user chooses "Login with Facebook" over other authentication options available, and they don't want to use other options, educating them with a good error message might help. Or you could remove the Facebook login option, if you (totally reasonably) don't want Facebook's failures to reflect poorly on you.
- ceejayoz 3y ago> If users log into your site with Facebook, then the login functionality of your site effectively is down when "Login with Facebook" is down. There are plenty of sites where "Login with Facebook" is a convenience but hardly the only way to log in. Reddit, for example, has "Login with Google" and "Login with Apple"; it would be highly misleading to claim "Reddit is down" if Google's OAuth flow was having an outage. > educating them with a good error message might help Nothing in the API or OAuth flow would make that doable in an automatic fashion with this outage. It'd have to be something you put up manually as a banner after hearing of the outage. > Or you could remove the Facebook login option, if you (totally reasonably) don't want Facebook's failures to reflect poorly on you. I don't particualrly care; we're talking about why DownDetector isn't necessarily ideal for assessing. It can be a useful signal, in some scenarios, but I've seen plenty of spurious signals come from it.
- ImPostingOnHN 3y ago> Nothing in the API or OAuth flow would make that doable in an automatic fashion with this outage. It'd have to be something you put up manually as a banner after hearing of the outage. That is fair: if I choose to architect my site such that a user-critical feature goes down when a 3rd party service goes down, it behooves me to monitor the 3rd party service and do whatever necessary to properly inform users what's going on. I edited my post unfortunately after you replied, but another option is removing the parts of your site that rely on 3rd parties, if you don't want the failures of those 3rd parties to reflect poorly on you (which they reasonably would). >we're talking about why DownDetector isn't necessarily ideal for assessing. It can be a useful signal, in some scenarios, but I've seen plenty of spurious signals come from it. Indeed, and if a bunch of users say that a feature of your site is down, even if it's a result of a 3rd party failure: chances are, that part of your site is down, and it's partially your fault for relying on a 3rd party for that feature. The users correctly don't care what the root cause is, they expect you to either mitigate it or don't have a feature they rely upon be unreliable.
- ceejayoz 3y agoAll that's fine, but totally misses the point. Take a look at https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/ ; scroll down to the comments. "SSH and Dbconnect stopped on all of my EC2 instances. Anyone else?" "I can't add a payment method" The chart shows a big spike this morning, but there was no AWS outage, nor does Amazon use Facebook login. Again, DownDetector can be a useful "is something unusual happening right now" signal, but it'd be a mistake to take its attribution at face value.
- ImPostingOnHN 3y agoIgnore the comments on DownDetector for a moment and check out that huge spike in reports recently. Clearly something wrong happened with AWS's user experience. That's something AWS needs to resolve, in the eyes of their users. >The chart shows a big spike this morning, but there was no AWS outage Are you sure? If hundreds of users simultaneously reported there was some sort of outage, particularly a huge spike like we saw, chances are there was an outage. >Again, DownDetector can be a useful "is something unusual happening right now" signal Exactly! Specifically, "is something unusual happening right now with my site, in the eyes of my users?" Every site owner should know when that condition is true. What you think about your site "up-ness" isn't as important as what your users think about your site "up-ness". What you attribute your downtime to, isn't as important as what your users attribute your downtime to (you.)
- baby_souffle 3y ago> It's almost never actually an outage impacting all of FAANG at once. Exactly. If you click through down detector when things are _up_ you'll see people still complaining that $site is down. Could be a local power outage or even a flaky connection in their own home. Down Detector is one of many signal sources and should have a "credibly" score associated with it that's proportional to the number of people complaining that something's down.
- whatwhaaaaat 3y ago[flagged]
- ceejayoz 3y ago> do you really think there are masses of people who can’t tell the difference between a single sign on service being down and individual sites being down and reporting it to downdetector? Yes, absolutely. 100%. > Even if there were doesn’t the outage graph give you exactly the information your asking be curated? https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/ Was there an AWS outage this morning? The graph sure looks like it, but there wasn't.
- samtheprogram 3y ago> do you really think there are masses of people who can’t tell the difference between a single sign on service being down and individual sites being down and reporting it to downdetector? Have you never seen The Website Is Down? https://www.youtube.com/watch?v=uRGljemfwUE https://www.youtube.com/watch?v=uRGljemfwUE The answer is: way more people than a software developer might think. Ask anyone in IT, or go to anywhere bugs are reported and read a handful.
- whatwhaaaaat 3y agoyeah that’s my point. No one who arranges icons by penis is taking time to go file a report on down detector.
- jordanthoms 3y agoThat's kinda the point though isn't it? DownDetector is showing an early indication of a major outage in both of your examples. The issue may not be caused by the indicated service, but it's still a useful information source especially when we can correlate reports on there with what we are seeing in our internal monitoring.
- ceejayoz 3y agoA big spike on DownDetector is an indication of something going on. Its attribution of what/who is often incorrect. You'll see "maybe it's more than Big Site X!" comments come up on every HN thread like this citing DownDetector; it's almost never the case, and folks on HN should know better.
- rimunroe 3y ago> When there's a major ISP outage, people report problems with all the major sites. When Facebook's down, people report problems with any site that has "Login with Facebook" as an option. Yes? That's how all top-level reporting is going to work. It's not going to tell you which part of your service is inaccessible. It's just telling you that people can't access it. You obviously have to do additional investigation to figure out why people are having trouble.
- ceejayoz 3y ago> It's not going to tell you which part of your service is inaccessible. Scroll up the thread a bit; https://news.ycombinator.com/item?id=39605354 https://news.ycombinator.com/item?id=39605354 Even here on HN, where people should know better, people take its incorrect attribution as useful info. TikTok isn't down. X isn't down. Google isn't down.
- deleted 3y ago[deleted]
- rimunroe 3y agoI would completely agree that people are bad at interpreting Down Detector-type results, but that doesn't mean it isn't providing a very useful signal.
- jajko 3y agoIndeed I haven't noticed any blip in functionality, but then again I don't ever do FB (or other external service) login. Absolutely no reason to do so, long term drawbacks are too serious to be lazy about this.
- ipaddr 3y agoYou can't login to snapchat, tiktok, youtube with your facebook
- ceejayoz 3y agoYes, those are all just Down Detector’s bog standard false positives. Pull the page up tomorrow and you’ll see the same morning spike there as people wake up.