3 ms·

In a company where I worked, the tool measuring downtime was at the same server, so even if the server was down they still showed 100% up. If the server didnt

by rvba 1y ago

In a company where I worked, the tool measuring downtime was at the same server, so even if the server was down they still showed 100% up. If the server didnt work - the tool too measure didnt work too! Genius

bityard 1y ago
This happened to AWS too. February 28, 2017. S3 went down and took down a good portion of AWS and the Internet in general. For almost the entire time that it was down, the AWS status page showed green because the up/down metrics were hosted on... you guessed it... S3. https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
- subpar 1y ago
  obligatory https://x.com/lintzston/status/791761626890469377 https://x.com/lintzston/status/791761626890469377
  - belter 1y ago
    Published in the same week of October ...9 years ago ...Spooky...
- CaptainOfCoit 1y ago
  Happened a couple of times :) - 2008 - https://news.ycombinator.com/item?id=116445 https://news.ycombinator.com/item?id=116445 - 2010 - https://news.ycombinator.com/item?id=1396191 https://news.ycombinator.com/item?id=1396191 - 2015 - https://news.ycombinator.com/item?id=10033172 https://news.ycombinator.com/item?id=10033172 - 2017 - https://news.ycombinator.com/item?id=13755673 https://news.ycombinator.com/item?id=13755673 (Postmortem: https://news.ycombinator.com/item?id=13775667 https://news.ycombinator.com/item?id=13775667) - 2024 - https://news.ycombinator.com/item?id=41770111 https://news.ycombinator.com/item?id=41770111
  - hinkley 1y ago
    Five times is no longer a couple. You can use stronger words there.
    - bapak 1y ago
      It happened a murder of times.
      - hinkley 1y ago
        Ha! Shall I bookmark this for the eventual wiki page?
  - casey2 1y ago
    https://www.youtube.com/watch?v=HxP4wi4DhA0 https://www.youtube.com/watch?v=HxP4wi4DhA0 Maybe they should start using real software instead of mathematicians' toy langs
- Scoundreller 1y ago
  Have we ever figured out what “red” means? I understand they’ve only ever gone to yellow.
  - kokanee 1y ago
    If it goes red, we aren't alive to see it
    - Cthulhu_ 1y ago
      I'm sure we need to go to Blackwatch Plaid first.
conductr 1y ago
Similar to hosting your support ticketing system with same infra. "What problem? Nobody's complaining"
decimalenough 1y ago
I used to work at a company where the SLA was measured as the percentage of successful requests on the server. If the load balancer (or DNS or anything else network) was dropping everything on the floor, you'd have no 500s and 100% SLA compliance.
bigiain 1y ago
I spent enough time ~15 years back to find an external monitoring service that did not run on AWS and looked like a sustainable business instead of a VC fueled acquisition target - for our belts-n-braces secondary monitoring tool since it's not smart to trust CloudWatch to be able to send notifications when it's AWS's shit that's down. Sadly while I still use that tool a couple of jobs/companies later - I no longer recommend it because it migrated to AWS a few years back. (For now, my out-of-AWS monitoring tool is a bunch of cron jobs running on a collections of various inexpensive vpses and my and other dev's home machines.)
- 6031769 1y ago
  Nagios is still a thing and you can host it wherever you like.
  - bigiain 1y ago
    Interestingly, the reason I originally looked for and started using it was an unapproved "shadow IT" response to an in-house Nagios setup that was configured and managed so badly it had _way_ more downtime than any of the services I'd get shouted about at if customers noticed them down before we did... (No disrespect to Nagios, I'm sure a competently managed installation is capable of being way better than what I had to put up with.)
hinkley 1y ago
I’ve been customer for at least four separate products where this was true. I can’t explain why Saucelabs was the most grating one, but it was. I think it’s because they routinely experienced 100% down for 1% of customers, and we were in that one percent about twice a year. <long string of swears omitted>
AbstractH24 1y ago
If its not on the dashboard, it didn't happen

by rvba 1y ago

bityard 1y ago

This happened to AWS too. February 28, 2017. S3 went down and took down a good portion of AWS and the Internet in general. For almost the entire time that it was down, the AWS status page showed green because the up/down metrics were hosted on... you guessed it... S3. https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/

subpar 1y ago
obligatory https://x.com/lintzston/status/791761626890469377 https://x.com/lintzston/status/791761626890469377
- belter 1y ago
  Published in the same week of October ...9 years ago ...Spooky...
CaptainOfCoit 1y ago
Happened a couple of times :) - 2008 - https://news.ycombinator.com/item?id=116445 https://news.ycombinator.com/item?id=116445 - 2010 - https://news.ycombinator.com/item?id=1396191 https://news.ycombinator.com/item?id=1396191 - 2015 - https://news.ycombinator.com/item?id=10033172 https://news.ycombinator.com/item?id=10033172 - 2017 - https://news.ycombinator.com/item?id=13755673 https://news.ycombinator.com/item?id=13755673 (Postmortem: https://news.ycombinator.com/item?id=13775667 https://news.ycombinator.com/item?id=13775667) - 2024 - https://news.ycombinator.com/item?id=41770111 https://news.ycombinator.com/item?id=41770111
- hinkley 1y ago
  Five times is no longer a couple. You can use stronger words there.
  - bapak 1y ago
    It happened a murder of times.
    - hinkley 1y ago
      Ha! Shall I bookmark this for the eventual wiki page?
- casey2 1y ago
  https://www.youtube.com/watch?v=HxP4wi4DhA0 https://www.youtube.com/watch?v=HxP4wi4DhA0 Maybe they should start using real software instead of mathematicians' toy langs
Scoundreller 1y ago
Have we ever figured out what “red” means? I understand they’ve only ever gone to yellow.
- kokanee 1y ago
  If it goes red, we aren't alive to see it
  - Cthulhu_ 1y ago
    I'm sure we need to go to Blackwatch Plaid first.

conductr 1y ago

Similar to hosting your support ticketing system with same infra. "What problem? Nobody's complaining"

decimalenough 1y ago

I used to work at a company where the SLA was measured as the percentage of successful requests on the server. If the load balancer (or DNS or anything else network) was dropping everything on the floor, you'd have no 500s and 100% SLA compliance.

bigiain 1y ago

I spent enough time ~15 years back to find an external monitoring service that did not run on AWS and looked like a sustainable business instead of a VC fueled acquisition target - for our belts-n-braces secondary monitoring tool since it's not smart to trust CloudWatch to be able to send notifications when it's AWS's shit that's down. Sadly while I still use that tool a couple of jobs/companies later - I no longer recommend it because it migrated to AWS a few years back. (For now, my out-of-AWS monitoring tool is a bunch of cron jobs running on a collections of various inexpensive vpses and my and other dev's home machines.)

6031769 1y ago
Nagios is still a thing and you can host it wherever you like.
- bigiain 1y ago
  Interestingly, the reason I originally looked for and started using it was an unapproved "shadow IT" response to an in-house Nagios setup that was configured and managed so badly it had _way_ more downtime than any of the services I'd get shouted about at if customers noticed them down before we did... (No disrespect to Nagios, I'm sure a competently managed installation is capable of being way better than what I had to put up with.)

hinkley 1y ago

I’ve been customer for at least four separate products where this was true. I can’t explain why Saucelabs was the most grating one, but it was. I think it’s because they routinely experienced 100% down for 1% of customers, and we were in that one percent about twice a year. <long string of swears omitted>

AbstractH24 1y ago

If its not on the dashboard, it didn't happen