4 ms·

I just find it comically ironic that the same community infamous for sleuthing and demanding the full root cause of failure from their favourite faang darlings

I just find it comically ironic that the same community infamous for sleuthing and demanding the full root cause of failure from their favourite faang darlings and various digital feudal estates has been so quiet, forgiving, and reflective about the HN outage and its cause. am I missing something? does anyone know what caused the failure?

civilized 4y ago
HN isn't Amazon or Facebook. It's not a for-profit operation relied on by millions for their business. It doesn't have a blog where its engineers document the latest way they squeezed a few percent more performance. It appears to be a shoestring operation run in a frugal fashion. Why anyone would expect people to have the same attitude towards that as Facebook is beyond me.
hoofhearted 4y ago
They have been super transparent about the HN stack and both outages. A disk failed in their collocated server. They migrated to their backup server, and lost a disk in that server in the same manner. They temporarily stood up an AWS instance from a backup, and are working on a permanent fix.
deleted 4y ago
[deleted]
arkitaip 4y ago
> quiet, forgiving, and reflective about the HN outage and its cause. If Cloudflare, AWS or Google goes down, it affects millions of people directly and often financially. If HN goes down, we simply find another way to waste time.
pmoriarty 4y ago
We weren't quiet. There were lots of questions about the cause of the outage... along with detailed answers, and discussions of those answers: [1][2][3] You just missed them. [1] - https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 [2] - https://news.ycombinator.com/item?id=32030400 https://news.ycombinator.com/item?id=32030400 [3] - https://news.ycombinator.com/item?id=32032235 https://news.ycombinator.com/item?id=32032235
- justsomehnguy 4y ago
- mysterydip 4y ago
  Thanks for all the posts! In the future, it might be helpful to use different (compatible) hardware for the backup server so it doesn't have the same issue that caused the main to fail, like how redundant aircraft systems are made by different teams.
armchairhacker 4y ago
here’s the deal: HN is a tiny forum which basically runs on a single machine with a single backup. There are surely only a few people monitoring it. It’s used by about ~1 million people a day, and nobody really “relies” on HN. GitHub, Facebook, Google, AWS etc. are massive clusters of over 1000 servers spread across continents, with multiple failure modes and run by many DevOps. Many people and services rely on them being up. They’re used by billions each day. Yet HN has more uptime than these behemoths, and way more uptime than e.g. Slack. It’s also faster, even though it’s only one server. HN went down for 2 (?) hours. HN also promptly posts why it went down. In this case because both the main and backup server failed. Of course HN is definitely more reliable because it’s only one server. But it’s still impressive and shows that you can reach a large audience with a simple setup and few resources. Maybe if Google, Facebook, AWS, etc. found more straightforward solutions they too could have more uptime.
- pmoriarty 4y ago
  "Yet HN has more uptime than these behemoths, and way more uptime than e.g. Slack." That's because HN does a lot less, and is a lot simpler.
  - water8 4y ago
    I have a plug-in clock with backup battery that has an uptime of always
  - vachina 4y ago
    > and is a lot simpler I think that’s a feature not a bug.
  - bob1029 4y ago
    > That's because HN does a lot less, and is a lot simpler. Complexity is not a virtue. HN has a lot of depth when you look beyond its interface. The abstractions don't always have to exist in code, operations or UI. What % of the Slack feature set is actually used by a majority of its users at least once a month?
- frosted-flakes 4y ago

civilized 4y ago

HN isn't Amazon or Facebook. It's not a for-profit operation relied on by millions for their business. It doesn't have a blog where its engineers document the latest way they squeezed a few percent more performance. It appears to be a shoestring operation run in a frugal fashion. Why anyone would expect people to have the same attitude towards that as Facebook is beyond me.

hoofhearted 4y ago

They have been super transparent about the HN stack and both outages. A disk failed in their collocated server. They migrated to their backup server, and lost a disk in that server in the same manner. They temporarily stood up an AWS instance from a backup, and are working on a permanent fix.

deleted 4y ago

[deleted]

arkitaip 4y ago

> quiet, forgiving, and reflective about the HN outage and its cause. If Cloudflare, AWS or Google goes down, it affects millions of people directly and often financially. If HN goes down, we simply find another way to waste time.

pmoriarty 4y ago

We weren't quiet. There were lots of questions about the cause of the outage... along with detailed answers, and discussions of those answers: [1][2][3] You just missed them. [1] - https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 [2] - https://news.ycombinator.com/item?id=32030400 https://news.ycombinator.com/item?id=32030400 [3] - https://news.ycombinator.com/item?id=32032235 https://news.ycombinator.com/item?id=32032235

justsomehnguy 4y ago
mysterydip 4y ago
Thanks for all the posts! In the future, it might be helpful to use different (compatible) hardware for the backup server so it doesn't have the same issue that caused the main to fail, like how redundant aircraft systems are made by different teams.

armchairhacker 4y ago

here’s the deal: HN is a tiny forum which basically runs on a single machine with a single backup. There are surely only a few people monitoring it. It’s used by about ~1 million people a day, and nobody really “relies” on HN. GitHub, Facebook, Google, AWS etc. are massive clusters of over 1000 servers spread across continents, with multiple failure modes and run by many DevOps. Many people and services rely on them being up. They’re used by billions each day. Yet HN has more uptime than these behemoths, and way more uptime than e.g. Slack. It’s also faster, even though it’s only one server. HN went down for 2 (?) hours. HN also promptly posts why it went down. In this case because both the main and backup server failed. Of course HN is definitely more reliable because it’s only one server. But it’s still impressive and shows that you can reach a large audience with a simple setup and few resources. Maybe if Google, Facebook, AWS, etc. found more straightforward solutions they too could have more uptime.

pmoriarty 4y ago
"Yet HN has more uptime than these behemoths, and way more uptime than e.g. Slack." That's because HN does a lot less, and is a lot simpler.
- water8 4y ago
  I have a plug-in clock with backup battery that has an uptime of always
- vachina 4y ago
  > and is a lot simpler I think that’s a feature not a bug.
- bob1029 4y ago
  > That's because HN does a lot less, and is a lot simpler. Complexity is not a virtue. HN has a lot of depth when you look beyond its interface. The abstractions don't always have to exist in code, operations or UI. What % of the Slack feature set is actually used by a majority of its users at least once a month?
frosted-flakes 4y ago