9 ms·

HN was down because the failover server also failed: https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/154540942911322

• hunterb123 4y ago
  What was the test to determine the dataloss?
  • Cerium 4y ago
    I came to the same conclusion by observing that there are posts and comments from only eight hours ago.
    • jbverschoor 4y ago
      So that means dataloss.. Probably restored from backup. Good news for people who were banned, or for posts that didn't get enough momentum :) edit: Was restored from backup.. so def. dataloss
      • joshuamorton 4y ago
        > So that means dataloss.. Probably restored from backup. If the server went down at XX:XX, and the backup they restored from is also from XX:XX, there isn't dataloss. If the server was down for 8 hours, the last data being 8 hours old isn't dataloss, it's correct.
      • dang 4y ago
        8 hours of downtime, but not data loss, since there was no data to lose during the downtime. Last post before we went down (2022-07-08 12:46:04 UTC): https://news.ycombinator.com/item?id=32026565 https://news.ycombinator.com/item?id=32026565 First post once we were back up (2022-07-08 20:30:55 UTC): https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 (hey, that's this thread! how'd you do that, tpmx?) So, 7h 45m of downtime. What we don't know is how many posts (or votes, etc.) happened after our last backup, and were therefore lost. The latest vote we have was at 2022-07-08 12:46:05 UTC, which is about the same as the last post. There can't be many lost posts or votes, though, because I checked HN Search (https://hn.algolia.com/ https://hn.algolia.com/) just before we brought HN back up, and their most recent comment and story were behind ours. That means our last backup on the ill-fated server was taken after the last API update (HN Search relies on our API), and the API gets updated every 30 seconds. I'm not saying that's a rock-solid argument, but it suggests that 30 seconds is an upper bound on how much data we lost.
        • deleted 4y ago
          [deleted]
        • sillysaurusx 4y ago
          Curiosity got the better of me. Why was there a 6 ID gap between the last post and first post? The answer seems to be that admins were making posts, which is neat. (There was also one lonely Flexport job ad.) Is your backup system tied to your API? Algolia is a third party service, and streaming the latest HN data to Algolia seems pretty similar to streaming it to a backup system.
          • dang 4y ago
            I posted a bunch of test things and then deleted them.
            • scott_s 4y ago
              I love this answer so much.
              • sillysaurusx 4y ago
                I really wanted to ask “How did you post things if the server was down?” but perhaps some things are better left as mysteries.
                • dang 4y ago
                  The server was up for us before it was up for everybody else.
                • O__________O 4y ago
                  You could see them via HN’s API before they were deleted, nothing interesting; API was back up before the www.
                  • dang 4y ago
  • sillysaurusx 4y ago
    Informal. My last upvote was pretty close to when HN went down, so I expected my karma to go down, but it didn't. Also I remember the "Why we're going with Rails" story on the front page from before it went down.
    • deleted 4y ago
      [deleted]
• dboreham 4y ago
  CP => !A
  • deleted 4y ago
    [deleted]
• pastor_bob 4y ago
  is dang pushing changes and such on his own? sounds like it is run by one guy
  • sillysaurusx 4y ago
    HN will be around a hundred years. I think it's more than just a forum. We've seen lots of people coordinate during disasters, for example. Dan and his team do a good job running it. (I'm not a part of it.) EDIT: My response was based on some edits that are now removed.
    • rat9988 4y ago
      You are overstimating HN way too much.
      • nominusllc 4y ago
        A hundred years, I give it 10 tops.
        • sillysaurusx 4y ago
          It’s already been around since 2007. How many decades does HN need to be around before people realize it’s an institution?
          • grepfru_it 4y ago
            Slashdot has been around since 1997 and people still rave about its moderation system today. However, while I have high hopes for HN, it could very well go the way of digg overnight
            • deleted 4y ago
              [deleted]
            • GekkePrutser 4y ago
• davedunkin 4y ago
  > Double disk failure is improbable but not impossible. It's not even improbable if the disks are the same kind purchased at the same time.
  • spiffytech 4y ago
    Yep: if you buy a pair disks together, there's a fair chance they'll both be from the same manufacturing batch, which correlates with disk defects.
    • dspillett 4y ago
      This is why I try to mismatch manufacturers in RAID arrays. I'm told there is a small performance hit (things run towards the speed of the slowest, separately in terms of latency and throughput) but I doubt the difference is high and I like the reduction in potential failure-during-rebuild rates. Of course I have off-machine and off-site backups as well as RAID, but having to use them to restore a large array would be a greater inconvenience than just being able to restore the array (followed by checksum verifies over the whole lot for paranoia's sake).
    • clintonwoo 4y ago
      This makes total sense but I've never heard of it. Is there any literature or writing about this phenomenon? I guess proper redundancy is having different brands of equipment also in some cases.
      • AceJohnny2 4y ago
        I don't know about literature, but in the world of RAID this is a common warning. Having a RAID5 crash and burn because the backup disk failed during the reconstruction phase after a primary disk failed is a common story.
      • eganist 4y ago
        Not sure about literature, but past anecdotes and HN threads yes. https://news.ycombinator.com/item?id=4989579 https://news.ycombinator.com/item?id=4989579
• deleted 4y ago
  [deleted]
• digitallyfree 4y ago
  By second disk failure do they mean that the disks on both the primary and fallback servers failed? Or do they mean that two disks (of a RAID1 or similar setup) in the fallback server failed? The latter is understandable, the former would be quite a surprise for such a popular site. That means that the machines have no disk redundancy and the server is going down immediately on disk failure. The fallback server would be the only backup.
  • spiffytech 4y ago
    14 hours ago HN failed over to the standby due to a disk failure on the primary. 8 hours ago the standby's disk also failed. Primary failure: https://news.ycombinator.com/item?id=32024036 https://news.ycombinator.com/item?id=32024036 Standby failure: https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/1545409429113229312
  • dang 4y ago
    The disks on both the primary and fallback servers definitely failed. Each was in a RAID setup, but those failed too in both cases.
    • digitallyfree 4y ago
      Ouch! I'm assuming the disks were from the same batch and installed at the same time, but having at least four fail like that is just crazy unlucky.
      • dang 4y ago
        Veteran programmer and HN user kabdib has proposed a striking theory that could explain everything: https://news.ycombinator.com/item?id=32028511 https://news.ycombinator.com/item?id=32028511.
• chippiewill 4y ago
  > Double disk failure is improbable but not impossible. It's actually surprisingly common for failover hardware to fail shortly after the primary hardware. It's normally been exposed to similar conditions to what killed the primary and the strain of failing over pushes it over the edge.
  • GekkePrutser 4y ago
    Isn't that more for load balancing than failover? For load balancing I would consider this very likely because both are equally loaded. But "failover" I would usually consider a scenario where a second server is purely in wait for the primary to fail, in which case it would be virtually unused. Like an active/passive scenario as someone mentioned below. But perhaps I got my terminology mixed up. I'm not working with servers so much anymore.
    • 0xbadcafebee 4y ago
      If it's active/active failover then they get the same wear, if it's active/passive most of the components don't, but the storage might. Then again if it's active/passive, flaws can "hibernate" and get triggered exactly when failing over. You know how they say to always test your backups? Always test your failover too.
• deltarholamda 4y ago
  >the failover server also failed Those responsible for the sacking have also been sacked.
• marcosdumay 4y ago
  If you have an active/passive HA setup and don't test it periodically (by taking the active server offline and switching them afterwards), my guess is that double disk failures will be more common than single disk failures for you. Still, I see no reason for prioritizing that failure mode on a site like HN.
• bell-cot 4y ago
  I'm extremely curious about the makes & models of the failed hardware...
• tgflynn 4y ago
  According to this comment: https://news.ycombinator.com/item?id=32024485 https://news.ycombinator.com/item?id=32024485 each server has a pair of mirrored disks, so it seems we're talking about 4 drives failing, not just 2. On the other hand the primary seems to have gone down 6 hours before the backup server did, so the failures weren't quite simultaneous.
  • dang 4y ago
    > so it seems we're talking about 4 drives failing, not just 2. Yes—I'm a bit unclear on what happened there, but that does seem to be the case.
• Pakdef 4y ago
  > Double disk failure is improbable but not impossible. Were they connected on the same power supply? I had 4 different disks fail at the same time before, but they were all in the same PC... (lightning)
  • deleted 4y ago
    [deleted]
  • mikiem 4y ago
    They were in two mirrors, each mirror in a different server. Each server in different racks in the same row. The servers were on different power circuits from different panels.
• sschueller 4y ago
  Depends on your vendor as well. A long time ago we had a Dell server which was pre setup raid from Dell (don't ask, I didn't order it). Eventually one disk on this server died, what sucked was that the second disk in the raid array also failed only a few minutes later. We had to restore from backup which sucked but to our surprise when we opened the Dell server the two disks had sequential serial numbers. They came from the same batch at the same time. Not a good thing to do when you sell people pre configured raid systems at a mark up...