9 ms·
Ask PG: Postmortem of the outage?
Clarification: This is not meant with any ill-will towards PG or any of the other individuals who help run HN. It is a simple request for a postmortem eventually. Perhaps it's an unneeded request, but I think a lot of HNers echo the sentiment.
- theGimp 13y agoIt seems all activity from the past two days has disappeared -- backup storage is something you never regret paying for. You've probably all seen it by now, but from @HNStatus: [1] Server back up and seemingly stable. Now restoring our latest backup to recover from limited filesystem corruption. [1] https://twitter.com/HNStatus/status/420179162138021888 https://twitter.com/HNStatus/status/420179162138021888
- t0 13y agoWere they not using raid or performing multiple database writes? A mechanical hard drive failure is pretty common and can be mitigated fairly easily.
- dvanduzer 13y agoRAID arrays fail all the time; the system has famously been one server, and the only visible recent scaling work has been front end caching. edit: the code has been public for a long time, and there is not a database to replicate. the site ran as a single server for years, and it is unlikely the front end caching has changed anything about the "database" components. Since RAID failures actually are somewhat common, they are probably looking at a higher level replicated storage system now, a la DRBD, or some kind of distributed file system, a la Gluster.
- imbriaco 13y agoDRBD and Gluster are not any more resilient to filesystem corruption than a RAID device is. In this kind of case you hope for either real-time replicated storage on a completely separate physical host or very recent backups.
- dvanduzer 13y agoWhat are DRBD and Gluster if not real-time replicated storage on completely separate physical hosts? Filesystem corruption without hardware failure is far rarer in my experience. Have you seen an instance that wasn't a proverbial user error?
- imbriaco 13y agoYes, I have in fact, in a DRBD configuration. The bug was esoteric, but it happened and was not the result of user error. DRBD and Gluster both allow faults in the VFS layer to propagate to all replicas.
- avifreedman 13y agoGluster should by design I think avoid replicating filesystem metadata corruption (but would replicate internal metadata issues in files on top of the filesystem) but DRBD won't... At high volumes I still regularly break Gluster but it'd probably be OK for lower bandwidth/ops use. Not sure what the HN disk usage pattern is though.
- username223 13y agoIIRC Glusrerfs was the thing that gave me multiple identically-named files in the same directory. Useless.
- nilsbunger 13y agoYou never ran reiserfs I see... Back in ~2004 I watched IT spend a whole day recovering our 60-person startup's main Linux NFS server, due to a software bug in the storage driver. Had to rebuild the whole system from backups.
- sitkack 13y agohttp://basho.com/riak/ http://basho.com/riak/
- tdubhro1 13y agoDeosn't RAID usually at least give some warning if you watch the syslogs? (Genuine question, I am not a sysadmin, we have linux servers with hetzner on software raid 1 and a couple have had single-disk issues which we spotted straight away in zenoss and had hetzner replace the disk. Am I incorrect in thinking this is normal?)
- dsr_ 13y agoRAID is a method for surviving hardware failure. If you have a software failure in, say, the VFS layer, RAID will happily accept the order to write garbage all over your inode trees and will carefully store and make sure that all the appropriate disks can return the same garbage every time. And yes, it should warn you when you need to replace a disk which is no longer returning the right garbage. Similarly, if you rm -rf a vital directory tree, RAID can ensure that it goes away reliably.
- andrewcooke 13y agoyes you're right. so replies will now switch to how they don't stop you from deleting data, because... well, i have no idea why. it seems to just be a law of nature.
- cbsmith 13y agoOr, I dunnoh... writing to S3? ;-)
- deleted 13y ago[deleted]
- venus 13y agoDatabases? What databases? HN is persisted to flat files.
- Sami_Lehtinen 13y agoI guess he meant having two separate logs. One for production, and secondary with his journal. In this case you could restore from backup the original data, and then replay rest of stuff from the external log. That's the solution I'm using with really important data where I cannot afford any data loss, even if down time is acceptable. On commit, it committed to two separate systems, but the secondary system is only journal which can be replayed.
- yaddayadda 13y agoSeveral comment threads that I was following when it went down are gone ("No such item."), although their original links are still valid.
- theGimp 13y agoIndeed you are right; the original links still work for me. Though the ones I checked so far look exactly the same as the Google Cache version, so I don't know what happened there.
- sehrope 13y agoYes I noticed this as well from (the lack of) my own comment activity. I don't comment that often but I had written something yesterday that has disappeared. On a more general note if anybody has backups and they aren't regularly tested restoring them, then you really don't have backups! As an added bonus, regular restoration tests let you practice for the "real deal" and you know how long the entire process will take.
- pacaro 13y agoThis. Sometime last century I had to restore an old codebase from a tape backup. Step one find the correct drive... We'll never need that old repo again.
- jon-wood 13y agoOne of the nicest ways I've experienced of making sure your backups are good is to sync up your development machine with them occasionally. Obviously there are situations like HIPAA where you can't do that, but if you can, do. You'll catch problems with your backups long before you actually need to use them.
- pconf 13y agoIMO data loss is less a symptom of untested backups than it is of developer-managed systems. I wonder if ycombinator has a sysadmin (who isn't also a developer)?
- aroman 13y agoYeah, I lost about 200 karma (what was about 15% of my total) in the crash. Good thing they're just silly internet points :)
- yaddayadda 13y agoThank you for making my paltry 22 lost karma points look so, well, so paltry. ;-) edit: to clarify "lost" karma
- firebones 13y agoIf this were reddit, you'd be getting tipped in Dogecoin as well.
- username223 13y agoHere, unfortunately, Internet play money is Serious Business.
- 8ig8 13y agoJust gave you an internet point for having perspective.
- allochthon 13y agoI gave you back an Internet point.
- techaddict009 13y agoI lost few comments :( Now commented back.. Hope the thread owner reads them and replies back.
- bvirkler 13y agoI lost 25% of mine! A whole point!
- 13y ago
- 8ig8 13y agoWhat's odd is that if you look at your submission history, you can up vote your own submissions. Maybe that's nothing new, but I just noticed it. Seems like a bug.
- neurostimulant 13y ago> What's odd is that if you look at your submission history, you can up vote your own submissions. It doesn't seem to do anything though.
- mixmastamyk 13y agoThat's a shame, I was really looking forward to the comments for the article below. Unfortunately I had it loaded, but hit Ctrl+R (like I sometimes do) and lost it forever. :/ The google cache got a few comments, but very few. http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/ http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/ https://news.ycombinator.com/item?id=7015438 https://news.ycombinator.com/item?id=7015438 http://webcache.googleusercontent.com/search?q=cache:9368SwVPHqIJ:https://news.ycombinator.com/item%3Fid%3D7015438+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:9368SwV...
- yaddayadda 13y agoI missed the original posting of https://news.ycombinator.com/item?id=7015438 https://news.ycombinator.com/item?id=7015438 and it's right up my alley (now it's tomorrow morning's reading :) )
- yaddayadda 13y agoI've reposted a link to the original here - https://news.ycombinator.com/item?id=7015767 https://news.ycombinator.com/item?id=7015767 I really don't want credit, I just want the original question and the couple of responses to have the opportunity to see the day of light, despite the HN outage. So please vote up the original post!!!
- hga 13y ago"backup storage is something you never regret paying for" Indeed, but the time and CPU it takes to do backups can be a very nasty trade off. For that matter, as I write this, due to a mistake I made after a 85 minutes power outage yesterday, I'm just now doing my daily incremental backup of my home machines to an LTO-4 tape drive. Keeping that drive fed fast enough to prevent "shoe-shining" took some effort, Bacula spools up to 100G at a time to a partition on a single 15K disk on a separate controller. But if I had a LTO-5 drive, from what I've heard there's no single disk in existence that can keep up with a drive (not counting SSDs, which are a very poor match for this use case).
- sitkack 13y agoMy feed array to the LTO-5 drive is 4 2TB (Hitachi) in RAID-10. Backup strategies, much like build systems are some constant factor harder than they appear. I'd like to migrate to ZFS but have yet to. Still just running EXT4. HN should be on a replicated data store like Riak. Losing a node or two shouldn't take the system down, or should at least run in a degraded state (read only) until hardware is restored.
- nhangen 13y agoBecause it's not good enough that the site is back, we need to pile on and complain too...
- lukeqsee 13y agoI'm not complaining. I'm asking a legitimate question. A popular site has been down for a significant period of time; I think a postmortem will be insightful.
- nhangen 13y agoIf you're asking, it means you know that PG authors them in most instances. The fact you felt the need to start a new thread, just minutes after the site is back, tells me you're anxious and overeager.
- lukeqsee 13y agoI don't disagree with your assessment. I think it's more of an opportunist running an (admittedly) self-serving social experiment of my own. And I was also curious (naturally) and well-aware someone would ask the question. I wasn't complaining, however. :) Edit: clarification on motivation.
- yaddayadda 13y agoAsking a technical site about a technical site outage isn't, in and of itself, complaining. It's an opportunity to learn.
- deleted 13y ago[deleted]
- bradly 13y agoAsking for a postmortem is not complaining.
- nhangen 13y agoThe site has been up for minutes and already asking for a report. I'm sure that's the last thing on PG's mind at this point. It's just clueless and classless.
- xmonkee 13y agoSocial experiment
- jader201 13y agoI considered this as well (seriously), until I saw the lost data from the past day or so.
- noblethrasher 13y agoYou may jest, but I once suggested something like that: https://news.ycombinator.com/item?id=2403880 https://news.ycombinator.com/item?id=2403880
- geerlingguy 13y agoWould like to read it too. And it looks like right now is a good time to get just about anything in the front page. Front pretty much == new.
- ewoodrich 13y agoAre you actually able to see 'top'? I'm still getting the error. EDIT: (never mind, it was just cached)
- geerlingguy 13y agoYeah, the 200 response during the outage is playing with everyone, I think; you have to do a hard refresh on any URL you had visited during the downtime :/
- stickhandle 13y agoIt bothered me more than it (probably) should
- sigvef 13y agoDuring the outage, https://twitter.com/HNStatus https://twitter.com/HNStatus went from somewhere around 300 to 1163 followers.
- lukeqsee 13y agoEarlier today I saw it had about 45 followers. I think it was a new account. (Please correct me if I'm wrong.)
- yaddayadda 13y agoTheir first post is dated 28 July - https://twitter.com/HNStatus/status/361707202123268096 https://twitter.com/HNStatus/status/361707202123268096 edited: "They're" --> "Their" (there/their/they're will be the death of me!)
- asveikau 13y ago> (there/their/they're will be the death of me!) I sure hope that isn't literal. I've heard of "grammar nazis" but that would be ridiculous. Stay safe!
- MichaelApproved 13y agohere's a tweet from July 28th, so it's not a new account https://twitter.com/HNStatus/status/361707202123268096 https://twitter.com/HNStatus/status/361707202123268096
- cenhyperion 13y agoIt has tweets from as far back as July of 2013.
- pairing 13y agoI'm surprised HN is using twitter instead of one one of their investments, Statuspage.io.
- lucb1e 13y agoI actually looked for a twitter account, but didn't know which would be official. Then jgrahamc retweeted hnstatus and I knew which to follow.
- yaddayadda 13y agoI find it interesting that this question is fresher (by a minute), has more points (67 v 42 at snapshot), and has more comments (18 v 10 at snapshot) than "HackerNews down, unwisely returning http 200 for outage message" but is ranked lower (2 v 1 at snapshot). snapshot - http://oi40.tinypic.com/2mmbv5y.jpg http://oi40.tinypic.com/2mmbv5y.jpg
- deleted 13y ago[deleted]
- Kronopath 13y agoSelf posts are penalized so they don't clog the front page for long. http://jacquesmattheij.com/The+Unofficial+HN+FAQ#selfposts http://jacquesmattheij.com/The+Unofficial+HN+FAQ#selfposts
- dschiptsov 13y agoIs there any plans to release a new version of Arc, if it exists or server side code (without business-critical stuff)? I guess that there are lots of improvements since last Arc release.)
- cenhyperion 13y agoI'm also interested in what the infrastructure of HN looks like. One of the tweets via @HNStatus seemed to imply that the site runs off of one application server.
- sigvef 13y agoHN is indeed running on a single (10 month old) server, it seems [1]. [1]: https://news.ycombinator.com/item?id=5229364 https://news.ycombinator.com/item?id=5229364
- morganherlocker 13y agoI was bummed that the conversation around openstreetmaps got killed in the middle of it, and now I do not see it on the front page. Does anyone have a link to that thread or did it disappear?
- stokedmartin 13y agoGoogle cache has a partial copy[0] [0] http://goo.gl/MnWNPw http://goo.gl/MnWNPw
- deleted 13y ago[deleted]
- eevilspock 13y agome too. i guess we can start over: https://news.ycombinator.com/item?id=7015502 https://news.ycombinator.com/item?id=7015502
- jpatel3 13y agoIts no longer on the front page anymore..
- dbaupp 13y agoIt's been resubmitted: https://news.ycombinator.com/item?id=7015502 https://news.ycombinator.com/item?id=7015502 (But I think the original thread was totally lost, I submitted it and it's not listed in my submission history.)
- tambourine_man 13y agoThe old discussion is still on Google's cache (for now): http://webcache.googleusercontent.com/search?q=cache:PmyjVgeonQIJ:https://news.ycombinator.com/item%3Fid%3D7015294+&cd=1&hl=en&ct=clnk http://webcache.googleusercontent.com/search?q=cache:PmyjVge...
- rhizome 13y agoPostmortem: it went down last night when people should have been going to sleep before their first day back at the job after holidays. It stayed down until the end of that day, with the last couple of days of vacation insanity erased. Appreciate the gift of perspective that has been given.
- carljoseph 13y agoInteresting that your perspective is locked into one side of the globe. ;) HN was down during the day my time, when we had already slept before returning to work. :) Appreciate the gift a new perspective gives you.
- rhizome 13y agoI scoped it to a YC framing.
- royalghost 13y agoI am sure pg is going to write an essay on this :-)
- joshuaheard 13y agoI'm not an expert in internet architecture, but shouldn't a site this important be running on redundant servers? The irony of a tech site going down due to technical issue is making me grin, however. Glad to see it back :)
- alan_cx 13y ago"Important"? Really? Obviously Im a fan of the site, etc, etc, but "important"? On what level? Im not even sure I'd call Facebook or Twitter important. Banking, yes. Weather warnings, yes. Things like that, sure. But, Im also pretty sure "important" is slightly over egging it for dear HN. (No offence PG xxxx)
- acjohnson55 13y agoI definitely find it important, from a career perspective.
- username223 13y agoWhaaa? I guess I can see that, in the sense that chatting and/or blowing off steam may have psychological benefits, but we're both just killing time here. Don't kid yourself.
- zerd 13y agoI've learned a lot on HN, and I think learning is important, ergo I think HN is important. Which is why I have a backup of it in case this happens again.
- arvidjanson 13y agoImportant, absolutely – important as in critical service requiring 100% uptime? Not so much.
- kamaal 13y ago>>Im not even sure I'd call Facebook or Twitter important. Imagine Twitter or Facebook being down during Egyptian revolutions.
- DonGateley 13y agoIf the outage was due to something malicious I don't really expect to see a postmortem.
- pg 13y agoI don't know the details. Nick Sivo is in charge of this stuff, and he'll post something about it. I know he thinks the root of the problem was a disk failure. The server got wedged, and when we rebooted, the file system was corrupted. I'm not sure exactly why it took so long to restore. I was out of town the whole time this was happening. The reason we lost so much data was that we only do nightly backups. That seemed enough when we started. Now that HN is a bigger part of more people's lives, we'll make more of an effort to make it proof against this sort of problem.
- saurik 13y agoAnother point to make is that, even with a lot of data loss, it at least would be good in the future to skip a bunch of post identifiers (potentially even just saying "well, we certainly didn't use 100,000 of them, so I skipped up to 7100000") in order to try to not cause data that used to be there, and may have been archived by various sources including Google Cache or in the various reader clients people use on various devices , to suddenly have been swapped out by different posts leading to wide-spread cache corruption (and further confusion or loss). I mean, even just for the sake of people who may have posted links to this content in various places: maybe they wrote an article or posted a tweet/comment somewhere referencing how great/horrible a post was, and now suddenly it is saying something entirely different and potentially quite awkward ;P. As a quick example, for those still not sure what I mean: what used to be an article about Python 2/3... http://webcache.googleusercontent.com/search?q=cache:9368SwVPHqIJ:https://news.ycombinator.com/item%3Fid%3D7015438+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:9368SwV... ...becomes a comment with numerous references for how to learn arduino hacking. https://news.ycombinator.com/item?id=7015438 https://news.ycombinator.com/item?id=7015438
- jaakl 13y agoThe problem happens also with HN own search, tried to get comments for first entry of https://www.hnsearch.com/search#request/all&q=openstreetmap https://www.hnsearch.com/search#request/all&q=openstreetmap
- 13y ago
- rcfox 13y agopg: I don't know how much you care to get back the data that was lost, but it seems like it's at least partially available in the hnsearch.com API: http://api.thriftdb.com/api.hnsearch.com/items/_search?pretty_print=true&sortby=create_ts%20desc http://api.thriftdb.com/api.hnsearch.com/items/_search?prett...
- pearjuice 13y agoDo we get the karma we lost refunded somehow? I am certain I am missing around 30 points.
- pbhjpbhj 13y agoBecause?
- nmc 13y agoDespite the website being back online, the root URL still redirects to the error page (at the time of writing this). So https://news.ycombinator.com/news https://news.ycombinator.com/news works, but https://news.ycombinator.com https://news.ycombinator.com still redirects to "Sorry for the downtime. We hope to be back soon.".
- watermel0n 13y agoIt's your browser cache.
- nmc 13y agoYes it was!
- Nodex 13y agoWho / what is "PG" - I have noticed this a lot over the last few days
- rainmaking 13y agoThis must have been the most productive time for the tech industry in months.
- ithkuil 13y agoNo, I just kept wasting time reloading HN home page or following notifications on twitter!
- ithkuil 13y agoI wonder how much effort would be reasonable to improve the resilience of HN to this kind of issues, given that's a relatively rare issue and HN doesn't really have a money loss in case of a downtime such as this.
- ams6110 13y agoProbably little. There's no ad revenue being lost, no business transactions that can't be completed, no life-saving information that can't be accessed. When you boil it down, it's a social/entertainment site, nothing anyone can't live without for a day or two.
- vanwilder77 13y agoDamn I made it to 1300 karma's! Now I'm back at 1273.