13 ms·

Why HN was down

1049 points by pg 14y ago

Hacker News was down all last night. The problem was not due to the new server. In fact the cause was embarrassingly stupid. On a comment thread, a new user had posted some replies as siblings instead of children. I posted a comment explaining how HN worked. But then I decided to just fix it for him by doing some surgery in the repl. Unfortunately I used the wrong id for one of the comments and created a loop in the comment tree; I caused an item to be its own grandchild. After which, when anyone tried to view the thread, the server would try to generate an infinitely long page. The story in question was on the frontpage, so this happened a lot. For some reason I didn't check the comments after the surgery to see if they were in the right place. I must have been distracted by something. So I didn't notice anything was wrong till a bit later when the server seemed to be swamped. When I tailed the logs to see what was going on, the pattern looked a lot like what happens when HN runs short of memory and starts GCing too much. Whether it was that or something else, such problems can usually be fixed by restarting HN. So that's what I did. But first, since I had been writing code that day, I pushed the latest version to the server. As long as I was going to have to restart HN, I might as well get a fresh version. After I restarted HN, the problem was still there. So I guessed the problem must be due to something in the code I'd written that day, and tried reverting to the previous version, and restarting the server again. But the problem was still there. Then we (because by this point I'd managed to get hold of Nick Sivo, YC's hacker in residence) tried reverting to the version of HN that was on the old server, and that didn't work either. We knew that code had worked fine, so we figured the problem must be with the new server. So we tried to switch back to the old server. I don't know if Nick succeeded, because in the middle of this I gave up and went to bed. When I woke up this morning, Rtm had HN running on the new server. The bad thread was still there, but it had been pushed off the frontpage by newer stuff. So HN as a whole wasn't dying, but there were still signs something was amiss, e.g. that /threads?id=pg didn't work, because of the comment I made on the thread with the loop in it. Eventually Rtm noticed that the problem seemed to be related to a certain item id. When I looked at the item on disk I realized what must have happened. So I did some more surgery in the repl, this time more carefully, and everything seems fine now. Sorry about that.

bramcohen 14y ago
You should probably make your code robust to this sort of data corruption in the future.
birken 14y ago
Do you have munin monitoring on the production HN server? That would really make situations like this easier to debug. First, it can pinpoint exactly when something started happening, which in this situation might have helped you realize the problem was caused by your change. Secondly, in this specific situation it probably would have been easier to differentiate a situation where you are running low on memory vs this completely different situation. As somebody who spent a lot of time professionally debugging large software systems when they were misbehaving (as a Google SRE), I can tell you that looking at graphs of many key metrics (disk IO, CPU, memory, then application specific things) was always the place to start when debugging a situation, because you can learn so many things right away. When did it start? Was it a slow buildup or an immediate thing? What is the general problem (Memory?, Disk IO?, CPU?, none of the above?)? Has a similar pattern happened in the past? Then you can start to get fancy and plot things like "messages/minute" or something and then it becomes easy to see when issues are affecting the site performance and when they aren't.
- stcredzero 14y ago
  That and something like the Smalltalk Change Log would have made this a no-brainer debug. (Yes, every REPL action in Smalltalk got logged by the same mechanism that logged every code change.) Such mechanisms aren't trivial, but they're not rocket science either, and they have tremendous ROI.
youngerdryas 14y ago
>On a comment thread, a new user had posted some replies as siblings instead of children. I posted a comment explaining how HN worked. But then I decided to just fix it for him by doing some surgery in the repl. No good deed goes unpunished! People sometimes reply as sibling because they too impatient wait for the built-in delay on child comments. Thanks for keeping the experiment going.
lifeisstillgood 14y ago
There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing. An awful lot of sites at 1.2m views would have outsourced the running and development of the whole thing - there are entreprenuers who say its not even worth our time to code up the MVP. I find this approach sensible from a business point of view, but still it does not sit right with me. I am supposed to have a nice website with lots of good content to attract inbound marketing - so I tried getting someone on textbroker to write an article for me. It read like a High School essay - no life, no anime. And so I will probably write my own CMS and my own content. And pg sits there and writes his site in his own language, with his own moderation tools. Apart from the hilarious idea he could find a ten person ruby shop to outsource to, its nice to see someone taking the time to play again. Its why I like to see jgc on here too. I am not entirely sure those thoughts are joined up (I am procrasting like crazy) but if they come to mean anything its we are playing in pg's sandbox. If the sand leaks it's his sand, and the only company this is mission critical to is YC.
- stcredzero 14y ago
  > It read like a High School essay - no life, no anime. ... I am not entirely sure those thoughts are joined up (I am procrasting like crazy) Your procrasting like crazy has much anime.
- luser001 14y ago
  > I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing. IIRC, sudo logs all commands to syslog. Which might come in handy. Yes, root commands will be logged by bash to .bash_history, but there are limits of # of commands lots, what happens if you are logged in multiple times into same account etc. Anyway, that's why I like sudo.
  - 14y ago
user911302966 14y ago
This, folks, is why NOSQL/new school/whizbang/* will never replace a properly designed RDBMS. Period.
neurotech1 14y ago
This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes. Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.
- jgrahamc 14y ago
  "No deflecting blame" Who were they going to blame?
  - wpietri 14y ago
    He could have blamed the new server. Or whatever distracted him. Or the user, for being dumb. I've seen people do all of those. Or he could have just dodged the blame entirely.
    - dguaraglia 14y ago
      Some people are just freaking difficult to work with. I've worked with people that wouldn't accept responsibility even after every other possible cause was ruled out. I've even gotten this reply: "well, you must have been unlucky to get the faulty e-mail, because it seems to work most of the time". Yeah, because that's how programming works: cowboy coding and hoping for the best. This guy actually called bugfixes "optimizations": "hey, X the feedback widget on the front page isn't working", "oh, yeah, I haven't worked on that because that's code that needs to be 'optimized', so it's low on my issues list". Ugh. I've learned my lesson now. In fact, that's an incredible lesson for a startup founder: never, ever, hire someone who dodges a question on an interview. And the first time they avoid taking responsibility for something that was clearly their fault, fire them. The last thing you want is someone who'll blame everyone and anything else for their issues. It's a great way to kill morale and create rifts in a small team.
      - artursapek 14y ago
        pg has an essay where he says the smartest people he knows are always willing to take blame or admit they don't know the answer to a question.
DanI-S 14y ago
n.b. that this is why time travel is a terrible idea.
- cmaggard 14y ago
  The grandfather paradox is the best solution to the halting problem.
- w_t_payne 14y ago
  Because the computers that run the Matrix will get overloaded?
benatkin 14y ago
So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?
- badgar 14y ago
  > So what do you do to avoid this in the future? It's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.
  - benatkin 14y ago
    > It's HN... there's no SLA, there's no postmortems I didn't mean to imply that there were. I was just curious. > there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all. Since there are multiple volunteers, I think that the site always feels important to at least one of them. I imagine that some of them have gone more than a week without giving a shit about HN, but not all of them at once. So I think there is doing things better in the future. In fact, HN keeps getting improvements behind the scenes, to keep it running, keep it interesting, and keep it from getting overrun with trolls.
  - martinced 14y ago
    I don't think parent did mean that he had a "right" to expect some level of quality or anything. It's just that we, as programmers, tend to take measures so that silly bugs do not happen anymore or that, at least, we leave big clues as to what went wrong. In a project I had a similar issue: I was wrapping lists inside immutable lists but, due to a silly bug, I kept wrapping immutable lists inside immutable lists at every save made. So saved files would grow bigger and bigger. And I did fix the bug and also added a big fat warning logs in case too many nested lists were detected. pg might just as well have now added something preventing infinite recursion inside the comment tree or some WARN logging telling when a generate page is getting too big, etc. I'd still find it very interesting to know what pg did, if any, to dodge / minimize / make it easier to determine if such an issue happens in the future.
nowarninglabel 14y ago
Happens to a lot of us. Great reason to always write tested cleanup scripts for this stuff instead of editing directly on the server. The only time I brought down my product last year was from a similar screwup, I was removing users by hand and somehow managed to end up with a 0 in my list of user ids, thus deleting the anonymous user, and causing havoc to my server, which took a long time to track down.
cool-RR 14y ago
Great debugging story! I guess the lesson is to have code that alerts you about comment loops without going into an infinite loop. Also another lesson would be to figure out a way to have better clarity into which requests are causing a timeout on the server.
deleted 14y ago
[deleted]
DanielBMarkham 14y ago
Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging http://en.wikipedia.org/wiki/Rubber_duck_debugging
- davesims 14y ago
  A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally. This serves several purposes: (1) It's less insane-sounding than actually talking to an inanimate object in an open work environment. (2) It actually feels better and forces me to think more clearly when I'm talking to an actual person -- the cognitive focus is higher when the object of conversation can actually, in theory, think and talk back (YMMV). (3) And finally, although it does require some focus on the part of the other coder, it's not nearly as taxing to them as actually helping me solve the problem or pairing up with me. So it's a good compromise somewhere between pair programming and talking to an actual rubber duck. Again, YMMV. Maybe I'll call it "Pair Ducking."
  - kamjam 14y ago
    This is so true, probably about 90% of the time my colleagues call me over about a problem they are facing, explain in detail what the problem is and then eureka! Most of the time it would actually take me much longer to figure out the exact issue since I don't know the ins/outs and subtleties of the code but it's exactly as you say. Of course, it makes me look really good cos I just "helped" them solve their issue :)
    - cientifico 14y ago
      Maybe because the solution is in asking the good questions.
      - 14y ago
lucb1e 14y ago
Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?
- martinced 14y ago
  "Are you saying you manually modify the database?" Oh manually modifying production database on the fly ain't unheard of. However it's still not "very Chuck Norris" on a scale of Chuck Norrisness compared to the modification of a running app directly in the REPL. I mean: it doesn't matter if you manually modify the DB itself or not when you directly modify the app from the REPL itself (the app being anyway "in charge" of the DB). Sure, modifying manually the production DB might be an issue to some. But I can guarantee you that it's the last of your worries when you're actually modifying production code directly from the REPL ; )
- RKoutnik 14y ago
  Sometimes that's easier (albeit more dangerous, as we just saw).
- sgt 14y ago
  I think I get what you're hinting at. Ok, so this is Hacker News, it's in the name, and most of us are aware that HN is also a research/hobby project. It's not made to be an rock-stable enterprise system doing bank transactions or what not, so I think what pg did was prefectly excusable. People make mistakes. Nobody will die without HN for a day or two, and it won't affect the site's popularity one bit.
  - lucb1e 14y ago
    No that's right, but I worry about apache2 being down for potentially one or two users or bots that visit/crawl my website during a one-minute reboot. Meanwhile the big boys are down for 16 hours because they do things that any other person would have gotten a decent scolding for. Just look at the points per hour this thread is getting, if I had posted this about my website on my website people would have said I was stupid. You are right though, making mistakes is human as they say, and nobody dies because of this. In fact, less popularity might be good for the site's content quality. I'm just surprised by how much they care about thousands of hourly users, that what I would dream of having.
sgt 14y ago
Much appreciated, pg. I knew that the "10 minutes of downtime" would not occur (fair enough, this was not related to the server upgrade).
RKoutnik 14y ago
It's nice to know that even the mightiest of us can still make mistakes. Thanks for being willing to admit mistakes so the rest of us can learn.
luser001 14y ago
I use assertions to protect against things like this. I liberally sprinkle my code with assertions (CS theory calls them pre-conditions and post-conditions, iirc) to crash early if the system is an invalid state. One my pet peeves is that few programmers seem to love assertions like I do. Would love to see to comments on this.
- swah 14y ago
  The kind of assertion he needed though, could only be ensured by the database, not application code (my impression).
  - luser001 14y ago
    Agreed, infinite loops are a little hard to protect using asserts. When I hit the first infinite loop bug on a code path, I frequently add code to assert that the number of calls is less than $A_LARGE_NUMBER to catch future occurrences of the same root cause.
    - badgar 14y ago
      > Agreed, infinite loops are a little hard to protect using asserts. assert(is_tree(comment_graph)) Typically, a composite entity (like an "item" on HN which has many "comments") will define invariants to ensure data integrity. In this case, the invariant is that an "item"'s comments form a tree. The database layer often contains this logic, but it depends on how you're building your application; NoSQL backends for example typically must put validation in the application layer. Since HN just uses files, a well-developed application layer should be riddled with invariants like this.
    - mpweiher 14y ago
      I dimly remember a language that just hard-limited loops. I thought it was John Pane's HANDS system, but I can't seem to find a reference in the thesis...can anybody refresh my memory? http://www.cs.cmu.edu/~pane/research.html http://www.cs.cmu.edu/~pane/research.html http://www.cs.cmu.edu/~pane/thesis/ http://www.cs.cmu.edu/~pane/thesis/ Pretty cool work regardless, I really like the way it deals with aggregates, for example.
dasil003 14y ago
I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.
- hnriot 14y ago
  he's not legendary for his IT skills.
  - dasil003 14y ago
    Low blow.
  - jbverschoor 14y ago
    Now he is ;)
  - mark-r 14y ago
    It was his IT skills that got him the big sale to Yahoo that got him the bucks to start YC. Not sure where this comment came from.
    - papsosouid 14y ago
      You really think yahoo bought viaweb because of pg's legendary ability to reboot servers and type "./configure && make && sudo make install"?
- larrys 14y ago
  Some of the most spectacular airplane crashes are by the most experienced pilots. If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.
  - stcredzero 14y ago
hnriot 14y ago
it's a good job it's your site, this type of thing is often what gets someone fired in a company. Modifying (meddling!) the production system directly.
- lucb1e 14y ago
  Upvoted. This is what I wanted to reply, but then thought better of it and moderated my response.
  - irahul 14y ago
    I think you people already know the answer. The amount of freedom and stake/reward system for pg is different from yours. I can't speak for pg, but personally, I am not going to write a migration to re-parent a single thread if the site in question is my side project, doesn't bring revenue, has some intangible benefits, but not so much that warrant putting much labor into it. Either it would be `thread.parent = new_paret_id`; or if it occurred to me that it might introduce a loop, changing `parent=` to take loops into account followed by `thread.parent = new_parent_id`. What were you expecting? A bug tracker discussion, code commit, review, change request and deployment?
    - hnriot 14y ago
      no, you just leave it alone, it would have sorted itself out if nothing had been done at all. failing that you put a cap in the code that generates the page. Simple stuff.
    - dap 14y ago
      The problem is thinking of it as the cost of implementing the feature vs. doing manual surgery on the production database, without realizing that if you choose the latter, you're also choosing the risk that you'll spend hours debugging the system when the surgery goes wrong. It's a tradeoff, to be sure, but it's not clear that the latter is cheaper on expectation.
- JohnBooty 14y ago
  I disagree. Very few companies would think negatively of an engineer if they made such a mistake on a non-essential, non-revenue-generating fun/research project. How many dollars did YC lose because of the outage? None. (Maybe they saved a few on bandwidth!) I also predict that exactly zero startups will say, "Man... I'm not going to take seed money from those guys! They had discussion forum downtime."
gruseom 14y ago
This is a particularly endearing piece of "hacker news". It's so easy to relate to.
jack57 14y ago
Are you sure that comment's name wasn't Phillip J Fry?
- jack57 14y ago
  Apologies for the trivial comment
Legion 14y ago
"We'll do it live!"
- eytanlevit 14y ago
  epic
- jbuzbee 14y ago
  "Hey, Hold my beer and watch this!"
T-zex 14y ago
Thank you for the honest explanation. This is not so easy especially for a famous person.
irahul 14y ago
Disclaimer: Hindsight is 20/20, and stuff. If reverting code didn't fix it, reverting server didn't fix it, incorrect data is the most likely culprit(I am not claiming this should have outright occurred to you; just thinking out loud). I take it you introduced non terminating recursion by making a thread its own parent, and you made the change on disk. But this analysis is the last thing that comes to mind when you already have introduced 2 new variables the same day - new code, new server. And an old, recurring variable(GCing too much) is in play as well.
hyh1048576 14y ago
The "More" button at the bottom is still not clickable sometimes. Get a "Unknown or expired link." message.
andreasklinger 14y ago
I appreciate (if not love) the fact that you bugfix and server-change yourself. True hacker spirit.
hyh1048576 14y ago
The "More" button at the bottom is still not clickable sometimes. Get a "Unknown or expired link." message.
xentronium 14y ago
Whoa, what an unfortunate coincidence. This whole bug would be so much easier to find, if it weren't for the new server.
- lmm 14y ago
  The bugs that actually hit production are always like this - a confluence of three or so factors - because if it were simpler you'd have caught it earlier. (Though I have to say, upgrading the code at the same time as you're restarting to fix a problem is really a rookie mistake. It's incredibly tempting because it saves so much time, but if you do it you will get it wrong sooner or later. One of the hardest skills in programming is acquiring that zen that you need to wait in a state of readiness for the effects of your first change to make themselves apparent, rather than changing something else)
naturalethic 14y ago
If the problem existed before the code update, why would you assume it was the code update that caused the problem?
larrys 14y ago
"But then I decided to just fix it for him by doing some surgery in the repl." I've always found it's a good idea to not deviate. Whether it be running, parking or anything else once you deviate from some regular behavior you run into potential problems that you hadn't anticipated. "For some reason I didn't check the comments after the surgery to see if they were in the right place. " More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur.
- irahul 14y ago
  > I've always found it's a good idea to not deviate. Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him? > More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur. How about the opposite scenario? He has done it so many times with desired results, that he didn't bother checking?
  - badgar 14y ago
    > Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him? This is a big difference between engineering and hacking. An engineer would never regularly do something so dangerous. But I suspect pg isn't an engineer when he works on HN, I suspect he is a hacker, and just does whatever he wants to, whenever he wants. Which is his prerogative.
- prawks 14y ago
  > I've always found it's a good idea to not deviate. > you run into potential problems that you hadn't anticipated. The second statement is no reason to live by the first. In fact, I think you'd be doing yourself a disservice by staying so comfortable. Being comfortable with the unanticipated, however, is a powerful quality to have.
ricardobeat 14y ago
Related question: what is the timing for the 'Reply' link to show up? I might be fantasizing but sometimes it takes 5, sometimes 10 minutes to appear, leading people to reply as a sibling instead.
- DanBC 14y ago
  Deeply nested comments tend to be hot, and so the reply link takes a while to show up to try to give people some time to think about what they're going to say.
  - ricardobeat 14y ago
    Interesting. So it slows down discussion as it progresses, until it either stops, is forgotten or becomes a series of long essays.
- beagle3 14y ago
  The deeper it is, the longer it takes to appear (I think at depth 7 or 8 it starts counting in hours, and at some point it just won't appear at all). Just like too many nested if(x) { if (y) { if (z) ... }} constructs, too deep a discussion nesting is also unreadable.
scotthtaylor 14y ago
Normality has returned :-)
btilly 14y ago
That explains something weird I saw. If I went to Google's cached copy, I could see threads, and then click on them. But the front page was down. But I could see individual threads. Very confusing.
wpeterson 14y ago
I guess it's time for NewRelic to add an Arc agent.
campnic 14y ago
The nice thing about surgery with a computer program on a server is that death is not permanent.
scotthtaylor 14y ago
PG, quick question: Did this impact the server hosting the YC Summer 2013 applications? When I tried to edit mine, it simply said "Thanks, scotthtaylor"
- scotthtaylor 14y ago
  Working again now.
orangethirty 14y ago
It makes me feel good knowing better programmers than me go through the same issues I face. :)
- thunderrabbit 14y ago
  word
louischatriot 14y ago
Funny to see that this happens to everyone. A week ago, while testing some stuff to locate a low-importance bug, I erased the whole user database. Fortunately we have a good restore so the problem was solved in a few minutes, but still, cold sweat here ...
neilxdsouza 14y ago
Isn't it curious that the comet incident over Russia happened so close to the pass of DA 14. In the intro to the book: http://ruby.bastardsbook.com/about/#why http://ruby.bastardsbook.com/about/#why is the note about surgical instruments left inside. It seems just like a coincidence that this happened so close to the switch to the new server, but I wonder if it's something deeper in the subconscious mind; the change to the new server is quite a big change (I know I feel that way when I have purchased a new computer (it feels different - even if it's running the same linux as before)) and could have upset the normal checks one has in place when tweaking things.
d0m 14y ago
Hacking code in the repl without testing the new behavior. We all did that. Don't lie. Once I wanted to quick fix a "gmail.ca" to "gmail.com", which I did.. but to all the users instead of just the one mistaken. Fortunately I realized by mistake really fast ;-)
- deleted 14y ago
  [deleted]
aaronh 14y ago
My pet peeve: You made an arbitrary change while debugging a problem. NOW YOU HAVE N^2 PROBLEMS!
sehugg 14y ago
Great postmortem and good lessons to learn here: * Don't manually modify database without a well-tested procedure and another pair of eyes * Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) I'm pretty sure I've repeated this exact same sequence before with similar results...
- scotty79 14y ago
  * while you are displaying a tree keep track of the items you already displayed so you can detect a cycle
  - sukuriant 14y ago
    I think the assumption there was that it was safe, since the code disallowed this from happening, naturally.
    - scotty79 14y ago
      Assumption is the mother of all screw ups. Even if you think that the code that creates and modifies your data will not put it in some undesired state, the code that uses this data should assume that the data may be in all undesired states you can dream up and should do its best not to do something seriously bad when that happens (like landing in endless loop/recursion or executing possibly user provided strings).
- JonM 14y ago
  * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) ^^ absolutely!
- zzzeek 14y ago
  use CHECK constraints to prevent invalid data patterns when possible.
eluos 14y ago
"I am my own grandpa"
tolmasky 14y ago
Why do "self posts" like this show up in the same light gray as posts with negative vote counts? My eyes aren't great and I find it hard to read
- roryokane 14y ago
  I don’t know pg’s reasons for making self-posts light gray, but you can fix problems like that with the bookmarklet Zap Colors: https://www.squarefree.com/bookmarklets/zap.html https://www.squarefree.com/bookmarklets/zap.html
- hayksaakian 14y ago
  Maybe post color is based on some get_text_post_color method that applies to self posts and comments, where the color depends on comment karma. Given that self posts like the OP are votable as if they were a normal link post, their comment karma value is probably 0.
- emillon 14y ago
  The rationale for this is that if you need to post a long text post, it should be in the form of a blog post instead. I agree with you that it's not really adapted for a meta post.
- vl 14y ago
  I'm using "Hacker News Enhancement Suite" Chrome extension - it fixes multiple problems, including this one.
fnordfnordfnord 14y ago
>>I caused an item to be its own grandchild. Please forgive me. I know you folks tend to hate jokes on here. Don't waste your time if you're immune to corny humor. "I'm My Own Grandpa- Ray Stevens" ( with family tree diagram) http://www.youtube.com/watch?v=eYlJH81dSiw http://www.youtube.com/watch?v=eYlJH81dSiw
carpathios 14y ago
Seems related: http://meta.stackoverflow.com/questions/66377/what-is-the-xy-problem http://meta.stackoverflow.com/questions/66377/what-is-the-xy...
mikedmiked 14y ago
> created a loop in the comment tree; I caused an item to be its own grandchild. Ah, the online forum equivalent of going back in time to kill your grandfather.
dap 14y ago
Thanks for the detailed explanation. It sounds like everything was done to fix the problem except try to figure out what the problem actually was. Why not use tools to see what the program is doing, form a hypothesis, gather data to confirm or reject the hypothesis, repeat until cause found, and then take corrective action that by this point you have high confidence will work? I realize HN is more of a side project than a production service, but the goal is the same in both cases: to restore service quickly so you can move on to other things. It feels like a more rigorous approach would allow restoring service much faster than randomly guessing about what could be wrong and applying (costly) corrective action to see if it helps. Besides that, in many cases (including this one), you cannot randomly guess the appropriate corrective action without finding the root cause.
IgorPartola 14y ago
The pink sombrero could have saved HN: http://www.bnj.com/cowboy-coding-pink-sombrero/ http://www.bnj.com/cowboy-coding-pink-sombrero/
deleted 14y ago
[deleted]
harrisreynolds 14y ago
Just about anyone that has programmed for any length of time has done something like this. It is one of those "fixes" that after it's actually fixed you try to never think of it again. Good to know PG is mortal. :-)
Uchikoma 14y ago
Appreciating the details. "Hacker News was down all last night." With the internet there is no "last night" ;-) Europe - and more so Asia I assume - had to live for many working hours without HN.
- pramodliv1 14y ago
  Yeah, I was more productive yesterday. But I did read google cached versions of HN.
znowi 14y ago
I wonder what exactly did distract you :) When I do surgery on a production server, I triple-check making sure everything works properly. I have two assumptions: 1. HN has a low priority in the overall scheme of things, 2. Self-confidence overflow :)
DocG 14y ago
I think we have a new king! Awesome explanation.
bobsoap 14y ago
After breaking many things myself due to similar, seemingly miniscule edits, I have implemented an ABC routine: Always Be Checking. Even if it was "just" something like moving a piece of code or something equally tiny, I always check after the fix. So far, it has been working great.
ibudiallo 14y ago
When hacker news is down, I finally lifted my head and realized that there is life beyond the screen on my phone. Now that it's back. I realized that it's finally time to create an account :)
w_t_payne 14y ago
That sort of thing is fine for a startup in it's first year or two of life, but HN has been around for a while now ... surely you must have some sort of process by now?
joeycfan 14y ago
Dude - your career will never go anywhere what with your fessing up to stuff like that. Let me fix that for you. --- Why HN was down Yesterday, while toiling long into the night to make Hacker News better for you, our readers, I was assaulted by Ninjas. While defending myself with the swift yet lethal Kung Fu that I picked up during my spell with the SAS, I was distracted long enough for them to sabotage the HN server. Mission accomplished, they then immediately withdrew, the wounded dragging the dead with them. After Herculean efforts, HN was brought back online a few hours ago, after a prolonged outage. Rest assure, our readers, that every precaution will be taken to make sure this doesn't happen again.
meshko 14y ago
TIL there are still large web sites out there that do not have staging environment.
dennisgorelik 14y ago
Did you add code that detects very deep nesting levels (e.g. depth more than 100) and throws meaningful exception to help developers to diagnose the problem?
richforrester 14y ago
Cheers for that pg - now I have to explain to my boss why I was actually productive yesterday.
corwinstephen 14y ago
It's never what you think it is. One time, I had a memory leak in a Rails app that took me TWO WEEKS to find. In the end, it came down to me putting a line of config code in the wrong section of the config file, which for some reason created a recursive loop and caused my servers to crash about once every 30 minutes. #weak
Nux 14y ago
I was almost sure it was Anonymous! ... Are you in Anonymous, pg? :D
deleted 14y ago
[deleted]
sideproject 14y ago
Thank goodness it's back. I lost the my meaning of existence for the entire day. I don't know where my yesterday went. I'm ok now. :)
dylangs1030 14y ago
Thanks for the explanation pg. As you said in the original thread, "you know how these things go..."
raheemm 14y ago
I'm curious how was RTM able to notice that the problem seemed related to a specific item id? It would be great if he might write a short blurb similar to yours. Which also makes me wonder, why does RTM not write much?
Posibyte 14y ago
I absolutely love post-mortems like this. It clearly identified that there was a problem, what the author tried to do to fix it, and if it was successful. Even if it ends with the author not knowing too much about the solution that was used, it's still so interesting to see the workflows and be able to derive something from it. It's also why I like to read pg's articles so much. They're so in-depth and detailed and it doesn't feel you left thinking something was left out for the sake of being hidden.
blantonl 14y ago
Sorry about that. No worries. So, are we back on the new server? Or was this too much for one transition :)
arundelo 14y ago
Even Homer nods!
DrJosiah 14y ago
Everyone fat-fingers a database at some point... Then you build interfaces so that you can't make the same mistake.
keikun17 14y ago
i hope that user wasn't me. i was editing a typo in comment right when it happened
rnadna 14y ago
I fall into a similar misdirected-focus trap, but mine is simpler: I waste an embarrassing amount of time in editing the wrong damned file. After a sequence of small tweaks that yield no change in the results, I make a huge change and see nothing, and then realize that I've done it yet again. I need to write a vim macro that blanks the screen every few minutes, displaying the message "are you SURE this is the right file?"
calinet6 14y ago
Ok, I'll just say it: that's just plain dumb. It's a rare case, but a simple condition would have checked and prevented this. :)
dahumpty 14y ago
pg, Just wondering as to why HN isn't hosted in the cloud? (e.g. on AWS, Rackspace etc.). How do you backup all the data?
- wtracy 14y ago
  Because that would cost more? I don't really know what the benefit of cloud hosting would be in this case.
mempko 14y ago
Did you hear about the tortious and the hare?
rjempson 14y ago
That is why some organizations don't allow adhoc data fixes to be run in production. Best practice is to backup the database, run the fix against the backup, test the fix against the backup, and all being well run the fix against production.
patrickwiseman 14y ago
Don't worry I just figured out the totally bone-headed programming mistake I made at noon today. Time is a good mediator between skill and stress.
nigo 14y ago
I appreciate pg's frankness here.
johnobrien1010 14y ago
Thanks for fixing it. Have you considered avoiding dipping into the repl to do these kind of fixes? You don't owe any of us any sort of uptime guarantee, and you're a much better programmer than I, but it strikes me as odd that you would hack against the live server instead of create some tool that would make it so you couldn't take down the whole site when making this kind of fix...
infoseckid 14y ago
"I don't know if Nick succeeded, because in the middle of this I gave up and went to bed." - Not a good example to your holding companies :) What would happend if they all went to bed when something goes wrong :) Just kidding.
deleted 14y ago
[deleted]
- brokenparser 14y ago
  Forgot your medicine, today?
deleted 14y ago
[deleted]
nournia 14y ago
It seems that in your new server and also latest pushed code, I can't do `like` anything. Honestly it's not a new bug and I got used to that, don't think about that.
KevinWaller 14y ago
I'm just happy you got it fixed! Thanks!
afshinmeh 14y ago
Same problem in Iran, I couldn't access to HN all last day.
robomartin 14y ago
Great story! Yup, this kind of thing happens. For some reason it reminded me of something that happened to me as a newbie engineer. It was really funny a week later. I was troubleshooting an intermittent problem in a piece of equipment. It had several boards full of mostly LS TTL logic chips (yes, them chips). It was the kind of problem that only happened once every other day or two. Nobody knew. So, I had all kinds of instruments attached to this thing and was watching it like a hawk waiting for a failure. It had probes attached to every point in the circuit where I suspected I could see something and learn about the source of the problem. I also tested for thermal issues with heat guns and freeze sprays, familiar troubleshooting techniques to anyone who's done this kind of thing. Anyhow, every so often the thing would go nuts. The three scopes I had connected to it showed things I simply didn't understand. I'd analyze but couldn't make any sense out of it. Still, again, every so many days it would happen again. Changed power supplies and the usual suspects. No difference. Well, finally, two weeks later, the other engineers in the office took pity on me and told me what was going on: They had connected a VARIAC to the power strip I was using to power the UUT (unit under test). The scopes and other test instruments remained on clean power. Every so often they'd reach into this drawer where the VARIAC was hidden and lower my power strip's voltage just enough for the power supplies to fall out of regulation and everything start beeping and sputtering. Those friggin SOB's. They had me going for days! I was pissed beyond recognition. Of course, after a while I was laughing my ass off alongside them. Good joke. Cruel, but good. My revenge: A CO2 fire extinguisher rigged to go off into his crotch when my buddy sat down to work. Fun place to work. We did this kind of stuff all the time. Today I'd be afraid of getting sued. People have really thin skins these days.
pilas2000 14y ago
That's funny because one of the top posts in progit yesterday was about the Hare and Tortoise algorithm
Jplenbrook 14y ago
Why does PG maintain the website himself? I would think he would have many better things to do with his time.
cincinnatus 14y ago
The cobbler's children have no shoes :-)
cranklin 14y ago
You are honest and I respect that. I'm sure many companies try to play off their downtime as something far more sophisticated when in fact, it was something too embarrassing to admit. I've certainly had my fair share of embarrassingly stupid mistakes that resulted in downtime.
thedaveoflife 14y ago
I think this demonstrates how many people browse the /threads?id=pg page (myself included)
GnarfGnarf 14y ago
That's funny -- I work in genealogy software, and loops ("being your own grandpa") happen all the time, due to data entry errors. To avoid infinite recursion, we always keep track of what records we've processed already, check whether "I've been there before", and bail out if the answer is affirmative.
bestest 14y ago
So, uh, still fixing stuff in production?
deleted 14y ago
[deleted]

Why HN was down

1049 points by pg 14y ago

bramcohen 14y ago

You should probably make your code robust to this sort of data corruption in the future.

birken 14y ago

Do you have munin monitoring on the production HN server? That would really make situations like this easier to debug. First, it can pinpoint exactly when something started happening, which in this situation might have helped you realize the problem was caused by your change. Secondly, in this specific situation it probably would have been easier to differentiate a situation where you are running low on memory vs this completely different situation. As somebody who spent a lot of time professionally debugging large software systems when they were misbehaving (as a Google SRE), I can tell you that looking at graphs of many key metrics (disk IO, CPU, memory, then application specific things) was always the place to start when debugging a situation, because you can learn so many things right away. When did it start? Was it a slow buildup or an immediate thing? What is the general problem (Memory?, Disk IO?, CPU?, none of the above?)? Has a similar pattern happened in the past? Then you can start to get fancy and plot things like "messages/minute" or something and then it becomes easy to see when issues are affecting the site performance and when they aren't.

stcredzero 14y ago
That and something like the Smalltalk Change Log would have made this a no-brainer debug. (Yes, every REPL action in Smalltalk got logged by the same mechanism that logged every code change.) Such mechanisms aren't trivial, but they're not rocket science either, and they have tremendous ROI.

youngerdryas 14y ago

>On a comment thread, a new user had posted some replies as siblings instead of children. I posted a comment explaining how HN worked. But then I decided to just fix it for him by doing some surgery in the repl. No good deed goes unpunished! People sometimes reply as sibling because they too impatient wait for the built-in delay on child comments. Thanks for keeping the experiment going.

lifeisstillgood 14y ago

There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to. I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it." I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing. An awful lot of sites at 1.2m views would have outsourced the running and development of the whole thing - there are entreprenuers who say its not even worth our time to code up the MVP. I find this approach sensible from a business point of view, but still it does not sit right with me. I am supposed to have a nice website with lots of good content to attract inbound marketing - so I tried getting someone on textbroker to write an article for me. It read like a High School essay - no life, no anime. And so I will probably write my own CMS and my own content. And pg sits there and writes his site in his own language, with his own moderation tools. Apart from the hilarious idea he could find a ten person ruby shop to outsource to, its nice to see someone taking the time to play again. Its why I like to see jgc on here too. I am not entirely sure those thoughts are joined up (I am procrasting like crazy) but if they come to mean anything its we are playing in pg's sandbox. If the sand leaks it's his sand, and the only company this is mission critical to is YC.

stcredzero 14y ago
> It read like a High School essay - no life, no anime. ... I am not entirely sure those thoughts are joined up (I am procrasting like crazy) Your procrasting like crazy has much anime.
luser001 14y ago
> I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing. IIRC, sudo logs all commands to syslog. Which might come in handy. Yes, root commands will be logged by bash to .bash_history, but there are limits of # of commands lots, what happens if you are logged in multiple times into same account etc. Anyway, that's why I like sudo.
- 14y ago

user911302966 14y ago

This, folks, is why NOSQL/new school/whizbang/* will never replace a properly designed RDBMS. Period.

neurotech1 14y ago

This should serve as a example template for how to accurately and transparently explain to users what went wrong. No deflecting blame, no useless platitudes. Credit to PG, RTM and the rest of the team for keeping the sites uptime as high at it is.

jgrahamc 14y ago
"No deflecting blame" Who were they going to blame?
- wpietri 14y ago
  He could have blamed the new server. Or whatever distracted him. Or the user, for being dumb. I've seen people do all of those. Or he could have just dodged the blame entirely.
  - dguaraglia 14y ago
    Some people are just freaking difficult to work with. I've worked with people that wouldn't accept responsibility even after every other possible cause was ruled out. I've even gotten this reply: "well, you must have been unlucky to get the faulty e-mail, because it seems to work most of the time". Yeah, because that's how programming works: cowboy coding and hoping for the best. This guy actually called bugfixes "optimizations": "hey, X the feedback widget on the front page isn't working", "oh, yeah, I haven't worked on that because that's code that needs to be 'optimized', so it's low on my issues list". Ugh. I've learned my lesson now. In fact, that's an incredible lesson for a startup founder: never, ever, hire someone who dodges a question on an interview. And the first time they avoid taking responsibility for something that was clearly their fault, fire them. The last thing you want is someone who'll blame everyone and anything else for their issues. It's a great way to kill morale and create rifts in a small team.
    - artursapek 14y ago
      pg has an essay where he says the smartest people he knows are always willing to take blame or admit they don't know the answer to a question.

DanI-S 14y ago

n.b. that this is why time travel is a terrible idea.

cmaggard 14y ago
The grandfather paradox is the best solution to the halting problem.
w_t_payne 14y ago
Because the computers that run the Matrix will get overloaded?

benatkin 14y ago

So what do you do to avoid this in the future? Do you stop doing surgery in the repl, or do you do the surgery with functions that check for cycles from now on?

badgar 14y ago
> So what do you do to avoid this in the future? It's HN... there's no SLA, there's no postmortems, there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all.
- benatkin 14y ago
  > It's HN... there's no SLA, there's no postmortems I didn't mean to imply that there were. I was just curious. > there's no doing things better in the future. pg just runs this site out of the good of his heart, we should be lucky the volunteers run it for us at all. Since there are multiple volunteers, I think that the site always feels important to at least one of them. I imagine that some of them have gone more than a week without giving a shit about HN, but not all of them at once. So I think there is doing things better in the future. In fact, HN keeps getting improvements behind the scenes, to keep it running, keep it interesting, and keep it from getting overrun with trolls.
- martinced 14y ago
  I don't think parent did mean that he had a "right" to expect some level of quality or anything. It's just that we, as programmers, tend to take measures so that silly bugs do not happen anymore or that, at least, we leave big clues as to what went wrong. In a project I had a similar issue: I was wrapping lists inside immutable lists but, due to a silly bug, I kept wrapping immutable lists inside immutable lists at every save made. So saved files would grow bigger and bigger. And I did fix the bug and also added a big fat warning logs in case too many nested lists were detected. pg might just as well have now added something preventing infinite recursion inside the comment tree or some WARN logging telling when a generate page is getting too big, etc. I'd still find it very interesting to know what pg did, if any, to dodge / minimize / make it easier to determine if such an issue happens in the future.

nowarninglabel 14y ago

Happens to a lot of us. Great reason to always write tested cleanup scripts for this stuff instead of editing directly on the server. The only time I brought down my product last year was from a similar screwup, I was removing users by hand and somehow managed to end up with a 0 in my list of user ids, thus deleting the anonymous user, and causing havoc to my server, which took a long time to track down.

cool-RR 14y ago

Great debugging story! I guess the lesson is to have code that alerts you about comment loops without going into an infinite loop. Also another lesson would be to figure out a way to have better clarity into which requests are causing a timeout on the server.

deleted 14y ago

[deleted]

DanielBMarkham 14y ago

Amazing that such a large percentage of debugging involves determining exactly what you are debugging. The definition of the problem, many times, is the solution. Might be a good time to mention Rubber Duck Debuggging. http://en.wikipedia.org/wiki/Rubber_duck_debugging http://en.wikipedia.org/wiki/Rubber_duck_debugging

davesims 14y ago
A few times a month, I'll look up at one of my colleagues and say, "hey, got a sec? I need to talk to the duck," and they know this means I'm going to talk to their head but they can basically keep doing what they're doing and nod occasionally. This serves several purposes: (1) It's less insane-sounding than actually talking to an inanimate object in an open work environment. (2) It actually feels better and forces me to think more clearly when I'm talking to an actual person -- the cognitive focus is higher when the object of conversation can actually, in theory, think and talk back (YMMV). (3) And finally, although it does require some focus on the part of the other coder, it's not nearly as taxing to them as actually helping me solve the problem or pairing up with me. So it's a good compromise somewhere between pair programming and talking to an actual rubber duck. Again, YMMV. Maybe I'll call it "Pair Ducking."
- kamjam 14y ago
  This is so true, probably about 90% of the time my colleagues call me over about a problem they are facing, explain in detail what the problem is and then eureka! Most of the time it would actually take me much longer to figure out the exact issue since I don't know the ins/outs and subtleties of the code but it's exactly as you say. Of course, it makes me look really good cos I just "helped" them solve their issue :)
  - cientifico 14y ago
    Maybe because the solution is in asking the good questions.
    - 14y ago

lucb1e 14y ago

Are you saying you manually modify the database? Like, shifting around things by id instead of just making admin buttons next to posts?

martinced 14y ago
"Are you saying you manually modify the database?" Oh manually modifying production database on the fly ain't unheard of. However it's still not "very Chuck Norris" on a scale of Chuck Norrisness compared to the modification of a running app directly in the REPL. I mean: it doesn't matter if you manually modify the DB itself or not when you directly modify the app from the REPL itself (the app being anyway "in charge" of the DB). Sure, modifying manually the production DB might be an issue to some. But I can guarantee you that it's the last of your worries when you're actually modifying production code directly from the REPL ; )
RKoutnik 14y ago
Sometimes that's easier (albeit more dangerous, as we just saw).
sgt 14y ago
I think I get what you're hinting at. Ok, so this is Hacker News, it's in the name, and most of us are aware that HN is also a research/hobby project. It's not made to be an rock-stable enterprise system doing bank transactions or what not, so I think what pg did was prefectly excusable. People make mistakes. Nobody will die without HN for a day or two, and it won't affect the site's popularity one bit.
- lucb1e 14y ago
  No that's right, but I worry about apache2 being down for potentially one or two users or bots that visit/crawl my website during a one-minute reboot. Meanwhile the big boys are down for 16 hours because they do things that any other person would have gotten a decent scolding for. Just look at the points per hour this thread is getting, if I had posted this about my website on my website people would have said I was stupid. You are right though, making mistakes is human as they say, and nobody dies because of this. In fact, less popularity might be good for the site's content quality. I'm just surprised by how much they care about thousands of hourly users, that what I would dream of having.

sgt 14y ago

Much appreciated, pg. I knew that the "10 minutes of downtime" would not occur (fair enough, this was not related to the server upgrade).

RKoutnik 14y ago

It's nice to know that even the mightiest of us can still make mistakes. Thanks for being willing to admit mistakes so the rest of us can learn.

luser001 14y ago

I use assertions to protect against things like this. I liberally sprinkle my code with assertions (CS theory calls them pre-conditions and post-conditions, iirc) to crash early if the system is an invalid state. One my pet peeves is that few programmers seem to love assertions like I do. Would love to see to comments on this.

swah 14y ago
The kind of assertion he needed though, could only be ensured by the database, not application code (my impression).
- luser001 14y ago
  Agreed, infinite loops are a little hard to protect using asserts. When I hit the first infinite loop bug on a code path, I frequently add code to assert that the number of calls is less than $A_LARGE_NUMBER to catch future occurrences of the same root cause.
  - badgar 14y ago
    > Agreed, infinite loops are a little hard to protect using asserts. assert(is_tree(comment_graph)) Typically, a composite entity (like an "item" on HN which has many "comments") will define invariants to ensure data integrity. In this case, the invariant is that an "item"'s comments form a tree. The database layer often contains this logic, but it depends on how you're building your application; NoSQL backends for example typically must put validation in the application layer. Since HN just uses files, a well-developed application layer should be riddled with invariants like this.
  - mpweiher 14y ago
    I dimly remember a language that just hard-limited loops. I thought it was John Pane's HANDS system, but I can't seem to find a reference in the thesis...can anybody refresh my memory? http://www.cs.cmu.edu/~pane/research.html http://www.cs.cmu.edu/~pane/research.html http://www.cs.cmu.edu/~pane/thesis/ http://www.cs.cmu.edu/~pane/thesis/ Pretty cool work regardless, I really like the way it deals with aggregates, for example.

dasil003 14y ago

I'm not sure whether it's terrifying or relieving to realize that if all I dream of comes to pass and I achieve something akin to the legendary status of pg in the hacker community that I will still be susceptible to the inevitable facepalm moments that come with direct database access. In any case I am thankful for the detailed explanation.

hnriot 14y ago
he's not legendary for his IT skills.
- dasil003 14y ago
  Low blow.
- jbverschoor 14y ago
  Now he is ;)
- mark-r 14y ago
  It was his IT skills that got him the big sale to Yahoo that got him the bucks to start YC. Not sure where this comment came from.
  - papsosouid 14y ago
    You really think yahoo bought viaweb because of pg's legendary ability to reboot servers and type "./configure && make && sudo make install"?
larrys 14y ago
Some of the most spectacular airplane crashes are by the most experienced pilots. If you've ever tried something new as a hobby you tend to be very careful. Once you gain confidence you take more chances and don't do what even a beginner might do.
- stcredzero 14y ago

hnriot 14y ago

it's a good job it's your site, this type of thing is often what gets someone fired in a company. Modifying (meddling!) the production system directly.

lucb1e 14y ago
Upvoted. This is what I wanted to reply, but then thought better of it and moderated my response.
- irahul 14y ago
  I think you people already know the answer. The amount of freedom and stake/reward system for pg is different from yours. I can't speak for pg, but personally, I am not going to write a migration to re-parent a single thread if the site in question is my side project, doesn't bring revenue, has some intangible benefits, but not so much that warrant putting much labor into it. Either it would be `thread.parent = new_paret_id`; or if it occurred to me that it might introduce a loop, changing `parent=` to take loops into account followed by `thread.parent = new_parent_id`. What were you expecting? A bug tracker discussion, code commit, review, change request and deployment?
  - hnriot 14y ago
    no, you just leave it alone, it would have sorted itself out if nothing had been done at all. failing that you put a cap in the code that generates the page. Simple stuff.
  - dap 14y ago
    The problem is thinking of it as the cost of implementing the feature vs. doing manual surgery on the production database, without realizing that if you choose the latter, you're also choosing the risk that you'll spend hours debugging the system when the surgery goes wrong. It's a tradeoff, to be sure, but it's not clear that the latter is cheaper on expectation.
JohnBooty 14y ago
I disagree. Very few companies would think negatively of an engineer if they made such a mistake on a non-essential, non-revenue-generating fun/research project. How many dollars did YC lose because of the outage? None. (Maybe they saved a few on bandwidth!) I also predict that exactly zero startups will say, "Man... I'm not going to take seed money from those guys! They had discussion forum downtime."

gruseom 14y ago

This is a particularly endearing piece of "hacker news". It's so easy to relate to.

jack57 14y ago

Are you sure that comment's name wasn't Phillip J Fry?

jack57 14y ago
Apologies for the trivial comment

Legion 14y ago

"We'll do it live!"

eytanlevit 14y ago
epic
jbuzbee 14y ago
"Hey, Hold my beer and watch this!"

T-zex 14y ago

Thank you for the honest explanation. This is not so easy especially for a famous person.

irahul 14y ago

Disclaimer: Hindsight is 20/20, and stuff. If reverting code didn't fix it, reverting server didn't fix it, incorrect data is the most likely culprit(I am not claiming this should have outright occurred to you; just thinking out loud). I take it you introduced non terminating recursion by making a thread its own parent, and you made the change on disk. But this analysis is the last thing that comes to mind when you already have introduced 2 new variables the same day - new code, new server. And an old, recurring variable(GCing too much) is in play as well.

hyh1048576 14y ago

The "More" button at the bottom is still not clickable sometimes. Get a "Unknown or expired link." message.

andreasklinger 14y ago

I appreciate (if not love) the fact that you bugfix and server-change yourself. True hacker spirit.

hyh1048576 14y ago

The "More" button at the bottom is still not clickable sometimes. Get a "Unknown or expired link." message.

xentronium 14y ago

Whoa, what an unfortunate coincidence. This whole bug would be so much easier to find, if it weren't for the new server.

lmm 14y ago
The bugs that actually hit production are always like this - a confluence of three or so factors - because if it were simpler you'd have caught it earlier. (Though I have to say, upgrading the code at the same time as you're restarting to fix a problem is really a rookie mistake. It's incredibly tempting because it saves so much time, but if you do it you will get it wrong sooner or later. One of the hardest skills in programming is acquiring that zen that you need to wait in a state of readiness for the effects of your first change to make themselves apparent, rather than changing something else)

naturalethic 14y ago

If the problem existed before the code update, why would you assume it was the code update that caused the problem?

larrys 14y ago

"But then I decided to just fix it for him by doing some surgery in the repl." I've always found it's a good idea to not deviate. Whether it be running, parking or anything else once you deviate from some regular behavior you run into potential problems that you hadn't anticipated. "For some reason I didn't check the comments after the surgery to see if they were in the right place. " More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur.

irahul 14y ago
> I've always found it's a good idea to not deviate. Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him? > More or less my point. If this wasn't a deviation from normal behavior you would have "checked the comments after the surgery" because it would have either become habit or the shear number of times you tried a fix resulting in an error would have made that more likely to occur. How about the opposite scenario? He has done it so many times with desired results, that he didn't bother checking?
- badgar 14y ago
  > Aren't you assuming "surgery in repl" is a deviation? What if it's normal course of action for him? This is a big difference between engineering and hacking. An engineer would never regularly do something so dangerous. But I suspect pg isn't an engineer when he works on HN, I suspect he is a hacker, and just does whatever he wants to, whenever he wants. Which is his prerogative.
prawks 14y ago
> I've always found it's a good idea to not deviate. > you run into potential problems that you hadn't anticipated. The second statement is no reason to live by the first. In fact, I think you'd be doing yourself a disservice by staying so comfortable. Being comfortable with the unanticipated, however, is a powerful quality to have.

ricardobeat 14y ago

Related question: what is the timing for the 'Reply' link to show up? I might be fantasizing but sometimes it takes 5, sometimes 10 minutes to appear, leading people to reply as a sibling instead.

DanBC 14y ago
Deeply nested comments tend to be hot, and so the reply link takes a while to show up to try to give people some time to think about what they're going to say.
- ricardobeat 14y ago
  Interesting. So it slows down discussion as it progresses, until it either stops, is forgotten or becomes a series of long essays.
beagle3 14y ago
The deeper it is, the longer it takes to appear (I think at depth 7 or 8 it starts counting in hours, and at some point it just won't appear at all). Just like too many nested if(x) { if (y) { if (z) ... }} constructs, too deep a discussion nesting is also unreadable.

scotthtaylor 14y ago

Normality has returned :-)

btilly 14y ago

That explains something weird I saw. If I went to Google's cached copy, I could see threads, and then click on them. But the front page was down. But I could see individual threads. Very confusing.

wpeterson 14y ago

I guess it's time for NewRelic to add an Arc agent.

campnic 14y ago

The nice thing about surgery with a computer program on a server is that death is not permanent.

scotthtaylor 14y ago

PG, quick question: Did this impact the server hosting the YC Summer 2013 applications? When I tried to edit mine, it simply said "Thanks, scotthtaylor"

scotthtaylor 14y ago
Working again now.

orangethirty 14y ago

It makes me feel good knowing better programmers than me go through the same issues I face. :)

thunderrabbit 14y ago
word

louischatriot 14y ago

Funny to see that this happens to everyone. A week ago, while testing some stuff to locate a low-importance bug, I erased the whole user database. Fortunately we have a good restore so the problem was solved in a few minutes, but still, cold sweat here ...

neilxdsouza 14y ago

Isn't it curious that the comet incident over Russia happened so close to the pass of DA 14. In the intro to the book: http://ruby.bastardsbook.com/about/#why http://ruby.bastardsbook.com/about/#why is the note about surgical instruments left inside. It seems just like a coincidence that this happened so close to the switch to the new server, but I wonder if it's something deeper in the subconscious mind; the change to the new server is quite a big change (I know I feel that way when I have purchased a new computer (it feels different - even if it's running the same linux as before)) and could have upset the normal checks one has in place when tweaking things.

d0m 14y ago

Hacking code in the repl without testing the new behavior. We all did that. Don't lie. Once I wanted to quick fix a "gmail.ca" to "gmail.com", which I did.. but to all the users instead of just the one mistaken. Fortunately I realized by mistake really fast ;-)

deleted 14y ago
[deleted]

aaronh 14y ago

My pet peeve: You made an arbitrary change while debugging a problem. NOW YOU HAVE N^2 PROBLEMS!

sehugg 14y ago

Great postmortem and good lessons to learn here: * Don't manually modify database without a well-tested procedure and another pair of eyes * Don't leave persistent problems (e.g. memory problems) uninvestigated so that you miss new problems with similar symptoms * Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) I'm pretty sure I've repeated this exact same sequence before with similar results...

scotty79 14y ago
* while you are displaying a tree keep track of the items you already displayed so you can detect a cycle
- sukuriant 14y ago
  I think the assumption there was that it was safe, since the code disallowed this from happening, naturally.
  - scotty79 14y ago
    Assumption is the mother of all screw ups. Even if you think that the code that creates and modifies your data will not put it in some undesired state, the code that uses this data should assume that the data may be in all undesired states you can dream up and should do its best not to do something seriously bad when that happens (like landing in endless loop/recursion or executing possibly user provided strings).
JonM 14y ago
* Don't push new code to production while operational problem is ongoing (unless it addresses the operational problem) ^^ absolutely!
zzzeek 14y ago
use CHECK constraints to prevent invalid data patterns when possible.

eluos 14y ago

"I am my own grandpa"

tolmasky 14y ago

Why do "self posts" like this show up in the same light gray as posts with negative vote counts? My eyes aren't great and I find it hard to read

roryokane 14y ago
I don’t know pg’s reasons for making self-posts light gray, but you can fix problems like that with the bookmarklet Zap Colors: https://www.squarefree.com/bookmarklets/zap.html https://www.squarefree.com/bookmarklets/zap.html
hayksaakian 14y ago
Maybe post color is based on some get_text_post_color method that applies to self posts and comments, where the color depends on comment karma. Given that self posts like the OP are votable as if they were a normal link post, their comment karma value is probably 0.
emillon 14y ago
The rationale for this is that if you need to post a long text post, it should be in the form of a blog post instead. I agree with you that it's not really adapted for a meta post.
vl 14y ago
I'm using "Hacker News Enhancement Suite" Chrome extension - it fixes multiple problems, including this one.

fnordfnordfnord 14y ago

>>I caused an item to be its own grandchild. Please forgive me. I know you folks tend to hate jokes on here. Don't waste your time if you're immune to corny humor. "I'm My Own Grandpa- Ray Stevens" ( with family tree diagram) http://www.youtube.com/watch?v=eYlJH81dSiw http://www.youtube.com/watch?v=eYlJH81dSiw

carpathios 14y ago

Seems related: http://meta.stackoverflow.com/questions/66377/what-is-the-xy-problem http://meta.stackoverflow.com/questions/66377/what-is-the-xy...

mikedmiked 14y ago

> created a loop in the comment tree; I caused an item to be its own grandchild. Ah, the online forum equivalent of going back in time to kill your grandfather.

dap 14y ago

Thanks for the detailed explanation. It sounds like everything was done to fix the problem except try to figure out what the problem actually was. Why not use tools to see what the program is doing, form a hypothesis, gather data to confirm or reject the hypothesis, repeat until cause found, and then take corrective action that by this point you have high confidence will work? I realize HN is more of a side project than a production service, but the goal is the same in both cases: to restore service quickly so you can move on to other things. It feels like a more rigorous approach would allow restoring service much faster than randomly guessing about what could be wrong and applying (costly) corrective action to see if it helps. Besides that, in many cases (including this one), you cannot randomly guess the appropriate corrective action without finding the root cause.

IgorPartola 14y ago

The pink sombrero could have saved HN: http://www.bnj.com/cowboy-coding-pink-sombrero/ http://www.bnj.com/cowboy-coding-pink-sombrero/

deleted 14y ago

[deleted]

harrisreynolds 14y ago

Just about anyone that has programmed for any length of time has done something like this. It is one of those "fixes" that after it's actually fixed you try to never think of it again. Good to know PG is mortal. :-)

Uchikoma 14y ago

Appreciating the details. "Hacker News was down all last night." With the internet there is no "last night" ;-) Europe - and more so Asia I assume - had to live for many working hours without HN.

pramodliv1 14y ago
Yeah, I was more productive yesterday. But I did read google cached versions of HN.

znowi 14y ago

I wonder what exactly did distract you :) When I do surgery on a production server, I triple-check making sure everything works properly. I have two assumptions: 1. HN has a low priority in the overall scheme of things, 2. Self-confidence overflow :)

DocG 14y ago

I think we have a new king! Awesome explanation.

bobsoap 14y ago

After breaking many things myself due to similar, seemingly miniscule edits, I have implemented an ABC routine: Always Be Checking. Even if it was "just" something like moving a piece of code or something equally tiny, I always check after the fix. So far, it has been working great.

ibudiallo 14y ago

When hacker news is down, I finally lifted my head and realized that there is life beyond the screen on my phone. Now that it's back. I realized that it's finally time to create an account :)

w_t_payne 14y ago

That sort of thing is fine for a startup in it's first year or two of life, but HN has been around for a while now ... surely you must have some sort of process by now?

joeycfan 14y ago

Dude - your career will never go anywhere what with your fessing up to stuff like that. Let me fix that for you. --- Why HN was down Yesterday, while toiling long into the night to make Hacker News better for you, our readers, I was assaulted by Ninjas. While defending myself with the swift yet lethal Kung Fu that I picked up during my spell with the SAS, I was distracted long enough for them to sabotage the HN server. Mission accomplished, they then immediately withdrew, the wounded dragging the dead with them. After Herculean efforts, HN was brought back online a few hours ago, after a prolonged outage. Rest assure, our readers, that every precaution will be taken to make sure this doesn't happen again.

meshko 14y ago

TIL there are still large web sites out there that do not have staging environment.

dennisgorelik 14y ago

Did you add code that detects very deep nesting levels (e.g. depth more than 100) and throws meaningful exception to help developers to diagnose the problem?

richforrester 14y ago

Cheers for that pg - now I have to explain to my boss why I was actually productive yesterday.

corwinstephen 14y ago

It's never what you think it is. One time, I had a memory leak in a Rails app that took me TWO WEEKS to find. In the end, it came down to me putting a line of config code in the wrong section of the config file, which for some reason created a recursive loop and caused my servers to crash about once every 30 minutes. #weak

Nux 14y ago

I was almost sure it was Anonymous! ... Are you in Anonymous, pg? :D

deleted 14y ago

[deleted]

sideproject 14y ago

Thank goodness it's back. I lost the my meaning of existence for the entire day. I don't know where my yesterday went. I'm ok now. :)

dylangs1030 14y ago

Thanks for the explanation pg. As you said in the original thread, "you know how these things go..."

raheemm 14y ago

I'm curious how was RTM able to notice that the problem seemed related to a specific item id? It would be great if he might write a short blurb similar to yours. Which also makes me wonder, why does RTM not write much?

Posibyte 14y ago

I absolutely love post-mortems like this. It clearly identified that there was a problem, what the author tried to do to fix it, and if it was successful. Even if it ends with the author not knowing too much about the solution that was used, it's still so interesting to see the workflows and be able to derive something from it. It's also why I like to read pg's articles so much. They're so in-depth and detailed and it doesn't feel you left thinking something was left out for the sake of being hidden.

blantonl 14y ago

Sorry about that. No worries. So, are we back on the new server? Or was this too much for one transition :)

arundelo 14y ago

Even Homer nods!

DrJosiah 14y ago

Everyone fat-fingers a database at some point... Then you build interfaces so that you can't make the same mistake.

keikun17 14y ago

i hope that user wasn't me. i was editing a typo in comment right when it happened

rnadna 14y ago

I fall into a similar misdirected-focus trap, but mine is simpler: I waste an embarrassing amount of time in editing the wrong damned file. After a sequence of small tweaks that yield no change in the results, I make a huge change and see nothing, and then realize that I've done it yet again. I need to write a vim macro that blanks the screen every few minutes, displaying the message "are you SURE this is the right file?"

calinet6 14y ago

Ok, I'll just say it: that's just plain dumb. It's a rare case, but a simple condition would have checked and prevented this. :)

dahumpty 14y ago

pg, Just wondering as to why HN isn't hosted in the cloud? (e.g. on AWS, Rackspace etc.). How do you backup all the data?

wtracy 14y ago
Because that would cost more? I don't really know what the benefit of cloud hosting would be in this case.

mempko 14y ago

Did you hear about the tortious and the hare?

rjempson 14y ago

That is why some organizations don't allow adhoc data fixes to be run in production. Best practice is to backup the database, run the fix against the backup, test the fix against the backup, and all being well run the fix against production.

patrickwiseman 14y ago

Don't worry I just figured out the totally bone-headed programming mistake I made at noon today. Time is a good mediator between skill and stress.

nigo 14y ago

I appreciate pg's frankness here.

johnobrien1010 14y ago

Thanks for fixing it. Have you considered avoiding dipping into the repl to do these kind of fixes? You don't owe any of us any sort of uptime guarantee, and you're a much better programmer than I, but it strikes me as odd that you would hack against the live server instead of create some tool that would make it so you couldn't take down the whole site when making this kind of fix...

infoseckid 14y ago

"I don't know if Nick succeeded, because in the middle of this I gave up and went to bed." - Not a good example to your holding companies :) What would happend if they all went to bed when something goes wrong :) Just kidding.

deleted 14y ago

[deleted]

brokenparser 14y ago
Forgot your medicine, today?

deleted 14y ago

[deleted]

nournia 14y ago

It seems that in your new server and also latest pushed code, I can't do `like` anything. Honestly it's not a new bug and I got used to that, don't think about that.

KevinWaller 14y ago

I'm just happy you got it fixed! Thanks!

afshinmeh 14y ago

Same problem in Iran, I couldn't access to HN all last day.

robomartin 14y ago

Great story! Yup, this kind of thing happens. For some reason it reminded me of something that happened to me as a newbie engineer. It was really funny a week later. I was troubleshooting an intermittent problem in a piece of equipment. It had several boards full of mostly LS TTL logic chips (yes, them chips). It was the kind of problem that only happened once every other day or two. Nobody knew. So, I had all kinds of instruments attached to this thing and was watching it like a hawk waiting for a failure. It had probes attached to every point in the circuit where I suspected I could see something and learn about the source of the problem. I also tested for thermal issues with heat guns and freeze sprays, familiar troubleshooting techniques to anyone who's done this kind of thing. Anyhow, every so often the thing would go nuts. The three scopes I had connected to it showed things I simply didn't understand. I'd analyze but couldn't make any sense out of it. Still, again, every so many days it would happen again. Changed power supplies and the usual suspects. No difference. Well, finally, two weeks later, the other engineers in the office took pity on me and told me what was going on: They had connected a VARIAC to the power strip I was using to power the UUT (unit under test). The scopes and other test instruments remained on clean power. Every so often they'd reach into this drawer where the VARIAC was hidden and lower my power strip's voltage just enough for the power supplies to fall out of regulation and everything start beeping and sputtering. Those friggin SOB's. They had me going for days! I was pissed beyond recognition. Of course, after a while I was laughing my ass off alongside them. Good joke. Cruel, but good. My revenge: A CO2 fire extinguisher rigged to go off into his crotch when my buddy sat down to work. Fun place to work. We did this kind of stuff all the time. Today I'd be afraid of getting sued. People have really thin skins these days.

pilas2000 14y ago

That's funny because one of the top posts in progit yesterday was about the Hare and Tortoise algorithm

Jplenbrook 14y ago

Why does PG maintain the website himself? I would think he would have many better things to do with his time.

cincinnatus 14y ago

The cobbler's children have no shoes :-)

cranklin 14y ago

You are honest and I respect that. I'm sure many companies try to play off their downtime as something far more sophisticated when in fact, it was something too embarrassing to admit. I've certainly had my fair share of embarrassingly stupid mistakes that resulted in downtime.

thedaveoflife 14y ago

I think this demonstrates how many people browse the /threads?id=pg page (myself included)

GnarfGnarf 14y ago

That's funny -- I work in genealogy software, and loops ("being your own grandpa") happen all the time, due to data entry errors. To avoid infinite recursion, we always keep track of what records we've processed already, check whether "I've been there before", and bail out if the answer is affirmative.

bestest 14y ago

So, uh, still fixing stuff in production?

deleted 14y ago

[deleted]