7 ms·

Update: An examination of the logs shows nothing remarkable happening at the moment of the crash. We may have run out of some resource. In that case it may be

by pg 14y ago

Update: An examination of the logs shows nothing remarkable happening at the moment of the crash. We may have run out of some resource. In that case it may be hard to know for sure what happened.

nodesocket 14y ago
Maybe add some monitoring such NewRelic and ServerDensity.
johnrob 14y ago
If it helps at all, I'm pretty sure there was a period around 3:30 when http was working but https was not.
- harshreality 14y ago
  I noticed that too. I'm not sure of the time, but it was more than a momentary thing. Https was either not responding at all or it was responding only after a long wait. A bit later, https was taking 20-30 seconds before returning the page, which I think was an improvement over the earlier period where I didn't see it return a page at all. I didn't personally notice any http outage or delay.
- sp332 14y ago
  Probably related, hackerne.ws kept working for me well after news.ycombinator.com stopped, possibly because I was accessing the latter over HTTPS.
pg 14y ago
Question: Did people find HN horribly slow this afternoon? (I found it unusably slow around 2 pm pacific, but there was little I could do about it from where I was then.)
- dt7 14y ago
  I found it horribly slow from about 18 hours ago (3pm GMT) onwards. I think the last time I tried it was about 9pm GMT, and it was slow them. It took about 30-60 seconds to load a page, maybe more. Edit: this was using https, I didn't try http.
- knowtheory 14y ago
  The site had seemed slow basically all day for me, from say about 8:30-9am central time (I'd been up and checking the site since 6) up to the point when it crashed.
  - pg 14y ago
    Were you using https or http?
    - michaelcampbell 14y ago
      I was getting the proxy error message with https. I never tried http then.
    - knowtheory 14y ago
      I've been using HTTPS consistently.
- mcherm 14y ago
  Yes: unusably slow both times I tried yesterday.
- brudgers 14y ago
  I was using my phone browser when the site was down. What I noticed was that the "secure" icon was displayed when I tried to connect. However, this may have been an artifact in the address bar from the previous site which was secure - I haven't paid that much attention to how the phone's browser handles slow loads. The reason it caught my attention was due to the recent https "Ask" threads, and my first thought was perhaps full time https had been implemented. I wouldn't bring it up except that it was unusual and correlated with HN content and the outage, even though it is likely to be a coincidence.
newman314 14y ago
More instrumentation/monitoring details needed to help diagnose future occurences?
ralph 14y ago
What were the most common activities done by the IPs that were "new" in the last twelve hours? New compared to the history of IPs available. Just wondering if it was orchestrated or whether it would give a clue as to the resource that was consumed.
FredBrach 14y ago
A -vague- suggestion: did you tried to run ghost users on HN by saving some bunch of users' session on the site?... You should be able to know if HN scale and maybe replicate the crash.
larrys 14y ago
I will repeat something I said below in my comments because I'm not sure you will see it: "But there is no question that it is a good idea to get to the bottom of why this happened. I have found in the past it always pays to do a post-mortem on any system problem as many times there is something lurking that will cause that same problem in the future." [1] I can't tell you how many times over the years some little thing has been an indicator of something else going on or caused a big problem later. It's normally better to diagnose this while you are not under duress, have room to breath, and aren't under any pressure. [1] Non system related last week my wife couldn't get into the garage because the GFI on the same circuit as the garage door opener had tripped. The same GFI is in the upstairs bathroom and had blown previously many times but it didn't seem to be a problem. We didn't realize it also controlled the garage door. So she couldn't get into the house (she didn't have her key). So by ignoring the problem (just keep resetting the GFI and hey - figure it out later) we had a larger impact and issue than it seemed on the surface.

by pg 14y ago

Update: An examination of the logs shows nothing remarkable happening at the moment of the crash. We may have run out of some resource. In that case it may be hard to know for sure what happened.

nodesocket 14y ago

Maybe add some monitoring such NewRelic and ServerDensity.

johnrob 14y ago

If it helps at all, I'm pretty sure there was a period around 3:30 when http was working but https was not.

harshreality 14y ago
I noticed that too. I'm not sure of the time, but it was more than a momentary thing. Https was either not responding at all or it was responding only after a long wait. A bit later, https was taking 20-30 seconds before returning the page, which I think was an improvement over the earlier period where I didn't see it return a page at all. I didn't personally notice any http outage or delay.
sp332 14y ago
Probably related, hackerne.ws kept working for me well after news.ycombinator.com stopped, possibly because I was accessing the latter over HTTPS.

pg 14y ago

Question: Did people find HN horribly slow this afternoon? (I found it unusably slow around 2 pm pacific, but there was little I could do about it from where I was then.)

dt7 14y ago
I found it horribly slow from about 18 hours ago (3pm GMT) onwards. I think the last time I tried it was about 9pm GMT, and it was slow them. It took about 30-60 seconds to load a page, maybe more. Edit: this was using https, I didn't try http.
knowtheory 14y ago
The site had seemed slow basically all day for me, from say about 8:30-9am central time (I'd been up and checking the site since 6) up to the point when it crashed.
- pg 14y ago
  Were you using https or http?
  - michaelcampbell 14y ago
    I was getting the proxy error message with https. I never tried http then.
  - knowtheory 14y ago
    I've been using HTTPS consistently.
mcherm 14y ago
Yes: unusably slow both times I tried yesterday.
brudgers 14y ago
I was using my phone browser when the site was down. What I noticed was that the "secure" icon was displayed when I tried to connect. However, this may have been an artifact in the address bar from the previous site which was secure - I haven't paid that much attention to how the phone's browser handles slow loads. The reason it caught my attention was due to the recent https "Ask" threads, and my first thought was perhaps full time https had been implemented. I wouldn't bring it up except that it was unusual and correlated with HN content and the outage, even though it is likely to be a coincidence.

newman314 14y ago

More instrumentation/monitoring details needed to help diagnose future occurences?

ralph 14y ago

What were the most common activities done by the IPs that were "new" in the last twelve hours? New compared to the history of IPs available. Just wondering if it was orchestrated or whether it would give a clue as to the resource that was consumed.

FredBrach 14y ago

A -vague- suggestion: did you tried to run ghost users on HN by saving some bunch of users' session on the site?... You should be able to know if HN scale and maybe replicate the crash.

larrys 14y ago

I will repeat something I said below in my comments because I'm not sure you will see it: "But there is no question that it is a good idea to get to the bottom of why this happened. I have found in the past it always pays to do a post-mortem on any system problem as many times there is something lurking that will cause that same problem in the future." [1] I can't tell you how many times over the years some little thing has been an indicator of something else going on or caused a big problem later. It's normally better to diagnose this while you are not under duress, have room to breath, and aren't under any pressure. [1] Non system related last week my wife couldn't get into the garage because the GFI on the same circuit as the garage door opener had tripped. The same GFI is in the upstairs bathroom and had blown previously many times but it didn't seem to be a problem. We didn't realize it also controlled the garage door. So she couldn't get into the house (she didn't have her key). So by ignoring the problem (just keep resetting the GFI and hey - figure it out later) we had a larger impact and issue than it seemed on the surface.