5 ms·

Still investigating. The site was slow all day. We got an immense spike in unique IPs. Typically we get a bit over 150k/day. Today we got 220k. Not sure if

by pg 14y ago

Still investigating. The site was slow all day. We got an immense spike in unique IPs. Typically we get a bit over 150k/day. Today we got 220k. Not sure if the downtime was related. I was travelling today and didn't have proper access to the server, or I would have been on it sooner.

andre3k1 14y ago
TechCrunch? http://techcrunch.com/2012/07/30/startup-claims-80-of-its-facebook-ad-clicks-are-coming-from-bots/ http://techcrunch.com/2012/07/30/startup-claims-80-of-its-fa...
- damiankennedy 14y ago
  That would be my guess as well. Although I am really pleased Codecademy has Python now. edit: removed the 'A'
- travisglines 14y ago
  I think so as well. Oddly enough that post on TC rose to the front page of reddit at the same time HN was down. http://www.reddit.com/r/technology/comments/xel55/startup_claims_80_of_its_facebook_ad_clicks_are/ http://www.reddit.com/r/technology/comments/xel55/startup_cl... It's been sitting on the front page for the past few hours. I suspect several sources led to the TC article and then through to the post on HN.
  - 7c8011dda3f3b 14y ago
    This was also submitted earlier today to /r/technology: http://www.reddit.com/r/technology/comments/xdwqk/ubisoft_uplay_drm_backdoor_allows_any_web_page_to/ http://www.reddit.com/r/technology/comments/xdwqk/ubisoft_up...
- culo 14y ago
  A lot from Techcrunch today ---> http://techcrunch.com/2012/07/30/mashape-public-launch http://techcrunch.com/2012/07/30/mashape-public-launch
- ig1 14y ago
  Unlikely, TC typically delivers <10,000 mentions to your site if the article is about your site. For a mention I doubt it generates significant traffic.
larrys 14y ago
"We got an immense spike in unique IPs. Typically we get a bit over 150k/day. Today we got 220k." I wouldn't exactly call that immense but more importantly it an increase like that certainly doesn't seem to be a reason for a site like this to go down (I didn't see it was down I'm going by the headline).
- cperciva 14y ago
  A 50% increase in unique IPs isn't necessarily something which could cause downtime, but it certainly could be related -- which is the word Paul used. For example: HN relies heavily on caching page data. If 70,000 robots suddenly started loading random items, it could easily cause both the jump in unique IPs and the downtime.
larrys 14y ago
"I was travelling today and didn't have proper access to the server, or I would have been on it sooner." The obvious question is isn't there anyone else who minds this tremendously popular site and is on duty if you are traveling?
- ocelotpotpie 14y ago
  I'm guessing the answer is something along the lines of: "Despite being immensely popular, it is a labor of love and generates little (or no?) revenue, so unfortunately, no there isn't anyone else minding the site."
- philwelch 14y ago
  Somehow I don't think any harm will come if Hacker News goes down for a few hours. Frankly, every time HN is down, I wonder whether this is finally the day PG has given up on the project, because it's slipped into dire form in recent years.
  - larrys 14y ago
    "Somehow I don't think any harm will come if Hacker News goes down for a few hours." What do you define as "harm"? People are creatures of habit. If something isn't available many times they will fork and find something else to do with their time which they might find as or more enjoyable or educational and then start to gravitate toward using that website. You only have so much time. If you find an alternative site that you like that might very well end up being your "addiction" and then that site could build critical mass. I used to start the morning with my laptop at Starbucks. I no longer do that. For the longest time I did that every morning. Then something changed and I found something else to do in the AM rather than that and achieved the same benefit. As an aside this is well known by Starbucks. I was told by one manager that at all costs they try to keep a SB being renovated open lest they loose people who have made it a habit to stop by and pickup their coffee. Now all this may or may not be important to PG. And sure a little downtime isn't going to have a major impact on usage. But there is no question that it is a good idea to get to the bottom of why this happened. I have found in the past it always pays to do a post-mortem on any system problem as many times there is something lurking that will cause that same problem in the future.
knwang 14y ago
what language/framework is HN built on?
- friism 14y ago
  Arc: http://en.wikipedia.org/wiki/Arc_(programming_language) http://en.wikipedia.org/wiki/Arc_(programming_language)
pg 14y ago
Update: An examination of the logs shows nothing remarkable happening at the moment of the crash. We may have run out of some resource. In that case it may be hard to know for sure what happened.
- nodesocket 14y ago
  Maybe add some monitoring such NewRelic and ServerDensity.
- johnrob 14y ago
  If it helps at all, I'm pretty sure there was a period around 3:30 when http was working but https was not.
  - harshreality 14y ago
    I noticed that too. I'm not sure of the time, but it was more than a momentary thing. Https was either not responding at all or it was responding only after a long wait. A bit later, https was taking 20-30 seconds before returning the page, which I think was an improvement over the earlier period where I didn't see it return a page at all. I didn't personally notice any http outage or delay.
  - sp332 14y ago
    Probably related, hackerne.ws kept working for me well after news.ycombinator.com stopped, possibly because I was accessing the latter over HTTPS.
- pg 14y ago
  Question: Did people find HN horribly slow this afternoon? (I found it unusably slow around 2 pm pacific, but there was little I could do about it from where I was then.)
  - dt7 14y ago
    I found it horribly slow from about 18 hours ago (3pm GMT) onwards. I think the last time I tried it was about 9pm GMT, and it was slow them. It took about 30-60 seconds to load a page, maybe more. Edit: this was using https, I didn't try http.

by pg 14y ago

andre3k1 14y ago

TechCrunch? http://techcrunch.com/2012/07/30/startup-claims-80-of-its-facebook-ad-clicks-are-coming-from-bots/ http://techcrunch.com/2012/07/30/startup-claims-80-of-its-fa...

damiankennedy 14y ago
That would be my guess as well. Although I am really pleased Codecademy has Python now. edit: removed the 'A'
travisglines 14y ago
I think so as well. Oddly enough that post on TC rose to the front page of reddit at the same time HN was down. http://www.reddit.com/r/technology/comments/xel55/startup_claims_80_of_its_facebook_ad_clicks_are/ http://www.reddit.com/r/technology/comments/xel55/startup_cl... It's been sitting on the front page for the past few hours. I suspect several sources led to the TC article and then through to the post on HN.
- 7c8011dda3f3b 14y ago
  This was also submitted earlier today to /r/technology: http://www.reddit.com/r/technology/comments/xdwqk/ubisoft_uplay_drm_backdoor_allows_any_web_page_to/ http://www.reddit.com/r/technology/comments/xdwqk/ubisoft_up...
culo 14y ago
A lot from Techcrunch today ---> http://techcrunch.com/2012/07/30/mashape-public-launch http://techcrunch.com/2012/07/30/mashape-public-launch
ig1 14y ago
Unlikely, TC typically delivers <10,000 mentions to your site if the article is about your site. For a mention I doubt it generates significant traffic.

larrys 14y ago

"We got an immense spike in unique IPs. Typically we get a bit over 150k/day. Today we got 220k." I wouldn't exactly call that immense but more importantly it an increase like that certainly doesn't seem to be a reason for a site like this to go down (I didn't see it was down I'm going by the headline).

cperciva 14y ago
A 50% increase in unique IPs isn't necessarily something which could cause downtime, but it certainly could be related -- which is the word Paul used. For example: HN relies heavily on caching page data. If 70,000 robots suddenly started loading random items, it could easily cause both the jump in unique IPs and the downtime.

larrys 14y ago

"I was travelling today and didn't have proper access to the server, or I would have been on it sooner." The obvious question is isn't there anyone else who minds this tremendously popular site and is on duty if you are traveling?

ocelotpotpie 14y ago
I'm guessing the answer is something along the lines of: "Despite being immensely popular, it is a labor of love and generates little (or no?) revenue, so unfortunately, no there isn't anyone else minding the site."
philwelch 14y ago
Somehow I don't think any harm will come if Hacker News goes down for a few hours. Frankly, every time HN is down, I wonder whether this is finally the day PG has given up on the project, because it's slipped into dire form in recent years.
- larrys 14y ago
  "Somehow I don't think any harm will come if Hacker News goes down for a few hours." What do you define as "harm"? People are creatures of habit. If something isn't available many times they will fork and find something else to do with their time which they might find as or more enjoyable or educational and then start to gravitate toward using that website. You only have so much time. If you find an alternative site that you like that might very well end up being your "addiction" and then that site could build critical mass. I used to start the morning with my laptop at Starbucks. I no longer do that. For the longest time I did that every morning. Then something changed and I found something else to do in the AM rather than that and achieved the same benefit. As an aside this is well known by Starbucks. I was told by one manager that at all costs they try to keep a SB being renovated open lest they loose people who have made it a habit to stop by and pickup their coffee. Now all this may or may not be important to PG. And sure a little downtime isn't going to have a major impact on usage. But there is no question that it is a good idea to get to the bottom of why this happened. I have found in the past it always pays to do a post-mortem on any system problem as many times there is something lurking that will cause that same problem in the future.

knwang 14y ago

what language/framework is HN built on?

friism 14y ago
Arc: http://en.wikipedia.org/wiki/Arc_(programming_language) http://en.wikipedia.org/wiki/Arc_(programming_language)

pg 14y ago

Update: An examination of the logs shows nothing remarkable happening at the moment of the crash. We may have run out of some resource. In that case it may be hard to know for sure what happened.

nodesocket 14y ago
Maybe add some monitoring such NewRelic and ServerDensity.
johnrob 14y ago
If it helps at all, I'm pretty sure there was a period around 3:30 when http was working but https was not.
- harshreality 14y ago
  I noticed that too. I'm not sure of the time, but it was more than a momentary thing. Https was either not responding at all or it was responding only after a long wait. A bit later, https was taking 20-30 seconds before returning the page, which I think was an improvement over the earlier period where I didn't see it return a page at all. I didn't personally notice any http outage or delay.
- sp332 14y ago
  Probably related, hackerne.ws kept working for me well after news.ycombinator.com stopped, possibly because I was accessing the latter over HTTPS.
pg 14y ago
Question: Did people find HN horribly slow this afternoon? (I found it unusably slow around 2 pm pacific, but there was little I could do about it from where I was then.)
- dt7 14y ago
  I found it horribly slow from about 18 hours ago (3pm GMT) onwards. I think the last time I tried it was about 9pm GMT, and it was slow them. It took about 30-60 seconds to load a page, maybe more. Edit: this was using https, I didn't try http.