6 ms·
Ask HN: Why is HN so often down?
It's daily now, that I'm seeing "We're having some trouble serving your request. Sorry!", maybe even 10-20 times during different times.
From Europe, especially during the lunch break hours, HN is down again and again. For instance as of the past 30 minutes, I see the above message more often than the expected pages. It's been like that also yesterday, the day before that.
I understand that this happens often based on searching, but I don't remember having it that often in the past.
Honest question: what happens, why do we get that message, especially during these times? Is there something that is going on the the early AM hours in the US? Maintenance work maybe?
Edit based on some comments below:
- could it be this has to do with being logged in or not? I will try to stay less often logged in.
- dang wrote: "but for those who are comfortable logging in and out: HN gets a lot faster if you log out, and it will reduce the load on the server if you do" (source: https://news.ycombinator.com/item?id=38309611)
- another reason could be rate limits, especially if not logged in
- rozenmd is monitoring HN, that looks awesome: https://hackernews.onlineornot.com/
- "Did you post something negative about Google/DEI? Or maybe just upvote it? Or skim one of those threads. There you go. Time out for you." <---- this is particularly disturbing IF TRUE
- mlvljr 3y ago[dead]
- adampencev 3y agoI’m new but been checking HN for the past weeks daily, multiple times a day usually. I haven’t experienced this on any single of my visits. Weird.
- merelysounds 3y agoI've had mixed results. Some visits were fine, especially those where I wasn't logged in. During others I've seen errors, often when writing comment replies or voting.
- vasco 3y agoJust guessing but I believe they run their own colocated infraestruture and don't have much or anything that autoscales, so if they're not overprovisioned enough and there's a lot of threads that have too many comments it happens. I also saw they are hiring for an infra engineer role so they might also be understaffed.
- rezonant 3y agoIt is a single one-core server, and I believe it's pretty old.
- begueradj 3y agoI remember I read somewhere here that it is recommended to log off your session when you do not need to stay online to comment or post something.
- sph 3y agoNo, it's the opposite, see: https://news.ycombinator.com/item?id=39548152 https://news.ycombinator.com/item?id=39548152 EDIT: I guess confirmation from dang isn't proof enough. Alright.
- marcus0x62 3y agoA much more recent comment from dang asked people to logout if they could when HN was under stress as it would make it faster for the user: https://news.ycombinator.com/item?id=38310213 https://news.ycombinator.com/item?id=38310213
- jcollins1991 3y agoI can also confirm that logging out (incognito tbh) fixed the errors for me in the past.
- sph 3y agoWhen the server is having issues, so the served front page is basically a cached static HTML file. The anti-bot limiter applies during normal operation. These are not the same thing. --- On Feb 28, 2023: > Hi sph > Yes, you ran into HN's anti-bot software, which blocked your IP. Sorry! Our intention is definitely not to block legit users, but we have to be somewhat aggressive about detecting bots because we don't have much performance to spare. > I've unblocked your IP. If you browse HN while logged in, that should immunize you against this happening again. Also, if it ever does happen again, you can unban your IP as described at https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html. But you have to do that from a different IP address, of course. > Let us know if you have any further trouble, > Daniel (dang) Emphasis mine.
- 3y ago
- GEBBL 3y agoMight be worth considering moving the arch of the site to a SaaS solution like netlify to really benefit from the scaling capabilities of serverless.
- Alifatisk 3y agoHaven’t the serverless horror stories scared you already?
- victorbjorklund 3y agoI hope this is a joke.
- sph 3y agoIt is not often down, but it's very easy to hit its rate limiter when upvoting or commenting. Waiting a minute usually fixes it. The rate limiter is even stricter when not logged in — for a period I wanted to take a break with getting into online arguments, so I stayed logged out, which would cause my IP to be banned after a day or two (a bit annoying, though easily unbanned with the self-service system: https://news.ycombinator.com/item?id=4761102 https://news.ycombinator.com/item?id=4761102) — IIRC dang confirmed that it would be better to stay logged in to avoid this anti-bot measure. I guess that's the price to pay to have such a popular website that's maintained by a single person.
- lukan 3y ago"I guess that's the price to pay to have such a popular website that's maintained by a single person." Why do you think, it is only one? There is definitely help with the moderation for example.
- EchoChamberMan 3y agoI can't find the source, but there is an article that mentions the entire site is moderated by 2 people. Maybe they've added a third since then.
- subtra3t 3y agoI think the 2nd person in the article you read was Scott Bell, who stopped working on HN around 2019 [1]. Still, dang isn't the only moderator here [2], just the only one who posts publicly. [1]: https://news.ycombinator.com/item?id=25055115 https://news.ycombinator.com/item?id=25055115 [2]: https://news.ycombinator.com/item?id=25235720 https://news.ycombinator.com/item?id=25235720
- EchoChamberMan 3y agoAh thank you for the clarification!
- 3y ago
- mnmalst 3y agoI have never seen that message before. Also from Europe.
- suddenclarity 3y agoI'm getting the same but only when using VPN. Not sure if there's a connection or just a correlation with me using the VPN during the day.
- neverrroot 3y agoI'm not using VPN
- nubinetwork 3y agoDang said they're working on it, I presume they are upgrading the hardware soon or something... (edit: in any case they are aware and it hopefully shouldn't be an issue long term)
- silverquiet 3y agoI was under the impression that a lot of the code is single-threaded and so throwing hardware at it won't do a whole lot.
- therealdrag0 3y agoSheesh. Isn’t this the easiest type of application to just cache the reads and queue the writes, then you can scale horizontally easily.
- vsnf 3y agoHe’s also been saying that for years, so.
- OJFord 3y agoIs that a reason to believe it less? You can also see from hiring threads that it's a very small (but growing) team and that inherently it is truly being worked on.
- tutfbhuf 3y agoHence, I would conclude that he is working on it for years.
- cutemonster 3y agoMaybe they have improved performance but traffic increased even more? There was this recently: A steep rise of Hacker News in Google rankings https://news.ycombinator.com/item?id=39423949 https://news.ycombinator.com/item?id=39423949
- zild3d 3y agoTraffic has probably been going up for years..
- treme 3y agoit's to remind you that there is no place like hn else where.
- pantulis 3y agoalso to remind you that one does not need to check HN when you have 5 minutes to kill. Go take a breath and have a walk.
- w-m 3y agoI've had that same message a couple of times over the last few days, while logged in (EU). It's kind of expected behavior when there's a thread with thousands of comments (like when OpenAI shafts their CEO [0]), but the last few days seemed relatively calm. So maybe the server is crumbling under the increase in traffic from higher HN rankings in Google [1]? [0]: https://news.ycombinator.com/item?id=38310213 https://news.ycombinator.com/item?id=38310213 [1]: https://news.ycombinator.com/item?id=39423949 https://news.ycombinator.com/item?id=39423949
- dontupvoteme 3y agoI was always under the impression it was a sort of shadowquasiban/shadowratelimit mechanic it definitely shows up a lot more via vpn
- shdon 3y agoI've been frequenting this site daily for over 15 years now, and I don't think I've had error messages more than a handful of times in that decade and a half. Today was indeed one such time - which is what triggered me to check out this particular post. But at least in my experience it's not what I'd call "often". Maybe you're just very unlucky, or perhaps I'm extremely lucky... Who can tell?
- gpderetta 3y agoI had never seen it for more than a decade, but in the last couple of years it has become very frequent.
- 7thaccount 3y agoI get it happening every few weeks for a short amount of time it seems. More so recently (also been a site user for approximately 10 years).
- eterm 3y agoFor the past couple of months I've had these errors multiple times a day, it's definitely often down now.
- neom 3y agoMonths? I'd say I'm mildly addicted to HN and I noticed this week it was hit or miss, but only this week. Very surprised to hear you say months.
- soneca 3y agoI confirm that “months” seems the right time range for me (accessing from Brazil). I open HN, idk, 10 or 20 times a day, I would guess I see the error message around 5 to 10 times a week.
- neom 3y agoDo you have a VPN you can try on? I run a VPN connected to a San Fran server 24/7 on my mac and I very very rarely have issued with HN (or any internet issues at all tbh).
- krisbolton 3y agoI've noticed it too - also in Europe. It could be a result of increased traffic from a recent Google search algorithm update which increased HN's ranking significantly. I understand HN infrastructure isn't enormous, so it would make sense. HN ranking discussion here - https://news.ycombinator.com/item?id=39423949 https://news.ycombinator.com/item?id=39423949
- monkeydust 3y agoIs there any open metrics on HN hits / usage etc? Curious on growth of this platform. For me is my #1 place I regularly go to each day. Other sites less important to hit frequently.
- nottorp 3y agoIt was down for me a bit today. But it's a great free service with no tracking/ads so I'm fine with that.
- TrueGeek 3y agoI agree it's a great free service and I've rarely seen downtime, but as a nitpick: The posts for YC backed companies that are hiring (where the comments are turned off) are ads.
- nottorp 3y agoI believe most regulars know who our sponsors are :) If you put it that way, the whole site is a giant ad for Y Combinator. It just happens to be a public service too. It's also one of the few sites* i frequent where uBlock Origin says it's not blocking anything. * Tbh the only other is Ars Technica, and there only because I'm logged in as a paying customer.
- hajmo97 3y agoSame here in the last few days between 10 AM and 1 PM (CET)
- Alifatisk 3y agoHN is my only way to see if something is down
- passwordoops 3y agoMaybe something to do with the cut cable and traffic routing? https://news.ycombinator.com/item?id=39506931 https://news.ycombinator.com/item?id=39506931
- reactordev 3y agoBackend upgrades I assume. The fact that this even exists, is free, is frequented, and has super valuable commentary is enough to make me wait. I get way more value out of HN than any other site I visit. It’s not my most frequented but it’s my most valued. Try not to compare it with a SaaS you pay for or that sells your information to pay for it.
- codegladiator 3y agodoes this bother you or you are asking out of technical curiosity ?
- neverrroot 3y agoConsidering how often it is, it is really bothering now, because it's wasting my time.
- codegladiator 3y agoI don't know if you are being clever or funny
- neverrroot 3y agoTo or or not to or, that is the question here.
- stanleychink 3y ago[dead]
- voytec 3y ago> what happens, why do we get that message This message is a HTTP 500 error page. 5xx are backend unreachability-related, so likely "the app" dies and has a long-ish boot time, or start is delayed by waiting for cron event.
- notso411 3y ago[dead]
- SCdF 3y agoMotivation for you to get back to work.
- prmph 3y agoI'm in Ghana, and can confirm that I'm seeing this a lot recently. Sometimes the problem persists even after multiple refreshes. I'm almost always logged in, and it happens even when I have not visited HN in some hours.
- ddev0 3y agoYou might be clicking on a bunch of links at once (triggering some anti-ddos/spam measure?). Happened to me once or twice. I assumed it might have been because my dynamic ip had been "used by robots" prior.
- urbandw311er 3y agoI can’t believe that this elite community of techies are chiming in with the equivalent of “works on my machine” :-)
- leoff 3y agoIt's very weird how most websites already figured out how to keep their services up under load, but a CRUD text-only forum, with an "elite community of techies", can't. I would be curious to know what is the big challenge here.
- diggan 3y agoProbably the fact that the software was written decades ago, hacked together and also the author decided to first write their own lisp, then implement HN software with this lisp (called Arc). I'm guessing that has something to do with the difficulty of changing it.
- 0x6c6f6c 3y agoI almost wonder whether they even care about changing it, because why? This site has been completely functional aside from some load-bearing issues here and there. It's not like it's an ad plagued, high stakes, profitable venture.
- ehnto 3y agoI agree, I would say it's perfectly capable just the target for success is different. It's not a commercial venture aiming for 99% availability and a dropped request here and there isn't that impactful.
- imiric 3y agoI would wager that it has more to do with the fact that it's served by a single one-core server: https://news.ycombinator.com/item?id=39548439 https://news.ycombinator.com/item?id=39548439 If anything, that showcases that the software stack is quite capable, however old and hacky it may be. And I suppose that HN is not too high up on the list of priorities for YC :) It works most of the time, and dang can put out any fires...
- kypro 3y agoSpeculation – but Google moved HN up in search recently and I suspect that's resulted in a spike in traffic. I assume the downtime is capacity related.
- JSDevOps 3y agoIt's NEVER down for me. I've been visiting for a few years now. Multiple countries, ISP's always up. Maybe it's your ISP censoring? Have a look at these https://github.com/tg12/RUBC https://github.com/tg12/RUBC https://en.wikipedia.org/wiki/HTTP_451 https://en.wikipedia.org/wiki/HTTP_451
- marban 3y agoYou're visiting too infrequently.
- 2devnull 3y agoUsed to happen whenever you criticized communist China so I assumed it wad an anti-naughty-posting feature. (Thankfully there’s been some relaxation on this, as evidenced by this post being visible.)
- OJFord 3y agoConsider the 'HN hug of death' we have patience for when a site linked on the front page is down. HN must be getting orders of magnitude higher traffic than that constantly.
- althea_tx 3y agoNot doubting the experiences of anyone else, but I have only been a daily user of this site for the past 18 months, but not once has it been down for me in the morning or evening when I reach for a dopamine hit.
- kitd 3y agoGo easy. The new hamster is taking a while to train up.
- piker 3y agoIsn’t the general ethos here that HN doesn’t care about some number of 9’s of reliability? They just upgrade stuff when necessary and probably bang on prod a lot more than big corporate sites. HN is a nice to have, and keeping it that way allows it to remain administered by a small crew.
- neom 3y agoDan gave some traffic numbers in 2015, and the site has grown a lot since then: "Roughly 2.6M views a day, 300K daily uniques, 3 to 3.5M monthly uniques. It depends on how you count, of course." https://news.ycombinator.com/item?id=9220098 https://news.ycombinator.com/item?id=9220098
- aaron695 3y ago[dead]
- nojvek 3y agoSame, it seems HN is getting more traffic but hasn't been keeping up with the load. Which is interesting because YC has funded many infra companies. I usually browse through hn.algolia.com and that has a much higher uptime than news.ycombinator.com In 2024, it is surprising how often HN is unavailable.
- rezonant 3y agoIt's still just one old server. Which itself is impressive, but with the search indexing changes maybe not enough anymore?
- Q6T46nT668w6i3m 3y agoYeah, it’s been down for me somewhat regularly in recent months but I find the outages charming.
- soneca 3y ago> ” Did you post something negative about Google/DEI? Or maybe just upvote it? Or skim one of those threads. There you go. Time out for you." <---- this is particularly disturbing IF TRUE” This is bullshit. I am sorry you decided to edit your comment to include that paranoia. I am seeing these errors a lot and a lot of times it’s when I just get to the site without any interaction. I usually comment very uncontroversial things and get the error all the same. Ps: I don’t know what’s “DEI”