11 ms·
Hacker News down, unwisely returning HTTP 200 for outage message
- dangrossman 13y agoHTTP 200 = "Cloudflare, please cache this status message instead of passing through a million requests to our dead server while it's busy restoring a backup". PG doesn't care about HN's search listings, so there's no drawbacks to doing that.
- Crito 13y ago> "PG doesn't care about HN's search listings" Yup. re: "Why does HN have a relatively low Google PageRank?" "Probably because we restrict their crawlers. But this is an excellent side effect, because the last thing I want is traffic from Google searches." https://news.ycombinator.com/item?id=5808990 https://news.ycombinator.com/item?id=5808990 If you need to search for something that you know was on HN, HNSearch is a great tool. I use it all the time.
- guyinblackshirt 13y ago>> HNSearch is a great tool. I use it all the time. Agreed - so is pinboard's search for #hn tags
- rangibaby 13y agoThanks for pointing out HNSearch. site:news.ycombinator.com (whatever) works nicely with Google too.
- cheapsteak 13y agoNot really. There were plenty of times when I tried to find an article a few days back but google was coming up blank even with `site:news.ycombinator.com`. Had to resort to scrolling through HN's facebook bot page (posts all the front-paged links)
- 6cxs2hd6 13y ago> "Probably because we restrict their crawlers. But this is an excellent side effect, because the last thing I want is traffic from Google searches." Am I the only one who finds that puzzling? This isn't Fight Club. It's not even Entrepreneur Club. It's a bunch of generally smart people talking about technology, with an emphasis on making money from it. It's one of my favorite sites, and I love it, but it's not an invitation-only club, is it? (I also find it weird that one of the go-to sites for web-savvy people would be like, "yeah, screw status codes and how the open, linked, web is supposed to work".) To be clear, I'm not Protesting a Great Evil. I just find it puzzling, as in, "That's odd, I must not understand what this is all about, after all."
- Crito 13y agoI can't speak for PG, but I think the general idea is that a slow influx of new users is less likely to alter the nature of HN as everyone has a chance to acclimatize (avoiding some sort of Eternal September), and the people who really "need" to be on HN (people interested in startups I guess?) will know about HN already, or be told about it. That last part might be a little "fightclub-ish" I guess, but it seems to be working alright.
- buttsex 13y agoCouldn't he just turn off registrations for new accounts? Not saying he needs to get HN to the top for a "startup" search query. I found HN by a Google search while looking for a good laptop to run Linux on.
- supergauntlet 13y agoThat happens. When too many people register accounts they're locked for the rest of the day and the register account option is no longer there, only login.
- Crito 13y agoIIRC sometimes the "Create Account" section of the login page is missing, but can still be accessed through https://new.ycombinator.com/submit https://new.ycombinator.com/submit That's not how it is right now so I'm not sure if I am remembering it correctly.
- ricardobeat 13y agoI seriously doubt Cloudflare's behaviour would be that stupid, wouldn't it momentarily cache error pages instead of hammering the server? At a minimum it would throttle/prevent concurrent requests.
- cbhl 13y agoUnfortunately, it took me 12 hours to find out the site was back up because the outage page had been cached for me. I had to realize I had to do a hard-refresh of the page.
- ricardobeat 13y agoThat's only because HN set a 10 year expiration header, as pointed out by the article.
- deleted 13y ago[deleted]
- X-Cubed 13y agoOther options would be to return status code 503 Service Unavailable with a Retry-After header, or drop connections while the dead server is busy. Both of those would be much better than returning 200 OK.
- mcgwiz 13y ago> pass through a million requests... How does that make sense? As if CloudFlare would honor status codes but not take advantage of cache headers (which in this case stipulated an absurd 10 year expiration).
- elwell 13y agocache was too long though. i didn't think to try ctl+f5 until i saw that it was on according to twitter.
- RobinUS2 13y agoCloudFlare also takes of the load of your dead server if the server is dead. It still has a "Try the original server"-button, but that is more manual.
- archildress 13y agoWelcome back, HN - I had an unusually productive (yet unstimulating) day at the office.
- Udo 13y agoDon't take this the wrong way, but after seeing hundreds of identical HN-outage productivity jokes/references on Twitter today, I really hope this is the last one I'll see in a while.
- archildress 13y agoWasn't really meant as a joke, but I completely understand where you're coming from. Haven't really been on Twitter today.
- frou_dh 13y agoI finally added tail-recursion optimisation to my Lisp interpreter while HN was down. Unless I hear otherwise, I'm taking the Onion crown!
- NigelTufnel 13y agoSomeone should write a twitter bot that monitors HN status and tweets "Hacker News down, productivity up". What the hell, I'll do it.
- foobarian 13y agoSeems like a fair point in principle, except in this case HN is one of the few sites I don't need Google to get to and don't care about any other tools that might rely on returned status codes.
- sysop073 13y agoTools like web browsers?
- JetSpiegel 13y agoOf course. foobarian sends an email with the URL to a service and it returns the webpage, which he reads in emacs.
- dragon1st 13y agoI just realized I've addicted to HN, have been kept refreshing the page millions time :) thanks HN
- eande 13y agoI am not a news junky, but I realized how dependent my newsfeed is from Hacker News. Seeing about 6-8 times the outage message reminded me on why I keep coming back and reading the quality entries of these forums.
- deleted 13y ago[deleted]
- ignostic 13y agoHe's right that for most sites this would be undesirable, but PG has stated that they aren't looking for a lot of Google traffic. But then, just because it doesn't matter to PG doesn't mean it doesn't matter at all. I've more than once found myself Googling old HN threads I'd like to find, but can't. Google's search (and site search) is miles better than HN's, but HN intentionally limits Google's crawl rate, thus limiting the amount of content crawled and indexed.
- yawboakye 13y agoBrowsers tend to cache 200 OK responses. When HN came up (as reported on Twitter) I kept getting the error page until I bust the cache and reloaded. Yup, that's what 200 OK for an error page can cause. A regular reload will still show your _down_ page
- ars 13y agoI had the same issue. Checking all day. I saw that it was back up so I tried it - and it was still down. Finally thought to try reload and it gave the page.
- tedd4u 13y agoWell by the letter of the RFC browsers and middle boxes (all the invisible caching proxies out there at your ISP, etc) are only supposed to cache if the cache/expires headers are set for that, but 200 is a bad choice for most people running a site for the other reasons listed above like crawlers/indexers. 5xx is correct for a problem on the server, usually 500, 503 or 504 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html). At the same time there are many erroneous implementations (intentional and unintentional) that, as you imply, just cache any 200 without checking cache headers. This is also a good practical reason to avoid doing this.
- 0x0 13y agoI think that was actually caused by the incredibly high cache-control expire header, which claimed cached versions of the page should be valid for 10 years (!)
- wiradikusuma 13y agoI know it suppose to make you more productive when it's down (I was), but suddenly I felt clueless during my commuting (which can take 2 hours total).
- joshsegall 13y agoThis isn't bad for just Google. RSS aggregators were also getting the "everything's fine" message. I thought I had a bug in my aggregator until I went to the site and realized it was down.
- tedivm 13y agoAs people mentioned this site is an exception to how to do things, in that PG actively does not care about search engine results. However, for the people who are interested, here's a few ways you can handle a situation like this. 1. If you add the "max-stale" header to your content you can tell your CDN to hold on to things longer in the event of an outage. What happens is that the CDN will look at the cache time you set and check in as normal when that expires, but if it gets back no response or a 503 code (server maintenance status code- if you work with CDNs this is your friend!) it will continue serving the stale content for as long as you tell it. 2. Lets say you're beyond that, or your site is too dynamic. Instead of setting up an error page that responds to everything, setup a redirect with the 302 status code (so crawlers and browsers know it's a temporary thing). Point that redirect at an error page and you're golden. The best part is these types of requests use minimal resources. What I do is keep a "maintenance" host up at all times that responds to a few domains. It responds to all requests with a redirect that points to an error page that issues the 503 maintenance code. Whenever there's an issue I just point things at that and stop caring while I deal with the problem. I've seen webservers go down for hours without people realizing anything was up, although with dynamic stuff everything is obviously limited. The other benefit to this system is that it makes planned maintenance a hell of a lot easier too. Oh, another thought- you can use a service like Dyn (their enterprise DNS) or Edgecast to do "failover dns". Basically, if their monitoring systems notice an issue they just change your DNS records for you to point at that maintenance domain. You can also trigger it manually for planned things.
- username223 13y ago"actively does not care about search engine results. " Or data -- they tossed a day or so of submissions and comments when restoring from backup, AFAICT. I'm not saying that's a problem; 4chan loses data and works just fine.
- n09n 13y agoThat's a poor comparison. Users expect 4chan to lose that same data as part of normal operation anyways, so you can hardly consider it a problem. That's not the case for HN.
- jwilliams 13y agoMaybe this was a prior error message, but the original CloudFlare "Origin Server" error was returning a 520 - which is a CloudFlare-custom HTTP Status Code. Edit: CloudFront -> CloudFlare
- garindra 13y agoIt's CloudFlare, not CloudFront.
- creativityland 13y ago@HNStatus helped keep me updated but it wasn't posted on the maintenance page until the end of the day for everyone.
- jacalata 13y agoAnd because of the caching, it wasn't posted on the maintenance page at all if you visited before it was added. (I was seeing it on one machine and not the other).
- leoh 13y agoOf hundreds of comments I have made on this site, only one has been snarky. Here's my second: chill your tits.
- grandalf 13y agoBy the logic of this blog post, a page like status.heroku.com should return a 503 when heroku is experiencing downtime and a 200 otherwise. 200 means that the page loaded as intended (which it did).It turns out that some of the page's content (the interesting stuff) was unable to be loaded, and the site's content reflected that. A 503 would be appropriate if there was a server problem, which might have actually been the case, but with Cloudflare's landing page there was not actually a server problem (since cloudflare served the substitute content properly w/o error)
- pizzeys 13y agoI disagree with this. By that logic, 404 pages should also return 200, because the error page stating that the content couldn't be found was indeed rendered successfully. The difference between your status.heroku.com example and this is that in the former case you are seeking to look at a page that tells you about their status, when in this case you were seeking HN's index but instead got a page about status - because there was a problem preventing you from getting what you wanted.
- randallsquared 13y agoAt my work, we have had arguments about what searches in an REST API should return in the case that no results (literally, in our parlance, "no documents") were found. Is that a 404 or a 200?
- danso 13y agoVia the W3 spec: > The request has succeeded. The information returned with the response is dependent on the method used in the request, for example: Is a query that turns up no documents when there is indeed no documents a success? I would argue yes, as the search service executed the query accurately...in fact, if the query turned up documents when it shouldn't have, that would probably be a problem... But the way the 404 is worded, it would also fulfill the meaning of "no results". > The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.
- fmax30 13y agoNow i know why i had to refresh when i first opened HN 10 minutes ago even though it was up at that time.
- datawander 13y agoEven if the author is correct in that Google's ranking algorithm for HackerNews will be affected by just 24 hours of downtime, wouldn't the algorithm update itself back to normal over the next 24 hours?
- cyanbane 13y agoI was 8% more productive today. Down-vote at will.
- anonfunction 13y agoFor anyone interested in HTTP statuses this is a great resource: http://httpstatus.es/ http://httpstatus.es/
- factorizer 13y agothose people certainly don't know their latin. just kidding.
- n1ghtmare_ 13y agoAnyone knows the reason for the downtime ?
- rplnt 13y agoI also noticed loss of data. For example: http://webcache.googleusercontent.com/search?q=cache:r53Zl8wLb94J:https://news.ycombinator.com/item%3Fid%3D7017250+&cd=1&hl=en&ct=clnk http://webcache.googleusercontent.com/search?q=cache:r53Zl8w... https://news.ycombinator.com/item?id=7017250 https://news.ycombinator.com/item?id=7017250
- kogir 13y agoThis was all me. I probably should have thought about it more, but just wanted to make it clear we knew something was wrong and were working on it. Load was not a concern. Though the article is correct, with everything else that was going on response codes and cache headers were the least of my worries. I think the best takeaway is that you will go down at some point, so it's best to have a reasoned plan in place for when you do. Handling it in the heat of the moment means you'll miss things.
- thaumaturgy 13y agoIf it becomes a big deal, and you can't get the data from ThriftDB for some reason, I've got a copy of HN data (submission & comment content, user, points, item date) up to id 7018491, a comment by kashkhan, time-stamped 2014-01-05 23:56:34. edit: Ack, I just realized that item_id got reset back to 7015126 on the reboot. My data matches HN up to 7015125, and then diverges after that.
- emillon 13y agoCan you make it available somewhere? I'd be interested in that data, and sure that others would be too. Thanks!
- thaumaturgy 13y agoSure. This is very temporary, I'll be removing the links sometime tomorrow: http://www.associatedtechs.com/tmp/hn_submissions_7015126.sql http://www.associatedtechs.com/tmp/hn_submissions_7015126.sq... http://www.associatedtechs.com/tmp/hn_comments_7015126.sql http://www.associatedtechs.com/tmp/hn_comments_7015126.sql After a semi-random sampling, the comments file appears to contain nothing but comments pre-crash. The submissions however got clobbered a little by the crawler at some point. There are some submissions in there pre-crash and some post-crash; I think everything's OK from 7015172 on, which only leaves 15 possibly damaged rows, and of those, I'd expect most of them didn't have id collisions. Sorting out the old stuff from the new stuff could be manually done. (Please let me know if there's anything I should be concerned about in those, or if they shouldn't be posted for some reason, or something. I'm recovering from flu and am still not entirely all here.)
- deleted 13y ago[deleted]
- Nodex 13y agoThe site is hella slow too, what on earth do they run it on ? 512mb VPS?
- lnanek2 13y agoCloudFlare really messes a lot of things up. I've seen CloudFlare refuse to give me error responses from forms before. Enter a bad value, get a cached page of the empty form, lol. Server was trying to return a page explaining the wrong entry, but CloudFlare refused to send it to me because it had a non-200 response.
- belorn 13y agoQuestion: Is the HNsearch still being worked on? It has broken "link" and "parent" for search results.
- Gonzih 13y agoSome people where speculating that HN banned goodle and other search engines. But from at least /robots.txt I can't see that. Do they do any ip based filtering? Does any one have information on that? I'm just curios.
- TeMPOraL 13y agoI know that pg "actively does not care about search engine results", but HTTP spec has other applications besides Google pagerank. It's hard to build amazing new technologies and improve the Web if people keep ignoring the standards without a good, technical reason. Please, for the sake of the example set to others, send the proper HTTP codes.
- nayefc 13y agowho cares...
- ck2 13y agoBy the way, HN still says it is down at this url https://news.ycombinator.com/ https://news.ycombinator.com/
- maxerickson 13y agoThe down pages had long cache times, you just need to refresh.
- Yuioup 13y agoBTW I just realised that Chrome has been serving me a cached version of HN the whole time. Didn't realise it was up again. When did it go back on-line? An hour ago? More?
- gedrap 13y agoMore than 10 hours ago
- Yuioup 13y agoThanks.
- dntrkv 13y agoWere you a member at computer-forums.net like 8 years ago? I know it's a long shot, but I remember someone with the same username as yours. I know it's not to the most unique username, but I thought I would give it a shot.
- Yuioup 13y agoNo sorry. I did come up with the name a long time ago but (like 1999 or something). I've used this username on /. for a while.
- brubaker 13y agoHN is probably one of the more unreliable sites on the internet.
- garthdog 13y agoI didn't know this was up until somebody told me to ctrl-reload. Nice one with the 10yr cached soft-500 page. :P Now that our brittle forum is back, let's get back to work nitpicking the Android UIs that aren't quite beautiful enough! This is not enough drop shadows!
- mountaineer 13y agoJust hit HN this morning and got the downtime message. Then I remembered this post and did a hard refresh to get back to normal. Browsers certainly aggressively cached it.
- HNJohnC 13y agoSame here, not sure why that isn't touted as a more important reason to provide the correct status codes rather than simply saying Google search isn't important. Standards are important.
- conanbatt 13y agoCompletely agree. I didnt even think of hard refreshing until i saw the "server is back up" status on twitter.
- nawitus 13y agoI learned about doing a hard refresh on Twitter (I didn't know about this post). I think it's a big problem, there's lot of users who can't currently access the site.
- catshirt 13y agoah. this explains why HN has been telling me it's down for 2 days now. had to open it in incognito to realize it was a cache issue.