6 ms·
The Onion cut 66% of their bandwidth by upstream caching 404s
- mattmaroon 17y agoSeems like there's a bit of a missed opportunity there if a lot of people are looking for content that no longer exists. I wonder if they did something that broke old linked URLs or something.
- frederickcook 17y agoBut you have to wonder how much of it really is spiders crawling over old pages with links or people actually clicking on links from old pages. Probably many more of the former, and that wouldn't be hard to test for.
- grayrest 17y agoThe linked post itself says it's mostly spiders following links from various place on the web to old/invalid URLs: > And the biggest performance boost of all: caching 404s and sending Cache-Control headers to the CDN on 404. Upwards of 66% of our server time is spent on serving 404s from spiders crawling invalid urls and from urls that exist out in the wild from 6-10 years ago. They're still serving the pages, but they're serving them off CDN instead of serving them off their main server.
- dminor 17y agoStill, I wonder what Google does with page rank that goes to a 404 vs a permanent redirect.
- john_onion 17y agoOkay you guys obviously aren't getting the whole picture. A minority of our links are from old content that I no longer know the urls for. We redirect everything from 5 years ago and up. Stuff originally published 6-10 years ago could potentially be redirected, but none of it came from a database and was all static HTML in its initial incarnation and redirects weren't maintained. This was before I took charge of the Onion's link management. Spiders make up the vast majority of my 404s. They request URIs that in no sane world should exist. They request http://www.theonion.com/video/breaking-news-some-bullshit-happening-somewhere,16928/google-analytics.com/ga.js http://www.theonion.com/video/breaking-news-some-bullshit-ha... They request http://www.theonion.com/articles/man-plans-special-weekend-to-reaffirm-commitment-t,17145/reddit.png http://www.theonion.com/articles/man-plans-special-weekend-t... even though that domain is explicitly set to http://media.theonion.com/ http://media.theonion.com/ and is not a relative url in the page source. I can't fix a broken spider and tell it not to request these links that do not even exist, but I still have to serve their 404s. Edit: In other words, because of broken spiders that try to guess URLs, I have roughly 15-20x as many 404s as 200 article responses, and there's nothing that can be done about it except move every single resource that exists on my page into a css spritemap. Edit #2: You can't see the urls very clearly, but what's happening is a spider is finding a filename on our page appending it to the end of the URL and requesting that to see if it exists.
- prodigal_erik 17y ago> from urls that exist out in the wild from 6-10 years ago From a glance at their archive, The Onion is still hosting articles from ten years ago. What in the world makes webmasters think it's okay to fail to maintain their mappings between URLs and live content? Cool URIs don't change: http://www.w3.org/Provider/Style/URI http://www.w3.org/Provider/Style/URI
- adnam 17y agoOne of those "holier than thou" topics, isn't it?
- boredguy8 17y agoOne of those, "I want people to find my website usable" things.
- ableal 17y agoOr self-interest. If I were a beancounter, and they told me that they didn't bother serving new ads with old content, because it was too much hassle to map the URLs, I'd be asking for cost/benefit figures ...
- araneae 17y agoOr a nice 301 redirect.
- eli 17y agoOnly if the resource actually moved and you know its new location. I hate it when people 301 /oldpages/* to the homepage.
- eli 17y agoSure in an ideal world every URL that was ever valid in the last 10 years would still work. Even the ones about a contest that ended in 2002 and the download page for your ActiveX-based Push application. In the real world, those tend to break somewhere around the second or third CMS upgrade.
- jeff18 17y agoThis seems more like a DailyWTF than a sweet optimization to be reminiscing about. I'd love to hear more details about this. I mean a single video is about the equivalent of tens of thousands of 404 pages. I'm not sure which is more mindboggling... Spending 66% of their bandwidth and 50% of their CPU on serving these trivially cacheable pages or the fact that they didn't correct the problem when they were serving more than 5%.
- showerst 17y agoThose videos are most likely being served from a CDN.
- jeff18 17y agoAs are the 404 pages, according to the comment.
- john_onion 17y agoI can confirm that videos, articles, images, 404 pages are all served by a CDN. Our 404 pages were not cached by the CDN. Allowing them to be cached reduced the origin penetration rate substantially enough to amount to a 66% reduction in outgoing bandwidth over uncached 404s. Edit: This is not to say that our 404s were not cached at our origin. Our precomputed 404 was cached and served out without a database hit on every connection, however this still invokes the regular expression engine for url pattern matching and taxes the machine's network IO resources.
- hartror 17y agoIf you read the entire thing (not just this select comment) you will find that they just re-engineered their site from Dupral to Django meaning they would have had this in place within days if not hours of discovering the bottle neck. http://www.reddit.com/r/django/comments/bhvhz/the_onion_uses_django_and_why_it_matters_to_us/ http://www.reddit.com/r/django/comments/bhvhz/the_onion_uses...
- john_onion 17y agoThose Onion web guys must be pretty dense.
- latch 17y agoTheir 404 page is 142KB...not huge, not small. Just sayin'
- joevandyk 17y agoIs all that HTML? Spiders generally only care about HTML -- they're not going to download the js and css and images and flash and whatever.
- latch 17y agoduh, I didn't think about that :) HTML is way smaller
- bluesmoon 17y ago8 years ago, maybe. Today all search engine spiders pull down and execute javascript & css as well just so that they can index a real user's experience.
- code_duck 17y agoThe better ones may, but perhaps not for all pages. Generally, when viewing logs I see search engine spiders make a single request, for only the html content and not js or css.
- danudey 17y agoI got 48 KB of HTML, 110 KB of stylesheets, 59 KB of images, and 136 KB of Javascript. I'm curious how you got your 142 KB number. Do you mean after it's been gzipped?
- tjpick 17y agoI dunno. I'd be more tempted to put a useful page behind the URLs that are already being requested, or redirect them to some relevant content.
- randomstring 17y agoOK, so what they are saying is that The Onion is throwing away 66% of its traffic by serving up 404s instead of serving up page views. Wouldn't they 1) have a better user experience by redirecting to the correct URL 2) have more page views and most important 3) make more money off of ads? The Internet is trying to give the gift of free traffic and The Onion is saying: "no thanks." Most sites have to pay for traffic, and wouldn't be trowing it away. Way to play it like Big Media. What's next? A pay wall?
- ShabbyDoo 17y ago"The Onion is throwing away 66% of its traffic by serving up 404s instead of serving up page views." Spidering and organic activity are bundled together in this 66% number. It would be interesting to know what percentage of this was people who wanted to see an Onion page.
- brc 17y agoIt's irrelevant that it is spidering mixed in. You can think of spidering as a future page view, assuming that page gets indexed. The spiders and organic visitors should have been 301 redirected to the correct location. Spiders learn the 301 redirects, and some of their articles may even benefited from better search rankings.
- john_onion 17y agoAs I said in my post below, spiders that request article_url/reddit.png or article_url/google-analytics.com/ga.gs do not get a 301 from me because they're not looking at an href of an <a> tag. They're guessing at a URL that never existed. They are legitimate 404 responses.
- juliusdavies 17y agoI feel sorry for John_Onion. The comments here are coming from people who have never looked at server logs (e.g. 'how do you know those are all spiders?'). Looking at my logs I would say at least 75% of my hits (juliusdavies.ca) are spiders. They come and visit every single page a few times a year to see if it's changed. For my own purposes I mirror some open source manuals and specifications (http://juliusdavies.ca/webdocs/ http://juliusdavies.ca/webdocs/). These have been on my site, unchanged, for at least 3 years, and the spiders come every couple months and check every page. These hits will never (and should never!) translate into even a single real user in my case.
- deleted 17y ago[deleted]
- Chirael 17y agoI believe Jakob Nielsen wrote about this almost 12 years ago, "Fighting Linkrot" http://www.useit.com/alertbox/980614.html http://www.useit.com/alertbox/980614.html