5 ms·
Bulletproof your blog – survive the Hacker News effect
- zrail 15y agoI wrote kind of a hybrid static blog setup the other day[1]. You write posts in markdown, check them in along side the app, and then push to Heroku or wherever else you want to host it. It's just a little Sinatra app, no database or anything. [1]: http://bugsplat.info/2011-12-14-yet-another-not-very-static-blog-generator.html http://bugsplat.info/2011-12-14-yet-another-not-very-static-...
- nodemaker 15y agoI find it funny (and perhaps ironic!) that people would want their thoughts kept in some obscure row and obscure column of a remote SQL database. But perhaps it is better than not storing and sharing your thoughts at all! In any case...Merry Christmas HN!
- zrail 15y agoHow is that ironic? Thoughts have to be stored somewhere and databases happen to be really good at storing things.
- nodemaker 15y agoIt is ironic because the real deep reason we share our thoughts is because we want them to stand out! Databases are indeed good at storing millions and billions of things and they are designed for it.But no human being is individually capable of creating content that really justifies a "dynamic programming oriented query plan generation" each time someone wishes to access his thoughts! tl;dr My argument is that static site generation for personal blogs is common sense!
- zrail 15y agoSure. My counter-argument is that for multi-tenant hosting like tumblr where millions of people keep their personal blogs, keeping posts in a database is common sense because they're a known quantity. Backup strategies, optimization techniques, operations procedures are all well known and battle tested. Also, creating content using a web-based editor that saves to a database is very easy for not-so-technical people to wrap their minds around. Static site generators? Not so much.
- nodemaker 15y agoYes sites like tumblr definitely make sense...My argument is against running your self-hosted personal blog with an SQL server! And of course...my argument is for the technical people only....perhaps someone among us can make a static content generation solution for everyone!
- de90 15y agoBut for the majority of people constructing the html\css to do that is a chore\"impossible". Storing it into a database and providing a simplistic word editor to write up new things is the easiest way. Storing the information in a db just allows it to happen easily.
- isocpprar 15y agoI don't think that's really irony...how is that worse then some obscure file in some obscure server somewhere? Or some obscure block of memory in some obscure cloud somewhere?
- nodemaker 15y agohaha good point but thats why I prefer the Dropbox based approach where I just create a text file and put it in my dropbox folder!
- kijin 15y agoIt's a shame that several years after the term "slashdot effect" was coined, self-hosted personal blogs are still vulnerable to this easily preventable phenomenon. The companies that host the majority of the world's small web sites, such as BlueHost, DreamHost, and (heavens forbid) GoDaddy, have not made any innovations in this department. Most VPS's are no different. NearlyFreeSpeech.NET is really good at handling traffic spikes, for both static and dynamic sites. It combines the convenience of shared hosting with the scalability of cloud computing. You just need to insert more coins when you run out of your prepaid traffic.
- marquis 15y agoWhere is the financial incentive for these hosting companies to do so? Most users will never see a traffic spike that can take down your average VPS or bluehost site. It seems there is a gap in the market for self-hosted sites that have built-in structures to build upon that allow for traffic spiking.
- edanm 15y agoIsn't the gap filled by companies like Posterous/Tumbler? For the average user, they answer the problem of "put something online and make sure it stays there" perfectly well.
- Joakal 15y agoNearlyFreeSpeech was the cloud before it became popular. The other services are just hosting many websites on a server.
- iamben 15y agoI had just under 90k uniques in a 24 hour period (and 170k-ish pageviews) on NFS last month and it held up without any problems :-) I think it cost me about 10 cents. Brilliant.
- ck2 15y agoSave dynamic page as html from browser. Upload to server as html. Add.htaccess rule to redirect old dynamic page to static page. (when traffic dies, remove rule) Or if you are running WordPress, just install wp-super-cache and when a traffic spike happens, click the "lockdown" button.
- sirn 15y agoEasier: just put this into your .htaccess on traffic spike. Header append X-Coral-Control "redirect-home" RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$ RewriteRule ^(.*)$ http://your.domain.nyud.net/$1 [R,L] Or even better, just redirect all traffic with news.ycombinator.com or reedit.com referer (before RewriteRule): RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?ycombinator\.com [OR] RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?reddit\.com [OR] You don't even have to manually disable caching after traffic dies down this way.
- jjcm 15y agoI personally would recommend hosting on a VPS with a reverse proxy like varnish set up. Even if you've had very minimal experience with linux systems, it's a cinch to setup and you can get quick, fast caching up in a couple of minutes.
- ggwicz 15y agoThe Jekyll recommendation is a great one. For quick one-off articles something like pen.io is great, as well. And GitHub Pages are just fantastic if you didn't know already :) In my experience, MediaTemple handles traffic spikes remarkably well. Maybe not ideal for the super-techies, but running WordPress on it with a caching plugin handles traffic pretty well at the $20 / month plan. And using S3 to serve static sites is fantastic. I'm on my phone right now so it's a pain to find the specific article, but Kyle Shank uses S3 and wrote a pretty handy article at http://kyleshank.com http://kyleshank.com. Check it out and try to find the article. Great read.
- tikhonj 15y agoHe forgot the geekiest option: write your own blogging engine. It's a perfect excuse to use that new language and framework you're learning! Or, if you're not learning anything now, it's a great excuse to start learning.
- masnick 15y agoWhile this is a great learning experience, it is hard to write a blog engine that will stand up to traffic spikes on the first try. If your goal is learning a new language or framework, this is a great idea. If your goal is to have a reliable, traffic-resistant blog I think it's best to stick with a tried and true platform.
- seunosewa 15y agoIt's not hard. Just write a blog engine that generates static html pages.
- masnick 15y agoRight, if you write something akin to Jekyll it will work fine. However, I think most people would probably be trying to learn a framework like Rails or Django if they were writing a blogging engine to learn a new language/framework.
- mapleoin 15y agoSo write your blogging engine as you normally would and then make a button which says: dump to html for each page and then figure out when you want that button to be pressed (i.e. manually or automatically). More things to learn. Yay!
- sunkencity 15y agoBuild it anyway you like but then wget --mirror before rsync to deploy.
- 15y ago
- danneu 15y agoI wasn't a big fan of the Jekyll workflow, especially when I wanted to write a simple post from another device. My solution to that, an app that wrapped Jekyll, just felt hacky. Instead, I just made a Rails app with fifteen minutes spent dropping in Active Admin and RDiscount to render @post.body Markdown into @post.rendered_body HTML. Then, I aggressively cached pages into the static `public/` directory. I chose Rails simply because I'm using Rails for more serious apps and I wanted a casual app that I could take seriously enough to practice on. Otherwise, I'd have as much fun building the same thing in any other Ruby framework for Ruby practice and then cache the pages into a static directory. Isn't that what any other static site generator does?
- chubs 15y agoDon't most people with high-traffic blogs simply use posterous or tumblr these days so they don't have to worry about these kinds of things? That's what works for me... then the next problem rears its ugly head: bulletproofing your DNS!
- kmfrk 15y agoPeople with basic blogging needs are spoiled beyond belief with services like Tumblr, but if you need more advanced things to play with, it becomes a little overwhelming. ExpressionEngine understood the need for people with a lot of needs, but I think everyone eventually have to roll their own (using some sort of framework), if they want something that satisfies their perfectionism.
- masnick 15y agoUsing a platform like Tumblr makes a lot of sense, but lots of people still don't do this as evidenced by all the stories on HN that link to overloaded servers.
- jyap 15y agoWhen my blog hit number 1 on Hacker News earlier this month (http://news.ycombinator.com/item?id=3328619 http://news.ycombinator.com/item?id=3328619) it never went down. Running Jekyll on GitHub with my own domain julianyap.com. 30,000 hits in 24 hours. I remember the "move your site off of GoDaddy" post last week going down this week which was run on Tumblr. Elsewhere Tumblr wasn't down so it depends on the resources they allocate to you. I suspect MG Seigler's blog is on a top tier Tumblr resource pool so I wouldnt recommend Tumblr unless you have constant popularity to your blog.
- hajrice 15y agoSurprised the author didn't mention WP Engine. We (along with SmartBear, Foursquare and many others) use it at Helpjuice.com for our blog and it works great.
- masnick 15y agoI hadn't heard of WP Engine. I'll have to check it out. Thanks for the tip.
- tvorryn 15y agoI was a little disappointed that when he talked about Jekyll blogs, he didn't mention Octopress. The default theme looks pretty good, it has a decent set of plugins and the code display looks great. It was actually pretty easy to set up too. http://octopress.org/ http://octopress.org/ My blog using the default theme: http://www.trevoroakes.com http://www.trevoroakes.com
- masnick 15y agoGood point, I'll add this.
- apsurd 15y agoSince discovering Jekyll and the fact that GitHub Pages will host your content for free, I think this would be the ideal solution. However Jekyll is a pain in the ass to use, which is why I've been working on http://jekyllbootstrap.com/ http://jekyllbootstrap.com/ Jekyll bootstrap is intended to deploy straight-away to GitHub pages so hosting is free. I'm not finished though so any more eyeballs I get on the source will definitely be helpful. Merry Christmas =)
- riobard 15y agoDisadvantages: Not for the non-geeky. You can only post from a computer where you have your static site generator set up Not necessarily. I have a very simple static site generator (basically a single Python script, plus a few dependencies) running on a EC2 Micro instance. It watches changes in a shared Dropbox folder (using inotify and dropboxd) of a bunch of Markdown files and templates, and generates HTML posts, index page, and RSS automatically. I could post from any machine as long as it has my Dropbox account, even on my iPhone! No more command lines or git stuff, just write. Dropbox will take care of the rest. Joe Hewitt's Nerve [1] works this way. I believe Marco Arment's Second Crack works similarly, too, but it's not open sourced yet. [1]: https://github.com/joehewitt/nerve/ https://github.com/joehewitt/nerve/
- bsphil 15y agoSounds like it still requires more setup than the average non-geek is willing to do.
- masnick 15y agoYes -- if you're comfortable with the command line and a scripting language, you can set up a script to handle your static blog. However, I suspect most people who want to blog don't have these skills or don't want to spend the time messing with the tech.
- jQueryIsAwesome 15y agoActually it would be pretty easy to create a Wordpress plugin to do something very similar; using ob_start of php and keeping the real wordpress installation inside a subfolder. I am pretty busy this days but if someone wants to create and opensource project that hundreds of people would love, that would surely be one.
- riobard 15y agoSure. I'm not addressing the non-geeky part of the problem :)
- zrail 15y agoIf someone hadn't set this up as a hosted system I would have done so tomorrow, but it turns out there's already a hosted markdown + Dropbox thing out there: http://calepin.co/ http://calepin.co/
- oldstrangers 15y agoIt is easier than ever today to "bulletproof" your blog. I'm running a hybrid server and have found wordpress + cloudflare + w3tc to be a very solid setup. I got hit with 73,230 unique visitors (94.46% new visitors) in a 24 hour period earlier this month and you would've never noticed.
- dsl 15y agoI'm really surprised by this article. I thought this was a pretty much solved problem now that CloudFlare exists...
- buster 15y agoor: http://cmdrtaco.net/2011/09/scaling-in-the-cloud/ http://cmdrtaco.net/2011/09/scaling-in-the-cloud/
- jaequery 15y agoi'm sorry but this author has failed to point out the most commong cause for why a site goes down. it's not because of failing to cache...etc...most dbs will cache for you anyway... the main reason is because you probably are on apache and serving static assets directly w/o hosting them off a CDN or nginx/litespeed. if your page had 150 images/js/css combined, that's 150 hits hitting your server per page load. now multiply that by 100's or 1,000's+ visitors HN brings to your site at any given moment, ofcourse it will go down. apache isn't really good at serving static files, just relieving static assets off to cdn/fileserver is huge.