3 ms·

Ask HN: Preparing for the TechCrunch/HN Effect

14 points by tce 16y ago

I'm a n00b when it comes to developing web apps, but the one I've been working on lately has gotten a fair amount of really positive feedback. Again, I'm not a sysadmin or anything near the sort, so I'm quite unsure as to how to go about ensuring my app is robust. This has been a learning experience for me to develop a cool app from scratch, using RoR and Heroku. In the event that my app does go viral, I'd like my app to not crash and burn. How could I ensure my app is robust and able to withstand thousands of potential users simultaneously? Even though getting featured on TC or other sites might be wishful thinking at this stage, I'd still like to prepare for the worst-case scenario in terms of handling unexpected traffic spikes etc. Any pointers will be helpful!

travisglines 16y ago
I wouldn't assume that your app goes viral by default. In fact the chances of this happening are very slim. The guys that made breakup notifier almost went off the new page when they went viral. What I would do is put it out there and be ready to take the steps you need to, to keep it alive if it does go viral. Planning too far ahead for this is probably a waste of your time as you most likely wont be able to see the weak points before hand. Just move quickly if it does.
- tce 16y ago
  I didn't say that I'm assuming my app will go viral by default. In fact, there's almost a 100% chance that it will not, of course, and I'm very cognizant of that fact. And I definitely do understand that planning too far ahead would be a waste of time. My issue is that I'm pretty clueless as to all this sysadmin stuff in the first place. If not for my current app, I'd at least like to have an idea on what steps to take should a future app go viral.
  - atgm 16y ago
    With this in mind, (price) scalable-CDNs are a good idea. Don't spend a lot of money up-front when you're not sure you'll get the traffic to merit it. I'm glad I didn't, because my site got all of five hits from HN... though, granted, it's not tech.
    - tce 16y ago
      Did you have any recommendations (e.g. Cloudfront)? My google-fu is pretty weak, it seems. I'm not able to find any examples of how apps built on Ruby on Rails and deployed with Heroku were ensured to be robust.
mryan 16y ago
) Stress-test your app by simulating loads of users performing realistic actions on your site. See what breaks first. Optimise that. Start again. ) Offload your static media to a CDN. The cheapest and easiest way to do this is to copy your CSS/images/javascript to Amazon's S3, then change your HTML to reference the new location. ) Put a caching server in front of your current server (which then becomes the "application server"). ) Watch Jacob Kaplan-Moss' (one of the core Django devs) Deploying Django workshop [1]. While it is obviously focused on Django, there are loads of useful language-agnostic tips on deploying web/caching/DB servers. [1] http://ontwik.com/python/django-deployment-workshop-by-jacob-kaplan-moss/ http://ontwik.com/python/django-deployment-workshop-by-jacob... Edited to add: Forgot the obligatory "don't optimise too early" thing. The time you spend making sure your app can sustain a million uniques per day might be better spent elsewhere.
mooism2 16y ago
Investigate caching, both at http level (and put a caching reverse proxy, e.g. varnish, in front of your webapp) and at component levels. Ask yourself which database queries are essential for the basic experience and which are for enhancing the experience beyond that. It may be worth doing without the second class of queries (or being more willing to use out of date cached values) when your website is under heavy load. Ensure your sshd runs at a higher priority than your webserver/database/etc, so you have a chance of fixing things should your site actually crash and burn in the end.
mskierkowski 16y ago
http://blog.phpfog.com/2011/03/16/scaling-php-up-out-and-around/ http://blog.phpfog.com/2011/03/16/scaling-php-up-out-and-aro... I want to share this article (which I wrote at PHP Fog) about scalability (in PHP but applies to all web apps) and the scenario you are asking about. I agree with travisglines (and everyone else on the thread), that this scenario you are talking about is in fact rare. It's one of those good problems to have. ("I have so much money, how am I going to spend it all") Most people who I talk to care about scalability, not because they need it now, but rather because they want it as insurance. It sounds like you are in the same boat. My recommendation is to pick a hosting platform that allows you to build your application the same way you normally would, but allows scalability if you do need it. Heroku already does this for you for RoR; Windows Azure does this for .NET apps; GAE for Java/Python; and PHP Fog for PHP. Those platforms basically make caching and scalability really easy. Other than scalability and caching, you should build your application with performance in mind; as having a fast site is important regardless of the number of users. mryan & mooism2 referenced some good resources for doing exactly that.

14 points by tce 16y ago

travisglines 16y ago

I wouldn't assume that your app goes viral by default. In fact the chances of this happening are very slim. The guys that made breakup notifier almost went off the new page when they went viral. What I would do is put it out there and be ready to take the steps you need to, to keep it alive if it does go viral. Planning too far ahead for this is probably a waste of your time as you most likely wont be able to see the weak points before hand. Just move quickly if it does.

tce 16y ago
I didn't say that I'm assuming my app will go viral by default. In fact, there's almost a 100% chance that it will not, of course, and I'm very cognizant of that fact. And I definitely do understand that planning too far ahead would be a waste of time. My issue is that I'm pretty clueless as to all this sysadmin stuff in the first place. If not for my current app, I'd at least like to have an idea on what steps to take should a future app go viral.
- atgm 16y ago
  With this in mind, (price) scalable-CDNs are a good idea. Don't spend a lot of money up-front when you're not sure you'll get the traffic to merit it. I'm glad I didn't, because my site got all of five hits from HN... though, granted, it's not tech.
  - tce 16y ago
    Did you have any recommendations (e.g. Cloudfront)? My google-fu is pretty weak, it seems. I'm not able to find any examples of how apps built on Ruby on Rails and deployed with Heroku were ensured to be robust.

mryan 16y ago

) Stress-test your app by simulating loads of users performing realistic actions on your site. See what breaks first. Optimise that. Start again. ) Offload your static media to a CDN. The cheapest and easiest way to do this is to copy your CSS/images/javascript to Amazon's S3, then change your HTML to reference the new location. ) Put a caching server in front of your current server (which then becomes the "application server"). ) Watch Jacob Kaplan-Moss' (one of the core Django devs) Deploying Django workshop [1]. While it is obviously focused on Django, there are loads of useful language-agnostic tips on deploying web/caching/DB servers. [1] http://ontwik.com/python/django-deployment-workshop-by-jacob-kaplan-moss/ http://ontwik.com/python/django-deployment-workshop-by-jacob... Edited to add: Forgot the obligatory "don't optimise too early" thing. The time you spend making sure your app can sustain a million uniques per day might be better spent elsewhere.

mooism2 16y ago

Investigate caching, both at http level (and put a caching reverse proxy, e.g. varnish, in front of your webapp) and at component levels. Ask yourself which database queries are essential for the basic experience and which are for enhancing the experience beyond that. It may be worth doing without the second class of queries (or being more willing to use out of date cached values) when your website is under heavy load. Ensure your sshd runs at a higher priority than your webserver/database/etc, so you have a chance of fixing things should your site actually crash and burn in the end.

mskierkowski 16y ago

http://blog.phpfog.com/2011/03/16/scaling-php-up-out-and-around/ http://blog.phpfog.com/2011/03/16/scaling-php-up-out-and-aro... I want to share this article (which I wrote at PHP Fog) about scalability (in PHP but applies to all web apps) and the scenario you are asking about. I agree with travisglines (and everyone else on the thread), that this scenario you are talking about is in fact rare. It's one of those good problems to have. ("I have so much money, how am I going to spend it all") Most people who I talk to care about scalability, not because they need it now, but rather because they want it as insurance. It sounds like you are in the same boat. My recommendation is to pick a hosting platform that allows you to build your application the same way you normally would, but allows scalability if you do need it. Heroku already does this for you for RoR; Windows Azure does this for .NET apps; GAE for Java/Python; and PHP Fog for PHP. Those platforms basically make caching and scalability really easy. Other than scalability and caching, you should build your application with performance in mind; as having a fast site is important regardless of the number of users. mryan & mooism2 referenced some good resources for doing exactly that.