11 ms·

Wow...this is hard to believe. Numbers like that make me wonder if the cost of going with ruby or python (in reddit's case) or insert-your-slower-dynamic-langua

by rufugee 16y ago

Wow...this is hard to believe. Numbers like that make me wonder if the cost of going with ruby or python (in reddit's case) or insert-your-slower-dynamic-language-here outweigh the benefits in the long run. How can anyone bootstrap a company and scale to the size of Reddit without quickly running into a "must fund or sale" situation? If it's truly a python issue, it should not be taken lightly by entrepreneurs. I hope it's not...I'm an avid user of both ruby and python, and want to believe they can both be used to create successful and maintainable (in terms of effort and cost) sites.

joshu 16y ago
Actually, I bet you it's a "we hit the database on every pageload" problem.
- giantfuzzypanda 16y ago
  Yep. That's why you shouldn't have pylons noobs running a site like reddit.
- emmett 16y ago
  I know the reddit guys, and I can tell you it's definitely not that. Hitting the DB is extremely rare - almost everything comes straight out of cache (either memcache, or memcachedb for caches which are more expensive to rebuild). Execution cost is dominated by the python app servers.
- blhack 16y ago
  The overwhelming majority of reddit's pageviews are users that are not logged in, which is very very cacheable.
  - emmett 16y ago
    I am pretty sure this is false. While the overwhelming number of unique visitors to reddit are not logged in, those users are far less engaged than average. Logged in users make up a huge percentage of the pageviews because of their higher engagement level.
  - jedberg 16y ago
    About 20% of the users are logged in, and they represent about 50% of the page views.
- jedberg 16y ago
  The database is rarely accessed in request. Almost all DB access comes from batch processes.
blueben 16y ago
How did you get from "It costs this much" to "Dynamic languages are bad"?
- apower 16y ago
  Cause a server with these languages has high overhead in memory and CPU to serve requests thus having lower throughput per server and thus needing more servers to sustain certain site-wide SLA and thus causing more money overall.
  - blueben 16y ago
    Do you have evidence that reddit (or any site using a dynamic language) would reduce overall operating costs by switching, or is this an assumption on your part? What about development costs? What about time to deploy new features? What about the cost to make platform revisions? Do you have evidence that a non-dynamic language would really be so much faster that it would require less hardware? Perhaps the problem is not the language, but the chosen architecture? What I'm really trying to say here is... prove it.
    - SkyMarshal 16y ago
      Difficult to prove, but an observation with precedent. 37 Signals is famous for justifying Ruby by saying that hardware is cheap, developer time is not, so use RoR and just throw servers at your app till it meets your performance needs. Hardware is cheap, but maybe not cheap enough to run a site like Reddit with intentionally scant ad revenue.
      - axod 16y ago
        Hardware is cheap. Amazon hosting is not.
    - apower 16y ago
      Do you have evidence that dynamic language have less overall development cost?
jnoller 16y ago
Bad architecture or poor design crosses languages. It's entirely possible and done every single day, to emit a broken system using even the most optimal code. That said; this looks, sounds and smells like database scaling issue more than anything. It is interesting to ponder the long-term costs of running a site like this on EC2 versus your own maintained server farm.
- vinhboy 16y ago
  I am not familiar with the cost of running a high profile website, is the price tag for Reddit considered a lot, or little?
  - axod 16y ago
    A very lot.
- steveklabnik 16y ago
  At least a few months ago, here's Reddit's architecture: http://blog.reddit.com/2010/03/and-fun-weekend-was-had-by-all.html http://blog.reddit.com/2010/03/and-fun-weekend-was-had-by-al...
tl 16y ago
Here's an anecdote for you: our corporate intranet site gets a peak spike of a few thousand users every day, around lunch. (People browsing the web over lunch and the computers in-house all have the site as their fixed homepage.) That web server is running separate from databases and other resources. It never sees load. We're using a static language, but even if ruby was 10x slower, it wouldn't matter. On the other hand, change the contents of a stored procedure so that it doesn't line up with indexing properly, and page load goes from >0.1s to <4s.
emmett 16y ago
14.3 million pageviews per day comes to 163 per second, on average. Reddit's traffic, like most sites of its kind, probably peaks during lunch hour during the workday and probably peaks at 5x that number. So we're looking at 815 pageviews per second. Given 80 servers total, we're looking at 10 pageviews per second per server. Let's compare this with Facebook. In October 2009, Facebook announced they had 30k servers[0]. In September 2009, there was a rough estimate that Facebook served 200 billion pageviews per month[1]. That implies 73k pageviews per second, or 2 pageviews per second per server. Clearly the pageviews are a rough estimate, but even if facebook served 1 trillion pageviews per month they still wouldn't be beating reddit for efficiency. I have a feeling if you run the numbers for any other highly dynamic site at scale, you'll find that amortized over every server in use, you won't get a lot better than 10 pageviews per second. [0] http://www.datacenterknowledge.com/archives/2009/10/13/facebook-now-has-30000-servers/ http://www.datacenterknowledge.com/archives/2009/10/13/faceb... [1] http://www.businessinsider.com/googles-estimates-for-facebooks-traffic http://www.businessinsider.com/googles-estimates-for-faceboo...
- moonpolysoft 16y ago
  That's terribly misleading, though. Facebook has huge clusters for performing offline business analytics. Therefore comparing them to Reddit, which presumably has a much smaller offline analytics backend, is not an apples to apples comparison in terms of efficiency.
  - mrkurt 16y ago
    It's an educated guess, but I suspect comparing the entire infrastructure of a site like reddit with the entire infrastructure of a site like Facebook is pretty fair. Neither of them are using their entire server capacity to serve traffic. reddit's at a smaller scale than Facebook, obviously, but it's a pretty apt comparison.
    - invisible 16y ago
      You think the functionality of Facebook compares equally to that of Reddit computational-wise? I mean, I know neither is the most complicated thing ever, but does Reddit even allow photos anywhere besides thumbnails? Does Reddit find things only specific to your account and show you a list of friends and their content you are allowed to see (or even something vaguely similarly complicated)? I also highly doubt that Facebook counts all of the ajax-goodness it delivers as pageviews. I could really go on for a long time (widgets, Facebook connect, Facebook Apps API...) about the huge differences that you glazed over since it's using PHP for the frontend.
teej 16y ago
I ran a web app with identical pageview numbers using Ruby on Rails and EC2. My server costs were 10x cheaper than what reddit is paying. Scaling is caching and architecture, not writing your app in Java.
- awa 16y ago
  It would be nice if you elaborate a little.
  - teej 16y ago
    I'm not sure what you want to hear, so I'll elaborate a lot. Warbook was a Facebook application written in Ruby on Rails I ran by myself in late 2007 - 2008. It grew to over 16 million pageviews a day. At the time it was more pageviews than Twitter. I scaled it using the following stack: Perlbal for load balancing, LightHTTPD for static assets, Mongrel for dynamic requests, Memcached for caching, and MySQL for relational data storage. I used two medium instances for load balancing, one medium instance for asset hosting, 15 small instances for mongrel, one XL instance for memecached, and one XL instance for MySQL. I used memcached as a "write-through" cache. Everything in cache was considered fresh. Every write of a cachable object would write to both MySQL and memcached. Every read of a cachable object would start with memcached first and failover to MySQL. This reduced reads on the database by 95%. Total hosting costs were ~$2,000 a month.
    - mdolon 16y ago
      If you don't mind me asking what happened to it? Facebook now reports about 5,500 monthly users. I'm guessing you were acquired by SGN but I'm don't see why that would cause traffic to drop so much.
      - teej 16y ago
        SGN switched their core focus from Facebook games to iPhone games in mid 2008. At that time, they droped support for all of their Facebook properties except (fluff)Friends. Games on the Facebook platform are doomed to rapid traffic loss without constant adaptation and viral tuning. Even with it, user retention is hard. Facebook giveth and Facebook taketh away.
Devilboy 16y ago
I run 2 Reddits on my calculator watch using C++ and CouchDB so clearly the technology is at fault here.

by rufugee 16y ago

joshu 16y ago

Actually, I bet you it's a "we hit the database on every pageload" problem.

giantfuzzypanda 16y ago
Yep. That's why you shouldn't have pylons noobs running a site like reddit.
emmett 16y ago
I know the reddit guys, and I can tell you it's definitely not that. Hitting the DB is extremely rare - almost everything comes straight out of cache (either memcache, or memcachedb for caches which are more expensive to rebuild). Execution cost is dominated by the python app servers.
blhack 16y ago
The overwhelming majority of reddit's pageviews are users that are not logged in, which is very very cacheable.
- emmett 16y ago
  I am pretty sure this is false. While the overwhelming number of unique visitors to reddit are not logged in, those users are far less engaged than average. Logged in users make up a huge percentage of the pageviews because of their higher engagement level.
- jedberg 16y ago
  About 20% of the users are logged in, and they represent about 50% of the page views.
jedberg 16y ago
The database is rarely accessed in request. Almost all DB access comes from batch processes.

blueben 16y ago

How did you get from "It costs this much" to "Dynamic languages are bad"?

apower 16y ago
Cause a server with these languages has high overhead in memory and CPU to serve requests thus having lower throughput per server and thus needing more servers to sustain certain site-wide SLA and thus causing more money overall.
- blueben 16y ago
  Do you have evidence that reddit (or any site using a dynamic language) would reduce overall operating costs by switching, or is this an assumption on your part? What about development costs? What about time to deploy new features? What about the cost to make platform revisions? Do you have evidence that a non-dynamic language would really be so much faster that it would require less hardware? Perhaps the problem is not the language, but the chosen architecture? What I'm really trying to say here is... prove it.
  - SkyMarshal 16y ago
    Difficult to prove, but an observation with precedent. 37 Signals is famous for justifying Ruby by saying that hardware is cheap, developer time is not, so use RoR and just throw servers at your app till it meets your performance needs. Hardware is cheap, but maybe not cheap enough to run a site like Reddit with intentionally scant ad revenue.
    - axod 16y ago
      Hardware is cheap. Amazon hosting is not.
  - apower 16y ago
    Do you have evidence that dynamic language have less overall development cost?

jnoller 16y ago

Bad architecture or poor design crosses languages. It's entirely possible and done every single day, to emit a broken system using even the most optimal code. That said; this looks, sounds and smells like database scaling issue more than anything. It is interesting to ponder the long-term costs of running a site like this on EC2 versus your own maintained server farm.

vinhboy 16y ago
I am not familiar with the cost of running a high profile website, is the price tag for Reddit considered a lot, or little?
- axod 16y ago
  A very lot.
steveklabnik 16y ago
At least a few months ago, here's Reddit's architecture: http://blog.reddit.com/2010/03/and-fun-weekend-was-had-by-all.html http://blog.reddit.com/2010/03/and-fun-weekend-was-had-by-al...

tl 16y ago

Here's an anecdote for you: our corporate intranet site gets a peak spike of a few thousand users every day, around lunch. (People browsing the web over lunch and the computers in-house all have the site as their fixed homepage.) That web server is running separate from databases and other resources. It never sees load. We're using a static language, but even if ruby was 10x slower, it wouldn't matter. On the other hand, change the contents of a stored procedure so that it doesn't line up with indexing properly, and page load goes from >0.1s to <4s.

emmett 16y ago

14.3 million pageviews per day comes to 163 per second, on average. Reddit's traffic, like most sites of its kind, probably peaks during lunch hour during the workday and probably peaks at 5x that number. So we're looking at 815 pageviews per second. Given 80 servers total, we're looking at 10 pageviews per second per server. Let's compare this with Facebook. In October 2009, Facebook announced they had 30k servers[0]. In September 2009, there was a rough estimate that Facebook served 200 billion pageviews per month[1]. That implies 73k pageviews per second, or 2 pageviews per second per server. Clearly the pageviews are a rough estimate, but even if facebook served 1 trillion pageviews per month they still wouldn't be beating reddit for efficiency. I have a feeling if you run the numbers for any other highly dynamic site at scale, you'll find that amortized over every server in use, you won't get a lot better than 10 pageviews per second. [0] http://www.datacenterknowledge.com/archives/2009/10/13/facebook-now-has-30000-servers/ http://www.datacenterknowledge.com/archives/2009/10/13/faceb... [1] http://www.businessinsider.com/googles-estimates-for-facebooks-traffic http://www.businessinsider.com/googles-estimates-for-faceboo...

moonpolysoft 16y ago
That's terribly misleading, though. Facebook has huge clusters for performing offline business analytics. Therefore comparing them to Reddit, which presumably has a much smaller offline analytics backend, is not an apples to apples comparison in terms of efficiency.
- mrkurt 16y ago
  It's an educated guess, but I suspect comparing the entire infrastructure of a site like reddit with the entire infrastructure of a site like Facebook is pretty fair. Neither of them are using their entire server capacity to serve traffic. reddit's at a smaller scale than Facebook, obviously, but it's a pretty apt comparison.
  - invisible 16y ago
    You think the functionality of Facebook compares equally to that of Reddit computational-wise? I mean, I know neither is the most complicated thing ever, but does Reddit even allow photos anywhere besides thumbnails? Does Reddit find things only specific to your account and show you a list of friends and their content you are allowed to see (or even something vaguely similarly complicated)? I also highly doubt that Facebook counts all of the ajax-goodness it delivers as pageviews. I could really go on for a long time (widgets, Facebook connect, Facebook Apps API...) about the huge differences that you glazed over since it's using PHP for the frontend.

teej 16y ago

I ran a web app with identical pageview numbers using Ruby on Rails and EC2. My server costs were 10x cheaper than what reddit is paying. Scaling is caching and architecture, not writing your app in Java.

awa 16y ago
It would be nice if you elaborate a little.
- teej 16y ago
  I'm not sure what you want to hear, so I'll elaborate a lot. Warbook was a Facebook application written in Ruby on Rails I ran by myself in late 2007 - 2008. It grew to over 16 million pageviews a day. At the time it was more pageviews than Twitter. I scaled it using the following stack: Perlbal for load balancing, LightHTTPD for static assets, Mongrel for dynamic requests, Memcached for caching, and MySQL for relational data storage. I used two medium instances for load balancing, one medium instance for asset hosting, 15 small instances for mongrel, one XL instance for memecached, and one XL instance for MySQL. I used memcached as a "write-through" cache. Everything in cache was considered fresh. Every write of a cachable object would write to both MySQL and memcached. Every read of a cachable object would start with memcached first and failover to MySQL. This reduced reads on the database by 95%. Total hosting costs were ~$2,000 a month.
  - mdolon 16y ago
    If you don't mind me asking what happened to it? Facebook now reports about 5,500 monthly users. I'm guessing you were acquired by SGN but I'm don't see why that would cause traffic to drop so much.
    - teej 16y ago
      SGN switched their core focus from Facebook games to iPhone games in mid 2008. At that time, they droped support for all of their Facebook properties except (fluff)Friends. Games on the Facebook platform are doomed to rapid traffic loss without constant adaptation and viral tuning. Even with it, user retention is hard. Facebook giveth and Facebook taketh away.

Devilboy 16y ago

I run 2 Reddits on my calculator watch using C++ and CouchDB so clearly the technology is at fault here.