5 ms·

Steve Huffman on Lessons Learned at Reddit

202 points by giu 16y ago

kingkilr 16y ago
Joe Stump gave a similar talk at EuroDjangoCon last year (the topic is officially: dealing with a fuckload of data, but a lot of the same lessons). Unfortunately I have no idea what became of the recordings.
- sumeeta 16y ago
  There’s also Scaling your Python application on EC2 from PyCon 2010: http://pycon.blip.tv/file/3257303/ http://pycon.blip.tv/file/3257303/
clemesha 16y ago
Their strong usage of both Memcached and Memcachedb makes me think of my favorite 'all-in-one' solution: Redis.
- philjackson 16y ago
  Cassandra (which they moved to in the end) is also an excellent all-in-one solution. It even supports Hadoop since 0.6.
jim-greer 16y ago
Did they say what forced them into emergency read-only mode this week?
- mr_justin 16y ago
  He says (transcript is below the video) that he left Reddit in the Fall of 2007 but continued working as a contractor until the Fall of 2009 so he doesn't really address the current events of Reddit.
  - jim-greer 16y ago
    Thanks, sorry I didn't RTFA.
  - kn0thing 16y ago
    That transcript is misleading. Steve stayed with reddit until Fall of 2009 when our contracts ended. He didn't leave to work as a contractor. One thing I should also mention is that our deal was structured in such a way that either one of us could've left reddit before our contract ended without too much pain (missing some carrots, granted) but we were still rather attached to reddit, the community, and our friends (team reddit). Personally, I did also feel a compulsion to fulfill my side of the contract and make sure Condé Nast got its money's worth. That may make me a tool in the eyes of most Hacker News folks, but so be it.
    - by 16y ago
      When you sold the site to Condé Nast they became your customer. Looking after your customers is the number one priority of any business. So it is totally correct that you made sure they received a good product and got the best from it.
    - mr_justin 16y ago
      So you're saying from the Fall of 2007 to Fall of 2009 he was a full time hire as opposed to a contractor like it says in the transcript?
akkartik 16y ago
The approach he recommends - no scheme, extreme denormalization, lots of redundancy - is a strong case for nosql.
- bh23ha 16y ago
  Shouldn't lots of redundancy be a fundamental part of which ever db you are using. As to extreme do-normalization, colour me sceptical, why not parallelise and throw more hardware at the problem?
  - dalore 16y ago
    Because it's more cost effective to de-normalize then to parallelise in this instance.
  - jbellis 16y ago
    Turns out it's difficult to parallelize without also denormalizing...
- alexro 16y ago
  Reddit in fact now runs on Cassandra - the top nosql solution for a social website
  - metachris 16y ago
    That's interesting. The guys at reddit previously compared various database systems including Amazon's SimpleDB, and found that PostgreSQL was the fastest (see 'Scaling your Python application on EC2' from PyCon 2010: http://pycon.blip.tv/file/3257303/ http://pycon.blip.tv/file/3257303/ ) Appearently Reddit switched over to Cassandra in March 2010 (http://blog.reddit.com/2010/03/she-who-entangles-men.html http://blog.reddit.com/2010/03/she-who-entangles-men.html)
    - jbellis 16y ago
      To clarify, Reddit replaced memcachedb w/ cassandra entirely, but a lot of the site is still on postgresql. My understanding is they plan to move more to Cassnadra gradually.
bryanh 16y ago
So many lectures on scaling are full of best-practices that are somewhat removed from reality and time/budget constraints. This lecture on the other hand has a lot of common-sense advice and solutions that may be obvious to some, but I must admit, this is going to help me (if I could get the traffic to worry about scaling...).
0nly1ife 16y ago
Interesting. I'm an indie Facebook developer. My most popular app has over 8 million monthly uniques and 350 million pageviews. I didn't realize how substantial said traffic was until reading this article. I scaled my app to that size on my own over the course of 12 months. I know of other FB devs who scaled bigger apps working solo. If average hackers like us can do it, anyone can. The hard part is building a popular app. Articles about scaling are a means for founders to brag about their success without giving away useful information about what made their app popular in the first place.
- staunch 16y ago
  Alexis Ohanian has talked about what made Reddit successful many times. Steve Huffman is talking about the technical side of things. It doesn't come off as bragging to me at all.
  - 0nly1ife 16y ago
    Bragging is the wrong word, but it is the first that came to mind. And I was excited to brag about the size of my app when I first read his article. I just think we spend too much time talking about scaling and not enough time discussing virality.
    - strlen 16y ago
      Scalability stories are generally interesting, as they're about solving problems that out-of-the-box commercial software doesn't. They're interesting to hackers, not all of whom are entrepreneurs. If you'd like to share a story, however, on using statistical techniques to measure and act on virality, doing multi-variate testing, et al that would be very interesting.
- percept 16y ago
  Care to blog about your experiences?
- clemesha 16y ago
  Wow, it sounds like your app is extremely popular - that sounds like huge traffic, especially for "scaling to that size in 12 months". Right? I am naive, or is this "fairly common" with decent Facebook games? What is your application? Are you profitable? Very profitable? (Oh god I hope you are very profitable, or some of my apps are screwed - but then again I guess it really depends on the business model and usage characteristics)
niktech 16y ago
Direct link to an iPhone watchable video: http://vimeo.com/channels/carsonifiedtv#10506751 http://vimeo.com/channels/carsonifiedtv#10506751
dimarco 16y ago
a note for Mixergy: this is how you provide a video transcript.
- AndrewWarner 16y ago
  Checking it out now. Thanks Mark.
evgen 16y ago
Let's see: - crash and let a supervisor restart you - isolate components via specialist processes and systems - keep schema/types flexible - be stateless - cache & denormalize - be redundant - let other processes handle non-realtime tasks in the background Sounds like he spent a long time learning the hard way that reddit should have been written in Erlang in the first place :)
- Hexstream 16y ago
  Well, they use (or used to use) RabbitMQ, which is written in Erlang.

202 points by giu 16y ago

Joe Stump gave a similar talk at EuroDjangoCon last year (the topic is officially: dealing with a fuckload of data, but a lot of the same lessons). Unfortunately I have no idea what became of the recordings.

sumeeta 16y ago
There’s also Scaling your Python application on EC2 from PyCon 2010: http://pycon.blip.tv/file/3257303/ http://pycon.blip.tv/file/3257303/

clemesha 16y ago

Their strong usage of both Memcached and Memcachedb makes me think of my favorite 'all-in-one' solution: Redis.

philjackson 16y ago
Cassandra (which they moved to in the end) is also an excellent all-in-one solution. It even supports Hadoop since 0.6.

jim-greer 16y ago

Did they say what forced them into emergency read-only mode this week?

mr_justin 16y ago
He says (transcript is below the video) that he left Reddit in the Fall of 2007 but continued working as a contractor until the Fall of 2009 so he doesn't really address the current events of Reddit.
- jim-greer 16y ago
  Thanks, sorry I didn't RTFA.
- kn0thing 16y ago
  That transcript is misleading. Steve stayed with reddit until Fall of 2009 when our contracts ended. He didn't leave to work as a contractor. One thing I should also mention is that our deal was structured in such a way that either one of us could've left reddit before our contract ended without too much pain (missing some carrots, granted) but we were still rather attached to reddit, the community, and our friends (team reddit). Personally, I did also feel a compulsion to fulfill my side of the contract and make sure Condé Nast got its money's worth. That may make me a tool in the eyes of most Hacker News folks, but so be it.
  - by 16y ago
    When you sold the site to Condé Nast they became your customer. Looking after your customers is the number one priority of any business. So it is totally correct that you made sure they received a good product and got the best from it.
  - mr_justin 16y ago
    So you're saying from the Fall of 2007 to Fall of 2009 he was a full time hire as opposed to a contractor like it says in the transcript?

akkartik 16y ago

The approach he recommends - no scheme, extreme denormalization, lots of redundancy - is a strong case for nosql.

bh23ha 16y ago
Shouldn't lots of redundancy be a fundamental part of which ever db you are using. As to extreme do-normalization, colour me sceptical, why not parallelise and throw more hardware at the problem?
- dalore 16y ago
  Because it's more cost effective to de-normalize then to parallelise in this instance.
- jbellis 16y ago
  Turns out it's difficult to parallelize without also denormalizing...
alexro 16y ago
Reddit in fact now runs on Cassandra - the top nosql solution for a social website
- metachris 16y ago
  That's interesting. The guys at reddit previously compared various database systems including Amazon's SimpleDB, and found that PostgreSQL was the fastest (see 'Scaling your Python application on EC2' from PyCon 2010: http://pycon.blip.tv/file/3257303/ http://pycon.blip.tv/file/3257303/ ) Appearently Reddit switched over to Cassandra in March 2010 (http://blog.reddit.com/2010/03/she-who-entangles-men.html http://blog.reddit.com/2010/03/she-who-entangles-men.html)
  - jbellis 16y ago
    To clarify, Reddit replaced memcachedb w/ cassandra entirely, but a lot of the site is still on postgresql. My understanding is they plan to move more to Cassnadra gradually.

bryanh 16y ago

So many lectures on scaling are full of best-practices that are somewhat removed from reality and time/budget constraints. This lecture on the other hand has a lot of common-sense advice and solutions that may be obvious to some, but I must admit, this is going to help me (if I could get the traffic to worry about scaling...).

0nly1ife 16y ago

Interesting. I'm an indie Facebook developer. My most popular app has over 8 million monthly uniques and 350 million pageviews. I didn't realize how substantial said traffic was until reading this article. I scaled my app to that size on my own over the course of 12 months. I know of other FB devs who scaled bigger apps working solo. If average hackers like us can do it, anyone can. The hard part is building a popular app. Articles about scaling are a means for founders to brag about their success without giving away useful information about what made their app popular in the first place.

staunch 16y ago
Alexis Ohanian has talked about what made Reddit successful many times. Steve Huffman is talking about the technical side of things. It doesn't come off as bragging to me at all.
- 0nly1ife 16y ago
  Bragging is the wrong word, but it is the first that came to mind. And I was excited to brag about the size of my app when I first read his article. I just think we spend too much time talking about scaling and not enough time discussing virality.
  - strlen 16y ago
    Scalability stories are generally interesting, as they're about solving problems that out-of-the-box commercial software doesn't. They're interesting to hackers, not all of whom are entrepreneurs. If you'd like to share a story, however, on using statistical techniques to measure and act on virality, doing multi-variate testing, et al that would be very interesting.
percept 16y ago
Care to blog about your experiences?
clemesha 16y ago
Wow, it sounds like your app is extremely popular - that sounds like huge traffic, especially for "scaling to that size in 12 months". Right? I am naive, or is this "fairly common" with decent Facebook games? What is your application? Are you profitable? Very profitable? (Oh god I hope you are very profitable, or some of my apps are screwed - but then again I guess it really depends on the business model and usage characteristics)

niktech 16y ago

Direct link to an iPhone watchable video: http://vimeo.com/channels/carsonifiedtv#10506751 http://vimeo.com/channels/carsonifiedtv#10506751

dimarco 16y ago

a note for Mixergy: this is how you provide a video transcript.

AndrewWarner 16y ago
Checking it out now. Thanks Mark.

evgen 16y ago

Let's see: - crash and let a supervisor restart you - isolate components via specialist processes and systems - keep schema/types flexible - be stateless - cache & denormalize - be redundant - let other processes handle non-realtime tasks in the background Sounds like he spent a long time learning the hard way that reddit should have been written in Erlang in the first place :)

Hexstream 16y ago
Well, they use (or used to use) RabbitMQ, which is written in Erlang.