10 ms·
Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video]
Jeremy Edberg shares some of the lessons learned scaling Reddit, advising on pitfalls to avoid.
- endlessvoid94 13y agoThrilled to finally see this on InfoQ -- already an underappreciated resource for technical talks. What is it about video + slides that appeals so little to people?
- kalleboo 13y agoThe video is 40 minutes long. If this was a blog post it'd take me maybe 5-15 minutes to read/skim.
- coolsunglasses 13y agoSite kinda sucks and is annoying to use for those of us that prefer textual resources and the ability to flip through slides ourselves without clicking around a teensy tiny 1-px sized gaps between slides.
- ics 13y agoLarge banner and tiny video frame, site rendering completely garbled for 8 seconds until fully loaded, signup required to view the slides (after realizing that the video doesn't cut to slides at all). The talks are interesting, and the interview transcription is nice though there are UX issues there as well– I need to click to view each response, and despite the entire question being hyperlinked clicking it actually does nothing. Show all works thankfully (ah, but the frame breaks my scroll). I'll probably be back to check out more of the videos, but definitely not because of the site. If the editing is good, YouTube is just fine, otherwise SlideShare plus an audio file is just perfect. Edit: [F] First Byte Time [C] Keep-alive Enabled [C] Compress Transfer [A] Compress Images [A] Progressive JPEGs [F] Cache static content [ ] Effective use of CDN Source: http://www.webpagetest.org/result/130816_PE_AYH/ http://www.webpagetest.org/result/130816_PE_AYH/
- masklinn 13y agoI will just comment in full support of this, it says everything.
- masklinn 13y ago> SlideShare plus an audio file is just perfect. Although slideshare is not as bad as infoq by a long shot, it's not very useable (or fast) either. I much prefer speakerdeck.
- ics 13y agoSpeakerdeck is actually what I was thinking of, but the gist is slides + audio = happy. A more polished UI would do some good for SlideShare, but I don't have any major issues.
- AndrewDucker 13y agoI can read a slide deck at a rate of 5s/slide (or faster), pausing to concentrate on the ones that interest me. I don't have the spare time to watch a video of someone, just in case the topic is interesting.
- justin66 13y agoCreating a video on youtube and linking to a pdf with the slides is easy enough to do. The value add of having the website procedurally flip slides for me is very, very small.
- codeka 13y ago> What is it about video + slides that appeals so little to people I don't have a solid block of 40 undisturbed minutes to listen to a talk. Give me a transcript and I can read a paragraph here and there as I do other things at my own pace. I might have ten minutes here, ten minutes there. I don't want to be constantly pausing/unpausing the video, or worse - switching between the video and my music. Plus, if I concentrate, I could read a 40 minute talk in 20 minutes or less. Basically, when I'm reading, I control the pace. I rarely watch videos that are longer than about 5 minutes (that aren't entertainment, which is entirely different).
- Terretta 13y agoWhat is it about video + slides that appeals so little to people? A full transcript with interleaved slide images takes a few minutes to read at most, and lets you control the pace of information absorption. A video with slides, especially when you cannot 2x the talking speed, is a painfully slow data transfer method. Video + Slides = analog modem Transcript w/ embedded slides = Google fiber :-) Of course, if you're into audio books instead of reading, maybe you consider that a feature. // I push video for a living. It's great for visual explanation like DIY instruction (e.g. woodworking, swapping RAM on a Mac Mini), emotional content, personal story telling, etc. Systems architecture is generally not in one of those categories.
- itry 13y agoWhat a horrible website. Made an account to get the mp3. Just to discover that the mp3 download link doesnt work even then. Thank god I used a throwaway email to sign up. Otherwise I would be even more pissed.
- paraxisi 13y agohttp://bugmenot.com/view/infoq.com http://bugmenot.com/view/infoq.com Bugmenot is great for sites like this.
- wjk 13y agohttp://10minutemail.com/ http://10minutemail.com/ is what i use, much simpler.
- amitparikh 13y agoAnother similar option is Mailinator: http://mailinator.com/ http://mailinator.com/
- Houshalter 13y agoSome websites block emails from mailinator which is pretty annoying.
- antsar 13y agoI wouldn't call that simpler. You have to register using a fake email, possibly actually go check the inbox and click a link, and possibly be required to log in again now that the account is registered. With bugmenot, all you do is copy and paste a valid username and password.
- oinksoft 13y agoFor Firefox users, made even smoother with: https://addons.mozilla.org/en-us/firefox/addon/bloody-vikings/ https://addons.mozilla.org/en-us/firefox/addon/bloody-viking...
- 13y ago
- diminish 13y agoI am curious how much a Go lang rewrite would make a difference in scaling up.
- noelwelsh 13y agoI laughed. Deft trolling. Bravo.
- solistice 13y agoHe forgot building the 2012 Servers outlined in Jeff Atwoods blog post "Building Servers for Fun and Prof... OK, Maybe Just for Fun". Gotta do that for bleeding (profusely) edge power.
- booop 13y agoWhy? Looks like a legitimate question to me.
- weego 13y agoPerhaps so, but then it would fit into the category of 'unaware of being a parody of itself' statements/questions that are also (unintentionally) humorous such as the classic "I can tell that site was built in rails from the design".
- noelwelsh 13y agoSome time ago the HN hivemind went through a period of blind Node.js love. Now it's going through blind Go love. Regardless of what you think of either language, "rewrite in X" is not a magic incantation that will spontaneously solve all your architectural issues. Designing a good architecture involves balancing many components, of which your primary implementation language is an important, but not exclusive, element. There are also the organisational issues -- hiring, spending time not adding new features, etc.
- deleted 13y ago[deleted]
- OGC 13y agoReddit's not profitable though..
- chroem 13y agoIf there is one lesson that HN needs to learn, it's that profitable is not the same thing as important.
- FooBarWidget 13y agoSo they are important, but need to live off ramen every day (figuratively). What good does that do to them? I don't think grandparent should be downvoted, he raises a good point. Tons and tons of people use Reddit, but Reddit has a hard time making a living.
- paulhauggis 13y agoWithout cash flow, how far can the site go? Someone needs to pay the bandwidth/server bill. Not to mention it costs time to run a site. If you are working full-time at another job (because you aren't bringing any money in), you won't have any time to work on it. Reddit succeeded largely because the company that bought them is making money elsewhere.
- straight_talk_2 13y ago>> Reddit succeeded largely because the company that bought them is making money elsewhere. Such a nonsense makes me angry. Surely you've heard about companies called Google, Facebook, Linkedin, that succeeded without being bought and for a long time not making a profit.
- perlpimp 13y agoFor me link works only in safari. Firefox plays only video and chrome just does not work.
- grandpoobah 13y agoReddit still uses Pylons, and I imagine moving to Pyramid would be very painful. He's not clear on this in the video.
- jedberg 13y agoI don't think I made any indications one way or another, but yes, it would probably be difficult to move.
- grandpoobah 13y agoGreat presentation man. Do you think Pylons will serve Reddit for years and years to come? Is there any need for you to switch to Pyramid or another framework? I made the choice to use Pylons for a recent project, and it just feels kind of odd using an old framework which is now in "maintenance only mode", but I truly did not like Pyramid... much less Django.
- billb2112 13y ago"ask me anything about running a profitable social media company". Except for that reddit is not profitable. http://blog.reddit.com/2013/08/reddit-myth-busters_6.html http://blog.reddit.com/2013/08/reddit-myth-busters_6.html http://www.reddit.com/r/TheoryOfReddit/comments/1ihwy8/ratheism_and_rpolitics_removed_from_default/cb50c8c http://www.reddit.com/r/TheoryOfReddit/comments/1ihwy8/rathe...
- jedberg 13y agoFunny story about that. When we were owned by Conde, the accounting was a little different (they took on some of the charges, like Akamai), and so we were actually told we were slightly profitable. When that blog post went up I was as surprised as you to see it wasn't profitable when I was there.
- danilocampos 13y agoBut weren't you "deeply involved in the business?"
- jedberg 13y agoYes. What point are you trying to make?
- 300bps 13y agoI think his point is that he doesn't understand accounting and how complicated it can be so he thinks that if you were deeply involved in the business you should know every facet of every debit and credit. If he's right I guess I wasted about 300 hours studying for Financial Reporting and Analysis section in the Chartered Financial Analyst (CFA) program.
- jedberg 13y agoSince this isn't reddit, I won't just say: ^^this. But yes, I believe he was trying to point out that I couldn't have been involved if I didn't know, but doesn't understand the ins and outs of G&A and other such accounting practices.
- russelluresti 13y agoDisappointment in the HN community has reached a new high today. So, instead of discussing the topics in the video, the majority of commenters here are discussing the flaws of the website it's hosted on or debating whether or not reddit is profitable. Neither of which has ANYTHING to do with scaling. I expected better, people. Seriously.
- diminish 13y agoThe dominance of non-relevance is interesting for HN. Was writing exact the same thing, and just saw yours. In this and some other technical topics, people end up discussing their personal tastes with web site's design, their individual UI frustrations with some button on the web site, the font, the color, and other random non-relevant topic; like now the profitability.
- kerkeslager 13y agoIt's no excuse, but I see a reason for this: most people feel they understand these irrelevant topics better than they understand the scalability features of the infrastructure Reddit has built. People talk about their comfort zone.
- dpatrick86 13y agoSpot on observation IMO
- alanctgardner2 13y agoThis is the fundamental rule of bikeshedding, which HN has discussed, decried and regurgitated since time immemorial.
- gwu78 13y agoMaybe they can't see the video to begin with because of excessive web advertising ploys. That might explain why they are not commenting on the video. Just a guess.
- heed 13y ago
- gwu78 13y agohttp://d1snlc0orfrhj.cloudfront.net/presentations/13-jul-scalingreddit.mp4 http://d1snlc0orfrhj.cloudfront.net/presentations/13-jul-sca...
- dap 13y agoI was confused about the memcached problem after moving to the cloud. I understand why network latency may have gone from submillisecond to milliseconds, but how could you improve latency by batching requests? Shouldn't that improve efficiency, not latency, at the possible expense of latency (since some requests will wait on the client as they get batched)? And while maybe efficiency is valuable, why would that be an improvement for a problem they didn't have before?
- jedberg 13y agoSorry that wasn't clear. The latency didn't get better, but what happened is that instead of having to make a lot of calls to memcache it was just one (well, just a few), so while that one took longer, the total time was much less.
- robotmay 13y agoI actually did some (simplistic) examples of this in a small presentation to illustrate the performance improvements of batching memcached requests, if anyone's interested: https://speakerdeck.com/robotmay/a-simple-introduction-to-effective-caching-in-ruby-on-rails?slide=11 https://speakerdeck.com/robotmay/a-simple-introduction-to-ef... (slides 11 to 14)
- jedberg 13y agoThat's a better explanation than mine. :) Thanks for the link.
- deleted 13y ago[deleted]
- Dave_Rosenthal 13y agoThough it might not be the situation in the video, introducing batching can decrease system latency. I made this little graph to show how this can work: http://imgur.com/Vr6BqDq http://imgur.com/Vr6BqDq
- mr_luc 13y ago
- hardwaresofton 13y agoSo did anyone else take the most important lessons in the video as: 1. Use AWS 2. Use Postgres 3. Use AWS 4. Use Cassandra 5. Use python, so later you can write C when shit needs to go super fast That's what I got.
- jedberg 13y agoI find this interesting, this is really great feedback for me actually. Those are some of the important lessons, although use (postgres|cassandra) are really too prescriptive. More like "use the right tool or tools for the job". Also, use consistent key hashing where appropriate is another important less that I should emphasize more. And "build for 3" is another important lesson. It makes scaling much easier.
- heed 13y agoSide question: is vote queuing the reason behind the sometimes large drops in score on highly active and popular submissions on reddit?
- jedberg 13y agoNo. That's generally because once something gets popular and jumps to the front page, it gets a huge boost in visibility, especially from people who weren't looking at the niche subreddit it comes from. A lot of those people aren't interested in that content, so it will suddenly get an influx of downvotes.
- heed 13y agoThanks for the reply. I'm glad there is an explanation based on user behavior for this phenomenon because admin level vote tampering is such a tired theory.
- conradfr 13y agoI wonder if the demise of Digg three years ago and the (supposedly) inflow of new users have been problematic at the time.
- jonursenbach 13y agoMost likely as Reddit had continued downtime around that time with EC2/EBS scaling issues.
- jedberg 13y agoThere wasn't actually a very large jump in traffic when Digg v4 was launched. Most of those folks were already reddit users. Traffic bumped a little bit, but not all that much. It's important to keep in mind that reddit was already doing twice as much traffic as Digg before they launched v4.
- conradfr 13y agoThanks for your input, didn't know that. I, for one, went from a /r/php lurker to real user at that time so I thought there was dozens of us, dozens !
- vegasje 13y agoDoes anyone know how the different storage systems are utilized, and why each system is utilized for that purpose? The presenter mentions using memcached, Cassandra, and PostgreSQL, and mentions the same type of data when discussing each (votes, for instance). I would definitely benefit from a more in-depth understanding of how each system is utilized, and why.
- jedberg 13y agoEach tool has a different use case. Votes is a great example. Memcache has no guarantees about durability, but is very fast, so the vote data is stored there to make rendering of pages as quick as possible. Cassandra is durable and fast, and gives fast negative lookups because of its bloom filter, so it was good for storing a durable copy of the votes for when the data isn't in memcache. Postgres is rock solid and relational, so it was a good place to store votes as a backup for Cassandra (we could regenerate all the data in Cassandra from Postgres if necessary) and also for doing batch processing, which sometimes needed the relational capabilities.
- vegasje 13y agoThat makes a lot of sense. Were the majority of your systems using this "durability chain" so to speak -- memcached -> Cassandra -> Postgres? Additionally, in retrospect do you find this type of chain to work well, and would you use it again (perhaps you already are over at Netflix)?