17 ms·
Tumblr Architecture - 15B Page Views a Month and Harder to Scale than Twitter
- retroafroman 15y agoThis may be a slightly naive question, but why list both "500 web servers" and "8 nginx"? Seeing as nginx IS a webserver, is it fulfilling a different function (serving static pages?) compared with the Apache servers?
- riledhel 15y agoMaybe the nginx servers are configured to serve static content, as reverse proxies or load balancers.
- huxley 15y agoNginx can also act a proxy for mail protocols: http://wiki.nginx.org/MailCoreModule http://wiki.nginx.org/MailCoreModule
- simonw 15y ago$ curl -i 'http://www.tumblr.com/ http://www.tumblr.com/ HTTP/1.1 302 Found Date: Mon, 13 Feb 2012 19:31:35 GMT Server: Apache P3P: CP="ALL ADM DEV PSAi COM OUR OTRo STP IND ONL" Location: https://www.tumblr.com/ https://www.tumblr.com/ Vary: Accept-Encoding X-Tumblr-Usec: D=15547 Content-Length: 0 Connection: close Content-Type: text/html; charset=UTF-8 $ curl -i 'http://assets.tumblr.com/images/favicon.gif?2 http://assets.tumblr.com/images/favicon.gif?2 HTTP/1.1 200 OK Server: nginx/0.8.53 Content-Type: image/gif Last-Modified: Fri, 15 Apr 2011 22:13:30 GMT Accept-Ranges: bytes Content-Length: 635 X-Varnish: 1795992965 Cache-Control: max-age=2196224 Expires: Sat, 10 Mar 2012 05:35:30 GMT Date: Mon, 13 Feb 2012 19:31:46 GMT Connection: keep-alive So it looks like their application is being served by Apache (and PHP), while their static assets are served by nginx behind Varnish.
- retroafroman 15y agoGood catch, that's along the lines I was thinking
- silverlight 15y agoWell, their assets are begin served from a CDN. Are they counting that in their server count? $ host assets.tumblr.com assets.tumblr.com is an alias for assets.tumblr.com.edgesuite.net. assets.tumblr.com.edgesuite.net is an alias for a1092.g.akamai.net. a1092.g.akamai.net has address 69.31.106.32 a1092.g.akamai.net has address 69.31.106.50
- vintagius 15y agoIts 2012 and we still embrace high Pageviews and architectures to handle them but ignore revenue. Its high time everypost about insane pageviews and servers be coupled with solid revenue numbers.
- citricsquid 15y agoThis is a post about architecture and scalability, not business.
- jcampbell1 15y ago15B pageviews with modest ad monetization would yield something around $15 million per month.
- streptomycin 15y ago$1 CPM is not modest, depending on the website. Many sites make much less than that.
- sanswork 15y agoIndeed $1CPM for this type of site would be quite high.
- ceejayoz 15y agoIt's a fairly shaky assumption that an engineer putting together a MySQL sharding presentation would know the revenue numbers for the company.
- tomkarlo 15y agoEven if they did, they shouldn't be disclosing them.
- epc 15y agotumblr gets a cut whenever someone "buys" a tumblr design. Have no idea what that equates to in revenue but they are getting revenue.
- cagenut 15y agoTumblr is an extremely pageview heavy design, but the industry has moved away from PV as an important metric for a few years now. Fortunately they're nice enough to post their quantcast data publicly: http://www.quantcast.com/p-19UtqE8ngoZbM http://www.quantcast.com/p-19UtqE8ngoZbM They're still phenomenal numbers, but IMHO should be much closer to a 100-server environment than a 1000 server one.
- jonknee 15y ago100 servers for billions of requests per day seems crazy optimistic. It sounds like they're running pretty lean as it is.
- NyxWulf 15y agoIt all depends on the nature of the request. Our nginx servers are tuned to burst to 7000 requests/second and run stable at 4800 requests/second for simple content. Our overall scale is about 1/5th of what Tumblr is doing daily and we have a much smaller infrastructure footprint. Although in fairness our content is no where near as dynamic as theirs is.
- sanswork 15y agoI've done ~1b/day on around 6-10 1gb/gogrid instances though admittedly the complexity was lower(ad serving platform) though not just a proxy/static server. When I read this I actually messaged my partner "Imagine what we could do with 1000 servers". I imagine a lot of those servers aren't directly related to serving the site though. The number of support servers required is usually way underestimated.
- encoderer 15y agoI think you're on to something when you contrast the 2 problem domains. Number of requests is a very naive way to look at load factors. At the startup I work, we've got 25-30 Million users, many stats similar to Tumblr, and we're running it on about 250 Ec2 instances of varying size. I think if Tumblr's numbers are high at all -- due to rapid iterations and no time to focus on deep optimizations -- it's maybe 10-15% high, not 90%. I'm saying this because I've seen periods where our usage numbers are somewhat flat, even falling, but our hardware demands rise as we provide more features. When there's just one or two primary ways to use a service (eg "I post status updates and comment on my friends' status updates") it can be quite easy to optimize. But add features. Photos. Chat. An in-house ad serving platform. I18N. Etc. You have different types of interactions with different acceptable service levels and varying storage requirements.
- nessus42 15y agoMy favorite part of the blog post: "Don’t hire people based on their survival through a useless technological gauntlet. Hire them because they fit your team and can do the job."
- rhizome 15y agoThose don't appear mutually exclusive to me.
- michaelbuckbee 15y agoNot mutually exclusive, but in some places, maybe not highly correlated either.
- nessus42 15y agoThey aren't. And neither is hiring only people with red hair.
- ArbitraryLimits 15y agoWell, if you remove the word "useless."
- cshesse 15y agoEspecially if the job is to survive a useless technological gauntlet.
- jebblue 15y agoAgreed and to me it sounds incredibly short sighted. People who have survived technological gauntlets know how to adapt and help instill a sense of continuity in fellow team members.
- nessus42 15y agoI, for the life of me, don't understand why people defend these gauntlets. Wasn't getting a degree from MIT enough of a gauntlet? Wasn't writing reams and reams of working, robust, well-documented code proof enough of one's software engineering skills? I think so!
- wtn 15y agoDo sites like Tumblr use graph databases?
- jrockway 15y agoWhat questions do they need to answer that require the entire graph? It seems like the most complex thing they may need to calculate is friends-of-friends, and that's easy to do even with an adjacency list in a SQL database.
- B-Scan 15y agoAfter fairly bad review about Tumblr's availability [1] this was something they needed to write. They had (and still have) pretty big challenges. It's not easy to scale something like that and this article confirms that. [1] http://news.ycombinator.com/item?id=3468879 http://news.ycombinator.com/item?id=3468879
- ck2 15y agoof the 500+ million page views a day, 70% of that is for the Dashboard "dashboard" is their admin area? So in theory 700 of their 1000 servers are for people to make a new post? If I remember correctly, wp.com has a similar problem.
- Avshalom 15y agoThe dashboard also aggregates all the posts from people you follow.
- jonathanmoore 15y agoThe dashboard is where you primarily view all of the posts from the people you follow on Tumblr. Think of it in terms of what you view on Twitter, all of the Tweets from people you follow plus the ability to send out a new Tweet.
- citricsquid 15y agoA relevant comment of my own from a few weeks ago: http://news.ycombinator.com/item?id=3468904 http://news.ycombinator.com/item?id=3468904 Tumblr is not a blogging platform like Wordpress, it's much more a community like Reddit. Yes, you can use Tumblr to host your blog, but the majority of people use the dashboard to interact with others and use their blog to share their content. Very few people will ever see username.tumblr.com, they'll see the posts via tumblr.com/dashboard. Like Twitter, very few people visit twitter.com/citricsquid, they follow me and see my tweets in their stream. For example, my dashboard right now: http://i.imgur.com/0YAYv.jpg http://i.imgur.com/0YAYv.jpg (potentially nsfw material, didn't check)
- skeletonjelly 15y agoThe list is NSFW depending on where you work for the curious. So I guess the dashboard is the feed. Not a graph based dashboard, nor an admin section.
- thesethings 15y agoDashboard is more like a "reader." Everybody you follow is found there. Also any tags you subscribe to are there. Interactions like replies are also found there. I'd compare it to your Facebook feed or Twitter.com stream. It's your base of operations. Lots of read-only content, and then a decent amount of interactions and events (messages/new follows/like notifications).
- feralchimp 15y ago> Posts are about 50GB a day. Follower list updates are about 2.7TB a day. Holy schnikeys! The graph changes are two orders of magnitude larger than the content additions.
- thesethings 15y agoThat caught my eye too, and I guessed on Twitter that it might be a typo/under-explained. Somebody from Tumblr (though on a different team) was nice enough to respond and guessed that it was because all media on S3 was not counted. https://twitter.com/#!/adamlaiacano/status/169140791732740096 https://twitter.com/#!/adamlaiacano/status/16914079173274009...
- mcherm 15y agoThanks. It seemed completely unreasonable to me until I saw this explanation.
- harryh 15y agoThis was very surprising to me as well. Assuming that IDs are 8 bytes (they appear to be) and that the mean post is 250 bytes (which seems conservative to me) then the mean fanout is something like 1600. That's big.
- joshu 15y agosounds like they have some twitter envy?
- its_so_on 15y agoI'm sorry, but I have a hard time believing that anything is harder to scale than Twitter. With an audience of 175 million users, a Twitter field of 140 bytes makes for 22 gigabytes of uncompressed text per person-tweet. If everyone Tweets ten tweets per hour, that's 220 gigabytes. A one-TB HD would only be good for 5 hours or so, meaning Twitter would need to buy a good 3000 hard-drives per year. That's probably more than Amazon has in total.
- recursive 15y ago> If everyone Tweets ten tweets per hour Well, luckily for them, that's not actually the case.
- sakai 15y agoYa... and Amazon almost certainly has more than 3000 TB of storage (3 PB).
- its_so_on 15y agoI was being so extremely sarcastic to show how ridiculous it is to say "harder to scale than Twitter". Of course there are nowhere near 175 million users, of course they don't tweet ten tweets per hour, and of course the tweets aren't uncompressed. My point is that even with all these assumptions, you have a ridiculous lack of scaling problems. My mention of amazon was meant to show that I was absolutely 100% being completely sarcastic when talking about Twitter's "scaling" problems.
- whatusername 15y ago
- DLarsen 15y ago> Initially an Actor model was used with Finagle, but that was dropped. The Akka folks do a good job of keep the actor model on your mind when building Scala systems, but it's obviously not always the right approach. I'd love to hear more about how they ended up abandoning the Actor model.
- famoreira 15y agoIt seems that the only way to scale a site like this is to build decoupled services maybe using a messaging queue (AMQP, etc..). How can I learn more about this kind of systems? Also is there a open source application[s] built in a distributed way that I could learn from?
- runjake 15y agoI hate to point out the obvious answer, but it's a good introduction with links to several implementations: http://en.wikipedia.org/wiki/AMQP http://en.wikipedia.org/wiki/AMQP
- famoreira 15y agoI meant some open source applications that are built in a distributed way (hence using something like AMQP in a publish/subscribe fashion)
- rabidsnail 15y agoI don't really know of any. Nobody writes these sort of distributed applications from scratch. They're almost always evolved from monolithic database-backed applications. And the process of evolution is half development half ops. I doubt that tumblr as it exists today could be deployed from scratch, even if you had all of the source code of all of the systems.
- rabidsnail 15y agoHowever, if you want to go truly distributed (much more so than any website I know of) there are open source projects you can learn from: http://freenetproject.org/ http://freenetproject.org/ http://amule.org http://amule.org (or http://mldonkey.sourceforge.net/Main_Page http://mldonkey.sourceforge.net/Main_Page if you're into functional programming) http://www.torproject.org http://www.torproject.org (especially the hidden services stuff) And you probably want to read this: http://pmg.csail.mit.edu/papers/osdi99.pdf http://pmg.csail.mit.edu/papers/osdi99.pdf
- matth 15y agoOff-topic, but I put together a list of HN members on Tumblr not too long ago: http://blog.dozierhudson.com/post/9596967319/list-of-hacker-news-members-on-tumblr http://blog.dozierhudson.com/post/9596967319/list-of-hacker-... Get in touch and I'll add you to the list.
- willvarfar 15y agohttp://williamedwardscoder.tumblr.com http://williamedwardscoder.tumblr.com
- fs111 15y agohttp://notes.kel.pe http://notes.kel.pe
- deleted 15y ago[deleted]
- mnutt 15y agoFor 500M req/day, that's 6000 req/sec on average. That could plausibly fit with 40,000 req/sec peak.
- fooandbarify 15y agoI don't think you should have been downvoted, but you certainly missed something. days != seconds
- democracy 15y agoA bit surprising is the tone of apology when touching JVM even with Scala gloves.
- soc88 15y agoReally? Didn't see that. Is this a political or a technical comment? When building highly-scalable and stable systems, there isn't much except the JVM. Sure, for some corner cases Erlang might be a solution and if reliability is less important also .NET.
- didip 15y agoI hope they managed to release staircar(http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications http://engineering.tumblr.com/post/7819252942/staircar-redis...) before phasing it out with Scala.
- Alind 15y ago"Internally they had a lot of people with Ruby and PHP experience, so Scala was appealing. " Anyone notice this ??? logical?
- spullara 15y agoScala looks a lot more like Ruby than Java does. It makes people feel more comfortable with the language. Also, a lot of the patterns, like collections, are more similar between Scala and Ruby than Java and Ruby. This will change with JDK 8, but for now it is true.
- willvarfar 15y agoWhich implies that disqus is getting hammered too. Most blogs have disqus comments on each post; every page-view on tumblr is at least one fetch on disqus; many more for index pages. Very clever strategy for tumblr to make comments someone else's problem ;)
- Avshalom 15y agoIt's not that bad. 70% of views are from the Dashboard, they don't see disqus, and I suspect that "most blogs" is an artifact of the circles you run in. Anti-cdotely none of the blogs I follow have disqus.
- gaius 15y agoWhen I look at the amount of effort that goes into this, I have to wonder if it wouldn't have been better to just write it in C in the first place.
- unicornporn 15y agoAside from all this, can anybody imagine what their Amazon bill is like? Lucky them they have not so demanding venture capitalists to pay their bills. I honestly wonder how long they will be able to keep growing like they do without making any money.
- wingo 15y ago> New Tumblr > Changed to a JVM centric approach for hiring and speed of development reasons. Interesting.
- douglasfshearer 15y agoIt seems that in New York it was easier to hire people who had worked at scale with the JVM, since that is what the nearby banking institutions might standardise on.
- thebluesky 15y agoAlways interesting to read about real-world usage of different tools and languages. From the article: ... • MySQL (plus sharding) scales, apps don't. • Redis is amazing. • Scala apps perform fantastically. ... Some of the organizations (Including Tumblr) using Scala were discussed as part of this talk: http://www.youtube.com/watch?v=qqQNqIy5LdM http://www.youtube.com/watch?v=qqQNqIy5LdM and slides here: http://mrkn.co/s/video_martin_odersky_what_s_next_for_scala,575/index.html http://mrkn.co/s/video_martin_odersky_what_s_next_for_scala,...
- andybak 15y agoI'm stunned by the traffic figures as I don't know one person that uses it (and I suspect the majority of my non-tech, Facebook loving friends have never heard of it). Is that anything to do with my demographic (41, M, UK based)?
- fooandbarify 15y agoYes (demographic). Although all sorts of people use tumblr, it seems to be most popular among (predominantly American) high-school age (and a bit older) teenagers. It has settled into a middle ground between Twitter and traditional blogs for a lot of these users -- single-paragraph posts or photo posts are the norm, and re-blogging is very common/encouraged.
- antinitro 15y agoI'm a bit younger (19, M, UK) and a lot of my facebook friends seem to use it. I think it appeals to a youngish demographic.
- hungryblank 15y agoYou do not write an article about technology by vomiting hundreds of bullet points mentioning every technology a company ever used. It's an appallingly bad example of technical journalism. This is not informational, it's just bullet point driven bike shedding.
- benjaminwootton 15y agoHe puts the articles together by researching from various sources, often without direct access to the company themselves. It's going to be a little fragmented, but nonetheless, I found this a hugely interesting article.
- toddh 15y agoI understand that hungryblank. But I don't consider myself a journalist. I'm a developer writing for developers so they can hopefully learn something that will help them build cool stuff. To that end I use a very concise easily digestible style with as little extraneous verbiage as possible. Nearly every point in the article could be exploded into a long form article and I already get in trouble for TL;DR :-)
- fauigerzigerk 15y agoI'm skeptical about the inbox model. I use Google Reader for two main reasons. One is that once I subscribe to a feed, I get access to all messages, not just new ones. The second is that I can search old messages as well. So if I understand Tumblrs inbox model correctly, that's exactly the kind of usage pattern that isn't supported.
- apgwoz 15y agoI don't see how this is a problem. Every cell has a copy of every post--that should make it possible to search it and get all older posts as well.
- fauigerzigerk 15y agoAbsolutely it should be possible. It just sounded to me as if they don't do it. But maybe I'm wrong. I'm not a Tumblr user.
- fooandbarify 15y agoThe UI is very similar to a Twitter dashboard (if you are familiar with that). I don't think it's possible to search for old posts in either case, and the style of the site doesn't really lend itself to that kind of use (for better or worse).