10 ms·
Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB
- pajju 15y agoThey have a very Interesting Stack as seen on Quora[1]: -Python + heavily-modified Django at the application layer -Tornado and (very selectively) node.js as web-servers. -Memcached and membase/redis for object and logical-caching -RabbitMQ as a message queue. -Nginx, HAproxy and Varnish for static-delivery and load-balancing. -Persistent data storage using MySQL. -MrJob on EMR for map-reduce. [1] http://www.quora.com/Pinterest/What-technologies-were-used-to-make-Pinterest http://www.quora.com/Pinterest/What-technologies-were-used-t...
- edwinnathaniel 15y agoPardon my ignorance: what is interesting about Pinterest stack? I think by now most startups, YC or not, in Silicon Valley will pretty much have a similar setup: 1) Choose the main web-stack (Rails or Django) 2) Choose the API framework (node.js, or something else) 3) Memcached for caching (or some NoSQL) 4) A message queue (ZeroMQ, RabbitMQ, or something ...) 5) nginx, HAProxy, Varnish, (or similar technology) 6) Hadoop for map-reduce (only if you need to go that path...)
- encoderer 15y agoNot trying to be a nit-pick, but wanted to mention that ZeroMQ is not a message queue in the sense that RabbitMQ (and Kestrel, Beanstalkd, SQS, etc) is. It's a socket library that you'd use to implement those technologies. And while this is truly unimportant, my experience with bay area startup stacks is that Redis has very thoroughly supplanted Memcached for most use cases. Though most still have a memcached cluster or two. That being said: 90% of software development in this world isn't done at tech companies and startups. So from that perspective, that stack is interesting.
- edwinnathaniel 14y agoCan you elaborate why Redis has supplanted Memcached? Is it because Redis occupies a space in between Memcached and MemcacheDB? (in the sense of Redis can be both in-memory and/or persisted on disk?) (Not that I care about starting the holy war of X vs Y, just out of curiosity why) You have a point regarding your last statement. Perhaps I've been here far too long to notice more of the 10% and less on the 90% :) ... which is true ... last I read someone is making almost $1M revenue developing Windows desktop app in 2012 and another company is making $60k/monthly revenue developing BB apps using in-apps Ads. I think I'm going to sign-off on HN and go to the other side... :D :D :D
- aneth 14y agoRedis is far more feature-ful, and can serve the purpose as memcache. Why have two parts when you only need one? Redis offers persistence, set operations, namespacing, you can delete multiple entries with wildcards...
- krobertson 15y agoI agree. I think the interesting aspect is that they built a service that serves an impressive number of users on fairly common technologies. There isn't anything very revolutionary about their stack by itself. FB didn't buy them for their technology... they could have replicated that on their own. If they want to be talking about infrastructure, the story of using common tools at high volume is more interesting that just what tools...
- raverbashing 15y agoOh, I wish it was that simple. You can't mix and match (you can theoretically) Let's say you began with Django, right? Or there was something already ready in Django. And then you begin to see the warts. But ok, you keep churning along. And the more you churn the more of a specialist in the deficiencies of each technology you become. Like the fact that Django's ORM runs like a dog. Or see all the "we moved to MongoDB and we regret it" discussions From that list, there are two technologies I would recommend strongly (if you know what you are doing): nginx and Redis The rest needs to be handled with care. And Redis is very powerful if you know how to use, but it's easy to go the "lazy way" with it, and it will be fast, but not as fast as it can.
- megaman821 14y agoYes, but the Django ORM helped them launch faster. There is nothing wrong using it to start with and then writing your own SQL queries when you need to scale or even changing your data model to accommodate scaling.
- raverbashing 14y agoI agree 100% (even though I don't know what or if they changed the Django ORM - most likely yes) Use the tools to launch faster, then optimize.
- drumdance 14y agoMy company is going through exactly this scenario right now. We launched on and use Rails for most stuff, but there are a couple of important cases where we have to do direct sql. Though I've had to work around some of Rails defaults for my use cases, there is no doubt that without it we'd not have been able to launch and iterate so quickly.
- cookiecaper 14y agoI hate the Django ORM and I don't know why Django doesn't just deprecate it and switch to SQLAlchemy as the new default, which is a much, much better ORM and used primarily by non-Django Python apps.
- wamatt 14y agoSeems like a sensible stack. However, is it not reasonable that would someone not consider node.js, as web stack as well? I will agree the focus is more towards backend and asynchronous API's traditionally, but the amount of front-end/UX orientated modules are growing every day.
- edwinnathaniel 14y agoSure why not. Some people don't even mind the lack of polished modules and instead spend time to write their own. Not a big fan of node.js (or JavaScript for that matter) and probably never will so I can't comment much.
- swah 14y agoI would think that Nginx + Django + SQL, and the rest is premature optization...
- garindra 14y agoEven simpler : Django + SQL in Heroku.
- rudiger 15y agoIt's cool that we're seeing companies reach 1 billion USD valuations with 12 employees.
- blantonl 15y agoIt will be even cooler when we see companies reach 1 billion USD net-income with 12 employees.
- edwinnathaniel 14y agoI heard Microsoft Visio team probably have less than 50 people working on it (granted the name Microsoft and the Office brand provide a huge selling point). Visio almost hit $1B according to insider... (can't verify).
- iusable 14y agoNo that's interesting! Gots to dig into this little tidbit.
- deleted 14y ago[deleted]
- shashashasha 15y agoTo be fair the employee number is as of last December, and they've grown quite a bit since then.
- mquinlan 15y agoBased off of their team page, Pinterest is up to 31 employees. http://pinterest.com/about/team/ http://pinterest.com/about/team/
- bbayer 15y agoThose kind of articles makes me feel so stupid. Even though those technologies are ready to go, making them work smoothly without any interruption always seems hard to me. I believe, there has to be lots of tips and trick. ( in other words you have to be experienced with all them or I am very lazy) I was wondering if any common receipt exists or if there is someone who can answer my couple of high scalability questions.
- raverbashing 15y agoYes, see "heavily-modified Django" (even though this is probably around scalability and connecting with other technologies) Scalability and uptime is still hard Still, Pinterest is laying on a great infrastructure (EC2, Elastic Map Reduce) and using it to the fullest.
- kmfrk 14y agoPinterest has performed quite awfully and has been very, very slow up until now. They may have fixed the problem, but it's not necessarily a model to follow. It's nice to know that even a site like Pinterest can't get it quite right all the time.
- teoruiz 15y agoAccording to the article Pinterest is spending on AWS EC2 >$30k to support 18M visitors/month. Data: $52/h (peak time, let's say 18 out of 24 hours) and $15/h (night time, let's say 6/24). Edit: as pointed in the comments $30k/month would only be the EC2 costs.
- carson 15y agoYou are just counting EC2 cost. The AWS cost for 410TB of S3 storage is around $39k. You would need to add in BW cost on top of that. It is also interesting that they seem to be using Akamai for a CDN instead of Cloudfront so not a completely AWS based solution. I wish they went into what they are storing in S3. 410TB is a lot of storage. My initial guess was cached images but 80M objects breaks down to 5MB per object and that is a lot more than what is needed for image caching.
- kitsune_ 14y agoFrom these figures they seem to burn over 100k per month on outsourced cloud services alone. Holy shit.
- tnash 14y agoYeah...that's insane. Since they're a "pre-revenue" company, they're just burning money.
- moe 14y ago"Burning" seems like an understatement. Looks like a switch to dedicated hardware would amortize within... 3 months.
- teyc 14y agoGiven their growth rates and cash in the bank, and the fact it is still looking for a business model, it is probably better for them to focus on their key problems before working on other issues. If they were self funded, like 37signals, and are running operations where they'd tighten the screws on cost, then the focus is different again.
- whalesalad 14y agoIt's nice to see that two companies valued at >= $1 billion are running a Python/Django stack. #teampony
- lionheart 14y agoYeah, I'm actually in the process of switching to Python/Django from Rails, so its great to hear how well it scales.
- whalesalad 14y agoI'm curious as to why you're switching. Care to share? I'm a long-time Django guy who's just been thrown into some Rails projects so I am going to be learning it shortly.
- irahul 14y agoFrom a Django, Rails, Flask, Sinatra guy; Rails is going to be fun if you leave your pre-conceived notions at the door. Rails is not Django, and Ruby is not Python, and any attempts to make it behave like how it's not supposed to behave will cause pain. However, if you can play per Rails rules(and you should), that would be a very pleasant experience. How are you learning Rails? Since you are an experienced programmer, the official rails guide will be the fastest way to get a feel of rails - http://guides.rubyonrails.org/getting_started.html http://guides.rubyonrails.org/getting_started.html The link above is the modified, famous "blog in 10 minutes". When I was starting, I found it very pleasant as it covers the 20% of the Rails which you use 80% of the time.
- whalesalad 14y agoThis is a great reply. Thanks for the tips. I think I might do a little messing around this evening.
- irahul 14y agoAt 18m visitors, I don't think your choice of Django or Rails matters. DB tends to be the limit for even the normal applications, let alone something with that kind of volume.
- bsg75 14y ago> Sharding is used, a database is split when it reaches 50% of capacity, allows easy growth and gives sufficient IO capacity Nice, compared to the usual, "The database was at 100% capacity, then we tried to shard/partition, and it did not go too well."
- ankimal 14y agoSmall, cohesive goal oriented teams with clear well defined goals can achieve so much.
- al_james 14y agoDoes anyone else feel that 410TB of user data seems quite a lot? If I have my maths right, even if all the 80 million objects are user data (as opposed to, say, logs) thats 5.3mb per object. Considering that most pinterest photos are from the web, that seems quite big.
- moe 14y agoYes, that number is ridiculous. For reference, that's about four times bigger than the iTunes music catalog (20mio MP3 files * 5MB average filesize = 100TB).
- skeletonjelly 14y ago5MB seems a bit small, they'd also have lossless copies right?
- moe 14y agoYes, their total storage consumption is bigger, but the MP3-part of the library is in that ballpark.
- tomwaddington 14y agoOriginal image plus two (or more) scaled thumbnails. Also, I don't think they're doing any reuse (upload the same pin for 2 different S3 objects of the same image). 5.3Mb / object is still really high though.
- al_james 14y agoWow, if they are not doing any reuse that really really dumb. I can't back this up, but would have thought at least 50% of all pins are repins.... if that 410TB figure is correct, this oversight is costing them $18k a month. It would be simple to have a separate asset id and pin id...
- neovive 14y agoInteresting. I'm migrating some old PHP apps over to Python and have been learning Flask + SQL Alchemy. Why would a site with so much traffic choose a full stack framework like Django that required so much modification?
- skeletonjelly 14y agoBecause when they started they didn't have the traffic, quicker to get to a running product. Probably easier now to cut back/change Django functionality than rewrite.