7 ms·

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years

261 points by aespinoza 14y ago

RyanZAG 14y ago
I'm fairly amazed by some of the choices - in particular the single huge user table that is hit on every profile page view to map url to shard. I'm guessing it has hot backups and similar - but wow - talk about a single point of failure. I guess having it like that actually makes it easier. If it fails, it should be trivial to get it back up quickly on a backup machine. The whole setup seems a lot simpler than nearly every 'web scale' solution I've heard discussed. EDIT: I'm not saying that is a bad thing - it's a good thing. Most of the articles around scaling on HN often deal with very complex Cassandra style technical solutions. This article is the opposite with a very 'just do it the simplest way' feel to it. One big mysql table for all users (probably with some memcache in front) is the sort of thing you can put together in an hour or two - yet it is the very core of their system. I (and I bet a lot of other engineers reading HN) try to focus too much on perfect implementations - if someone asked me to design the shard look-up for something as heavily hit as this, I'd probably reach for a distributed in-memory DB with broadcasted updates - and I'd probably be wrong to do so. The mysql/memcache approach is just going to be simpler to keep running.
- coolsunglasses 14y ago
  You don't really want to get into the alternatives. It's pretty common to shard everything but the core user data (email, PBKDF2, username) and scale that with slaves and vertical upgrades. SSDs can take you a long way if you're fastidious about offloading everything else off the machine responsible for keeping user accounts consistent. The alternatives aren't nice.
  - aespinoza 14y ago
    I agree. Usually it is better to scale vertically on the user data. Even if it seems painful. The alternatives like partitioning are not really that easy either. There is pain involved as well. I think, of the existing solutions, one is not better than the other. The choice in the end is which one fits best with your team and infrastructure, and specially which disadvantages are you willing to put up with.
- chiph 14y ago
  Since they're MySQL pros, they could put that on a MySQL Cluster and get high availability (up to 30 or so machines in the cluster, use a larger replica count, on top of drive mirroring, etc.) Also, it's a read-heavy workload, so hide it behind memcached.
- jokull 14y ago
  That’s not what they’re doing. They are sharding horizontally on the user. ID’s are 64 bits with the db location (logical shard id) in the first 16 bits. This will allow anyone with an ID to go to the right db for querying, granted they have logical to physical shard mappings.
  - js2 14y ago
    From the article: User table is unsharded. They just use a big database and have a uniqueness constraint on user name and email. Inserting a User will fail if it isn’t unique. Then they do a lot of writes in their sharded database. Get the user name from the URL. Go to the single huge database to find the user ID. I believe the parent is correct.
    - jokull 14y ago
      I see. We tackle the same issue by having a redis keyspace with emails. It can get pretty huge without any problems.
- jacques_chester 14y ago
  One table for users is a no-brainer. The outer limit on table size is ~7 billion rows. And that's just not a very big table these days.
- mcartyem 14y ago
  This might be a silly thing to ask, but why don't they save their data in flat files on a shared filesystem? They wouldn't need Memcached, since the OS caches files anyway; replication with rsync becomes easy; they don't need transactions anyway; and they wouldn't have so much software to manage.
  - hboon 14y ago
    Storing, tracking and managing billions of tiny files directly on a file system is a nightmare
    - mcartyem 14y ago
      What about it is a nightmare? Isn't storing, tracking, and managing billions of entries done directly in a database anyway?
      - boyter 14y ago
        Its a real pain when you want to inspect the files, delete or copy them. Try taking 300,000 files and copy them somewhere. Then copy 1 file which has the size of the 300,000 combined. The single file is MUCH faster (its also why we usually do a tar operation before copying stuff if its already compressed). Any database that's not a toy will usually lay the 300,000 records out in a single file (depending on settings, sizes and filesystem limits). The 300,000 files end up sitting all over the drive and disk seeks kill you at run-time. This may not be true for a SSD but I don't have any evidence to to suggest this or otherwise. Even if the physical storage is fine with this I suspect you may run into filesystem issues when you lay out millions if not hundreds of millions of files over a directory and then hit it hard. I have played with 1,000,000 files before when playing with crawling/indexing things and it becomes a real management pain. It may seem cleaner to lay each out as a singe file but in the long run if you hit a large size the benefits aren't worth it.
        hboon 14y ago
        In addition, It doesn't have good querying utilities. You'd have to build your own indexer and query engine. Since you can't put billions of files in a single directory, you'd have to split them into a directory tree. That alone requires some basic indexing functionality and rebalancing tools (in case a single directory grows too large or too small). This is without any more sophisticated querying capabilities like “where X=val” where X isn't the object ID/filename. Write performance is going to be very very horrible. Existing tools around managing, backing up and restoration, performance optimisation and monitoring aren't suitable for handling huge number of small files as well as a subset of them (give certain criteria related to the data itself) You could build specialized tools to resolve all of these issues, but in the end, you'd end up with some kind of database after hundreds of man-years anyway.
garydarna 14y ago
Great article, thanks for sharing.
nasalgoat 14y ago
I wonder if their dislike for Cassandra is based on previous versions pre-2.0. From my perspective of looking at it as it stands now, it's pretty compelling.
- zcam 14y ago
  You probably mean pre 1.2 as 2.0 isn't there yet.
  - cnlwsu 14y ago
    DataStax did call it 2.0 and now 3.0
    - pkolaczk 14y ago
      DataStax Enterprise 2.0 shipped with Cassandra 1.0 DataStax Enterprise 3.0 shipped with Cassandra 1.1 DataStax Enterprise is a different product than Cassandra. Cassandra is one of its components, but there are more things bundled, e.g. Solr and Hadoop.
- bloodredsun 14y ago
  Not if you want strong consistency. Cassandra's performance sucks in comparison with the likes of MongoDb or Couchbase when reading with strong consistency since the clients have no idea of the server topology.
  - cnlwsu 14y ago
    umm what? Cassandra is just as fast/faster (depending on both configurations and load) compared to MongoDB with consistant read/writes. Definitely with writes but reads get tricky.
    - bloodredsun 14y ago
      Firstly, these sorts of applications are always going to be more read heavy so the reads are more important. Secondly, Cassandra cannot and will not be as good as something like Couchbase since the client libraries are not aware of the server topology so they cannot make direct connection to the server hosting the data. This means that depending on your consistency requirements, Cassandra will be merely okay to occasionally terrible depending on whether you care about 99th percentile. This behaviour was one of the reasons my company moved away from Cassandra This is probably the best benchmark of Cassandra/MongoDb/Couchbase http://www.slideshare.net/renatko/couchbase-performance-benchmarking http://www.slideshare.net/renatko/couchbase-performance-benc...
anderspetersson 14y ago
great talk, inspired me to simplify the tech stack. Pyres looks like a good alternative for Celery in the Python world when you don't need Celery's cron-like features.
rikacomet 14y ago
A simply awesome article, that I would refer to back and again, can someone please refine this with his comments as well. I mean it would be really helpful to point out any mistakes Pinterest did, at any point, what I want to know is, what could they have done better. Also, business is a dynamic environment, so somethings might have changed, so can someone please also point out, if something was right when pinterest did it, but now there are better alternatives. Thanks a lot in advance. I would be really reading many of the comments properly. ~Rika
fotcorn 14y ago
Does someone now what programming language and framework they use? Still Python+Django? PyPy?
- goronbjorn 14y ago
  From what I've read, it's Python and "heavily modified" Django. I recall one of their engineers saying that, if they were to do it again, they would've gone with a more lightweight Python framework because of how many heavy modifications they made.
  - misiti3780 14y ago
    I have read that also (multiple times actually) and the statement never made sense to me - of course in hindsight you would start with something like Flask because you ran into all of the scaling problems with Django, but there would be no way to fix these problems before they occurred - and if you did try to fix them before they would occurred, it would be premature optimization. It seems like a example of what Taleb's calls the "Narrative Fallacy" http://cij.inspiriting.com/?p=364# http://cij.inspiriting.com/?p=364#
dlhavema 14y ago
Very cool, I thought pinterest came out of nowhere and is now everywhere, but i didn't realize how fast they did it and how large they've grown. thanks for posting!
bloodredsun 14y ago
What is most interesting to me is that in many ways they have reinvented Couchbase. I think that the only reason they didn't go with this technology was the financial cost for their scale was too high.
cagenut 14y ago
I really like how when you look at it, most of their scaling was removing things from the environment and really honing the platform down to what it needed to be.
dmourati 14y ago
More specifics about the decision to dump MongoDB would be extremely useful.
sontek 14y ago
How does SOA improve scaling? I've never had to build anything as big as pinterest and it seems interesting to me. I don't quite understand how it reduces connections. If you have your front-end tier connecting to your service tier, you'll still have the same amount of requests going to the database, just with a little middle-man... If I understand correctly?
- aaronbrethorst 14y ago
  I'm no expert, but hopefully by abstracting out different services from your core app and making them available through some connection protocol (Thrift, a message queue, http, whatever) you can more easily scale up and scale out systems that become hotspots, and consumers of that service will never be any the wiser, since they only interact with it through the published API. Looking at it another way, when I talk to the GitHub API, I have no idea what sort of database I'm ultimately talking to. Their interface abstracts away all of that complexity. Each request they receive from me might be handled by a different machine for all I know. It makes no difference.
- bloodredsun 14y ago
  By itself SOA doesn't do anything but it does free you up to do other things that allow you to scale like read-write separation and event driven or fan-out caching. One of the most important thing SOA does is that it quickly stops you from doing dirty hacks that work right now but are killers for future work. It's encapsulation at the application level
- arocks 14y ago
  Service orientation allows specialization of architectural components. If you want to build on scale, it is easier to replicate specialized components than a generalized one. It also benefits from better fault tolerance. Additionally, services layer forms an interface between the client and service provider, which makes it easier to scale.
jacques_chester 14y ago
> Architecture is doing the right thing when growth can be handled by adding more of the same stuff. You want to be able to scale by throwing money at a problem which means throwing more boxes at a problem as you need them. If you are architecture can do that, then you’re golden My understanding is that the term of art is horizontal scalability.
jnankin 14y ago
2 questions: 1) What do they do with the Pinterest-generated ID? Do they store it with each row in addition to the local ID? 2) Why randomly select a shard? Shouldn't you do this based on DB size across all boxes, or at least based on physical space left across all boxes?
- jnankin 14y ago
  Oops, 1 is directly mentioned in the talk.
- yashh 14y ago
  We decompose 64 bit ids to shard_id, type and local_id. We connect to the shard go to the table and read the local_id. We randomly select a shard for a new user only. Ofcourse we capacity plan this and open new shards and even out users eventually.
jnankin 14y ago
what happens if 1 shard contains a whole slew of large companies (macy's, gap, etc), or over time users on one shard grow to the point that they cannot be contained within one physical machine? Are they just assuming that this will be very unlikely?
- yashh 14y ago
  No not at all. We already have this problem. Right now we are able to take the DB hit but we will eventually attack this problem from cache perspective. We will have replicated caches for this scenario. In worst case if a shard is overloaded we stop creating new users in it and put it on a dedicated physical machine. I dont think we will let the situation get there.

261 points by aespinoza 14y ago

I'm fairly amazed by some of the choices - in particular the single huge user table that is hit on every profile page view to map url to shard. I'm guessing it has hot backups and similar - but wow - talk about a single point of failure. I guess having it like that actually makes it easier. If it fails, it should be trivial to get it back up quickly on a backup machine. The whole setup seems a lot simpler than nearly every 'web scale' solution I've heard discussed. EDIT: I'm not saying that is a bad thing - it's a good thing. Most of the articles around scaling on HN often deal with very complex Cassandra style technical solutions. This article is the opposite with a very 'just do it the simplest way' feel to it. One big mysql table for all users (probably with some memcache in front) is the sort of thing you can put together in an hour or two - yet it is the very core of their system. I (and I bet a lot of other engineers reading HN) try to focus too much on perfect implementations - if someone asked me to design the shard look-up for something as heavily hit as this, I'd probably reach for a distributed in-memory DB with broadcasted updates - and I'd probably be wrong to do so. The mysql/memcache approach is just going to be simpler to keep running.

coolsunglasses 14y ago
You don't really want to get into the alternatives. It's pretty common to shard everything but the core user data (email, PBKDF2, username) and scale that with slaves and vertical upgrades. SSDs can take you a long way if you're fastidious about offloading everything else off the machine responsible for keeping user accounts consistent. The alternatives aren't nice.
- aespinoza 14y ago
  I agree. Usually it is better to scale vertically on the user data. Even if it seems painful. The alternatives like partitioning are not really that easy either. There is pain involved as well. I think, of the existing solutions, one is not better than the other. The choice in the end is which one fits best with your team and infrastructure, and specially which disadvantages are you willing to put up with.
chiph 14y ago
Since they're MySQL pros, they could put that on a MySQL Cluster and get high availability (up to 30 or so machines in the cluster, use a larger replica count, on top of drive mirroring, etc.) Also, it's a read-heavy workload, so hide it behind memcached.
jokull 14y ago
That’s not what they’re doing. They are sharding horizontally on the user. ID’s are 64 bits with the db location (logical shard id) in the first 16 bits. This will allow anyone with an ID to go to the right db for querying, granted they have logical to physical shard mappings.
- js2 14y ago
  From the article: User table is unsharded. They just use a big database and have a uniqueness constraint on user name and email. Inserting a User will fail if it isn’t unique. Then they do a lot of writes in their sharded database. Get the user name from the URL. Go to the single huge database to find the user ID. I believe the parent is correct.
  - jokull 14y ago
    I see. We tackle the same issue by having a redis keyspace with emails. It can get pretty huge without any problems.
jacques_chester 14y ago
One table for users is a no-brainer. The outer limit on table size is ~7 billion rows. And that's just not a very big table these days.
mcartyem 14y ago
This might be a silly thing to ask, but why don't they save their data in flat files on a shared filesystem? They wouldn't need Memcached, since the OS caches files anyway; replication with rsync becomes easy; they don't need transactions anyway; and they wouldn't have so much software to manage.
- hboon 14y ago
  Storing, tracking and managing billions of tiny files directly on a file system is a nightmare
  - mcartyem 14y ago
    What about it is a nightmare? Isn't storing, tracking, and managing billions of entries done directly in a database anyway?
    - boyter 14y ago
      Its a real pain when you want to inspect the files, delete or copy them. Try taking 300,000 files and copy them somewhere. Then copy 1 file which has the size of the 300,000 combined. The single file is MUCH faster (its also why we usually do a tar operation before copying stuff if its already compressed). Any database that's not a toy will usually lay the 300,000 records out in a single file (depending on settings, sizes and filesystem limits). The 300,000 files end up sitting all over the drive and disk seeks kill you at run-time. This may not be true for a SSD but I don't have any evidence to to suggest this or otherwise. Even if the physical storage is fine with this I suspect you may run into filesystem issues when you lay out millions if not hundreds of millions of files over a directory and then hit it hard. I have played with 1,000,000 files before when playing with crawling/indexing things and it becomes a real management pain. It may seem cleaner to lay each out as a singe file but in the long run if you hit a large size the benefits aren't worth it.
      - hboon 14y ago
        In addition, It doesn't have good querying utilities. You'd have to build your own indexer and query engine. Since you can't put billions of files in a single directory, you'd have to split them into a directory tree. That alone requires some basic indexing functionality and rebalancing tools (in case a single directory grows too large or too small). This is without any more sophisticated querying capabilities like “where X=val” where X isn't the object ID/filename. Write performance is going to be very very horrible. Existing tools around managing, backing up and restoration, performance optimisation and monitoring aren't suitable for handling huge number of small files as well as a subset of them (give certain criteria related to the data itself) You could build specialized tools to resolve all of these issues, but in the end, you'd end up with some kind of database after hundreds of man-years anyway.

garydarna 14y ago

Great article, thanks for sharing.

nasalgoat 14y ago

I wonder if their dislike for Cassandra is based on previous versions pre-2.0. From my perspective of looking at it as it stands now, it's pretty compelling.

zcam 14y ago
You probably mean pre 1.2 as 2.0 isn't there yet.
- cnlwsu 14y ago
  DataStax did call it 2.0 and now 3.0
  - pkolaczk 14y ago
    DataStax Enterprise 2.0 shipped with Cassandra 1.0 DataStax Enterprise 3.0 shipped with Cassandra 1.1 DataStax Enterprise is a different product than Cassandra. Cassandra is one of its components, but there are more things bundled, e.g. Solr and Hadoop.
bloodredsun 14y ago
Not if you want strong consistency. Cassandra's performance sucks in comparison with the likes of MongoDb or Couchbase when reading with strong consistency since the clients have no idea of the server topology.
- cnlwsu 14y ago
  umm what? Cassandra is just as fast/faster (depending on both configurations and load) compared to MongoDB with consistant read/writes. Definitely with writes but reads get tricky.
  - bloodredsun 14y ago
    Firstly, these sorts of applications are always going to be more read heavy so the reads are more important. Secondly, Cassandra cannot and will not be as good as something like Couchbase since the client libraries are not aware of the server topology so they cannot make direct connection to the server hosting the data. This means that depending on your consistency requirements, Cassandra will be merely okay to occasionally terrible depending on whether you care about 99th percentile. This behaviour was one of the reasons my company moved away from Cassandra This is probably the best benchmark of Cassandra/MongoDb/Couchbase http://www.slideshare.net/renatko/couchbase-performance-benchmarking http://www.slideshare.net/renatko/couchbase-performance-benc...

anderspetersson 14y ago

great talk, inspired me to simplify the tech stack. Pyres looks like a good alternative for Celery in the Python world when you don't need Celery's cron-like features.

rikacomet 14y ago

A simply awesome article, that I would refer to back and again, can someone please refine this with his comments as well. I mean it would be really helpful to point out any mistakes Pinterest did, at any point, what I want to know is, what could they have done better. Also, business is a dynamic environment, so somethings might have changed, so can someone please also point out, if something was right when pinterest did it, but now there are better alternatives. Thanks a lot in advance. I would be really reading many of the comments properly. ~Rika

fotcorn 14y ago

Does someone now what programming language and framework they use? Still Python+Django? PyPy?

goronbjorn 14y ago
From what I've read, it's Python and "heavily modified" Django. I recall one of their engineers saying that, if they were to do it again, they would've gone with a more lightweight Python framework because of how many heavy modifications they made.
- misiti3780 14y ago
  I have read that also (multiple times actually) and the statement never made sense to me - of course in hindsight you would start with something like Flask because you ran into all of the scaling problems with Django, but there would be no way to fix these problems before they occurred - and if you did try to fix them before they would occurred, it would be premature optimization. It seems like a example of what Taleb's calls the "Narrative Fallacy" http://cij.inspiriting.com/?p=364# http://cij.inspiriting.com/?p=364#

dlhavema 14y ago

Very cool, I thought pinterest came out of nowhere and is now everywhere, but i didn't realize how fast they did it and how large they've grown. thanks for posting!

bloodredsun 14y ago

What is most interesting to me is that in many ways they have reinvented Couchbase. I think that the only reason they didn't go with this technology was the financial cost for their scale was too high.

cagenut 14y ago

I really like how when you look at it, most of their scaling was removing things from the environment and really honing the platform down to what it needed to be.

dmourati 14y ago

More specifics about the decision to dump MongoDB would be extremely useful.

sontek 14y ago

How does SOA improve scaling? I've never had to build anything as big as pinterest and it seems interesting to me. I don't quite understand how it reduces connections. If you have your front-end tier connecting to your service tier, you'll still have the same amount of requests going to the database, just with a little middle-man... If I understand correctly?

aaronbrethorst 14y ago
I'm no expert, but hopefully by abstracting out different services from your core app and making them available through some connection protocol (Thrift, a message queue, http, whatever) you can more easily scale up and scale out systems that become hotspots, and consumers of that service will never be any the wiser, since they only interact with it through the published API. Looking at it another way, when I talk to the GitHub API, I have no idea what sort of database I'm ultimately talking to. Their interface abstracts away all of that complexity. Each request they receive from me might be handled by a different machine for all I know. It makes no difference.
bloodredsun 14y ago
By itself SOA doesn't do anything but it does free you up to do other things that allow you to scale like read-write separation and event driven or fan-out caching. One of the most important thing SOA does is that it quickly stops you from doing dirty hacks that work right now but are killers for future work. It's encapsulation at the application level
arocks 14y ago
Service orientation allows specialization of architectural components. If you want to build on scale, it is easier to replicate specialized components than a generalized one. It also benefits from better fault tolerance. Additionally, services layer forms an interface between the client and service provider, which makes it easier to scale.

jacques_chester 14y ago

> Architecture is doing the right thing when growth can be handled by adding more of the same stuff. You want to be able to scale by throwing money at a problem which means throwing more boxes at a problem as you need them. If you are architecture can do that, then you’re golden My understanding is that the term of art is horizontal scalability.

jnankin 14y ago

2 questions: 1) What do they do with the Pinterest-generated ID? Do they store it with each row in addition to the local ID? 2) Why randomly select a shard? Shouldn't you do this based on DB size across all boxes, or at least based on physical space left across all boxes?

jnankin 14y ago
Oops, 1 is directly mentioned in the talk.
yashh 14y ago
We decompose 64 bit ids to shard_id, type and local_id. We connect to the shard go to the table and read the local_id. We randomly select a shard for a new user only. Ofcourse we capacity plan this and open new shards and even out users eventually.

jnankin 14y ago

what happens if 1 shard contains a whole slew of large companies (macy's, gap, etc), or over time users on one shard grow to the point that they cannot be contained within one physical machine? Are they just assuming that this will be very unlikely?

yashh 14y ago
No not at all. We already have this problem. Right now we are able to take the DB hit but we will eventually attack this problem from cache perspective. We will have replicated caches for this scenario. In worst case if a shard is overloaded we stop creating new users in it and put it on a dedicated physical machine. I dont think we will let the situation get there.