10 ms·
How Pinterest scaled
- deleted 3y ago[deleted]
- snihalani 3y agoEverytime I read one of these, no one tells me what resource was the bottleneck in vertical scaling. Does it feel like silicon valley is addicted to horizontal scaling?
- John23832 3y agoCost (due to higher effective utilization) and single points of failure? Vertical scaling is more expensive without the benefit of failover. With modern clouds, Horizontal scaling is cheaper with commodity hardware and gives the added benefit of failover redundancy.
- wongarsu 3y agoIf you are already in one of the big 3 cloud providers, vertical scaling is more expensive, especially once you add failover. If you rent bare metal it's often the other way around. So maybe one of the core reasons for Silicon Valley's love for horizontal scaling is due to plentiful AWS credits.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- John23832 3y agoWhether people use modern clouds is orthogonal to whether they choose to use vertical scaling. I can both horizontally and vertically scale in AWS. I can both horizontally and vertically scale in my home office. Horizontally scaling is just more cost efficient if you can dynamically adjust to load. Clouds enable that because they have massive reserve compute. In the startup world, there infinite compute, but finite money. It’s cost savings.
- CaptainOfCoit 3y ago> In the startup world, there infinite compute, but finite money. It’s cost savings. If you're using cloud stuff like EC2 on AWS for cost savings, you're in for a bad time. It makes sense for stuff that you need to be able to dynamically scale on a whim, but once you know what kind of setup you need, it's almost always cheaper to go dedicated.
- wongarsu 3y agoBare metal, doubling the amount of RAM and using a more powerful CPU is cheaper than adding a whole server that needs space, network, its own Motherboard, case, power supplies etc. This makes vertical scaling much more cost effective (up to a point, which is currently at about 2TB RAM). At AWS's data center AWS also buys big servers. But their advantage is that they only have big servers, and then rent it out in pieces. They don't care whether they rent you 2 48 core 96GB instances or one 96 core 192GB instances. Both options run as VMs on much larger servers and take the same resources, so both are priced the same (a machine of the same category with twice the resources costs exactly twice as much). Thus there's a much smaller benefit to vertical scaling. If you compare actual prices on large-ish instance types between AWS and regular dedicated servers (rented in a DC, not doing your own thing) then AWS is very expensive. And the hopes of paying less due to better utilization somehow never pan out. AWS has lots of advantages, but price isn't one of them; and their price structure will influence the architecture you choose.
- klooney 3y agoAWS charges the same money per core and gig of ram, whether you get a big instance or 5 small instances. I guess their top end isn't as top end as hardware you can get in your DC, though.
- roncesvalles 3y agoThe 21st century love for horizontal scaling is for not wanting to be locked in with IBM, because in the limit vertical scaling means buying an IBM mainframe.
- dub 3y agoTypically the price of not having horizontal scaling is felt more by the engineers than the users, at first: - Data migrations, schema changes, backfills, backups & restores, etc., take so long that they can either cause or risk outages or just waste a ton of engineer time waiting around for operations to complete. If you have serious service level objectives regarding time to restore from backup, that alone could be a forcing function for horizontal sharding (doing a point-in-time backup of a 40TB database while dropping some unwanted bad DELETE transaction or something like that from the transaction log is going to be very slow and cause a long outage). - The lack of fault isolation means that any rogue user or process making expensive queries impacts performance and availability for all users, vs being able to limit unavailability to a single shard - When people don't have horizontal scalability, I've seen them normalize things like not using transactions and not using consistent reads even when both would substantially improve developer and end-user experience, with the explanation being a need to protect the primary/write database. It's kind of like being in an abusive relationship: you internalize the fear of overloading the primary/write server and start to think it's normal not to be able to consistently read back the data you just wrote at scale or not to be able to use transactions that span multiple tables or seconds as appropriate.
- dalyons 3y agoTo your first point I find in these discussions the “just buy a bigger server” crowd massively underestimate the operational problems with giant db servers. They are no fun to babysit, and change gets really hard and tedious to not accidentally bring the whole thing down. It becomes a massive drain on the velocity and agility of the business.
- baobun 3y agoGiant sharded DB clusters aren't that much more fun or less precarious... Ever run Cassandra or Clickhouse at scale? IME vertically scaled replicas/hot-stand-bys are a lot more stable to operate if your requirements allow you to get away with it. OtoH you better already be prepared if/when you hit scaling limits.
- baz00 3y agoIt's always the accountants. Those big ass SQL boxes cost serious money.
- angarg12 3y agoFor us blast radius is a reason to intentionally limit the size of our servers. We could have smaller fleets with bigger servers, but then if anything went wrong it would impact a larger number of customers. That being said I'm partial of having fleets as small as possible to simplify operations. While trying different instance sizes, I found our bottleneck ended up being network. The largest instance types saturate the network first, and hence we can't use servers as efficiently.
- BiteCode_dev 3y ago11m monthly user is not that of a big number. Any stack will do providing you don't do something stupid. Our stack was similar than pinterest, django but postgres and redis. Team of 2. Indeed, keep it simple. Cache things. Use queue, etc. You can scale vertically a lot. Db perfs and disk space were our points of focus.
- kugelblitz 3y agoI think 11m monthly users is a lot. Sure, compared to Facebook 2023 it's minuscule. But assuming those 11m monthly users log in 2 times a week (~8 per month) and check out 15 pages each time that comes out to 1.3bn pageviews a month. Divided by (60s/m * 60m/h * 24h/d * 30d/m) comes out to about sustained 500 pageviews per second. I assume since they're a viral / addictive platform the numbers will be higher. Plus they're not a blog or so, but rather a personalized social media platform, which makes stuff more complicated (you can't just cache their news feeds, they're all personalized). It's easy to say meh, 11m is nothing, there are other platforms with more users (esp. since you don't bring up examples how you managed otherwise). But I think it's a big technical feat to do this with 6 people.
- blackoil 3y ago500 pageviews per second are big but 90 Web Engine + 50 API engine? i.e. ~6 rps per web server and 10*x rps/ API server. Apart from that ~250 servers for DB+cache. I know my comment smells so HN and kudos for the team that they build and scale it. But as an outsider hmmm.
- snovv_crash 3y agoA single SSD can serve over 100k ops/s. Pubsub systems built for the finance industry operate in excess of 60M messages/second [0]. I understand having multiple machines for failover reasons, but I can't help but feel that the majority of scale-out is due to the people who are doing the development not having the skills to properly optimise their systems. 0. https://youtu.be/8uAW5FQtcvE https://youtu.be/8uAW5FQtcvE
- 3y ago
- vishnugupta 3y agoAm I the only one to think this is bit too much for 11m users? 88DB insurances, ~300 across cache and nosql DB, ~200 web servers?
- pjmlp 3y agoNo, I feel the same. Serving ecommerce sites with a couple of Java or .NET servers, backed by Oracle or SQL Server instances, surviving heavy load days like black Friday. See Stackoverflow architecture, for example.
- FridgeSeal 3y agoStackOverflow gets away with their arch because they have a much bigger read load than write load, and a design that can “hide” slow writes. I would hazard a guess the Pinterest usage model has a somewhat higher write-load, and is more sensitive to slow writes/non-propagating writes. Do I think their arch is still overkill? Yeah, do I think equating them with SO and friends is reasonable? No.
- pjmlp 3y agoStill, the amount of servers is most likely a side effect of using Python as well. The scaling issues we had with Tcl in the dot-com wave with our AOLServer clone, is one of the reasons why I wouldn't use something like Python.
- FridgeSeal 3y agoDefinitely. There was a post on HN a couple of months back about Pinterest saving 2m/month on infra costs by swapping from Python to Elixir/Erlang. It copped flack in the comments because basically “ppphhh $2m that’s not even worth saving, how dumb, Python is good enough”.
- deleted 3y ago[deleted]
- theshrike79 3y agoI think people commenting how overblown the architecture for Pinterest is haven't seen a true Pinterest Person in the real world. There are people whose lives rotate around having different boards on Pinterest. They collect ideas for recipes, interior design, clothing, vacations and _everything_ on Pinterest. It's not a site where people click three links and come back in a week for their next three clicks.
- iforgotpassword 3y agoI always thought the sole purpose of Pinterest was to show images in Google image search that you can't actually see when you click them.
- kramerger 3y agoYeah, that's how I see it too. This is so annoying people have created browser extensions to remove Pinterest links from search results
- timrichard 3y agoHmm, sounds like the relationship I have with Trello.
- threeseed 3y agoWe really need to ban sites like this. It is just repackaged content from the original Pinterest presentation back in 2012: https://www.infoq.com/presentations/Pinterest/ https://www.infoq.com/presentations/Pinterest/
- lionkor 3y agoI wasnt here in 2012 so this was new info for me - I liked the article
- engineercodex 3y agoI’m the writer of this. I linked to this presentation very clearly throughout the article as a source. This was an article meant to resurface learnings from something that happened a decade ago, with added images and a clear distillation so that somebody doesn’t have to watch the 45 minute presentation to understand what happened. I’m sorry you didn’t like it. But I hope you can see the value I’m trying to provide here, as someone myself who doesn’t really have the time to sit through hour-long presentations to learn something.
- rmbyrro 3y agoI think most people here see the value of the article. Thank you for publishing it. Parent comment was also valuable to me, for seeing how important it is to listen to others before jumping to conclusions about the worthiness of something.
- wokwokwok 3y ago[flagged]
- clippyplz 3y agoHow do you figure this 45 minute talk with slides is "literally the same" as a five minute article with pictures?
- engineercodex 3y agoI don’t really understand this comment. I didn’t “slap” any date on top of the article. 2 Oct 2023 is the date I published my article. (This is how blogs on the Internet work.) The first line of the article says that this all happened in 2012. I’m not sure I could get any clearer there. The “GPT distillation” phrase is pretty rude. Anyone reading the article can see the difference between it and the presentation. Every “hand-drawn” image in the article is created by me in Excalidraw, and any images from the presentation are sourced with a very clear link to the presentation. If you have any constructive feedback, I am happy to listen and take it into account. Otherwise, I’m not really a fan of the rudeness. Thanks.
- phartenfeller 3y ago"The team removed table joins and complex queries from the database layer." As a database person, I dislike reading this. I understand their need and priority to keep things running. But I would have loved it if they emphasized that doing something like this has so many disadvantages. Normalization, referential integrity and a powerful query language bring so many benefits. I see young people oftentimes seeing relational vs. NoSQL as two totally valid opinions, like Coke vs. Pepsi. And not as one thing runs the world for decades and is perfect for >90% of use cases, and the other one is for niche cases and fast hyper-scalers.
- t8sr 3y agoI don’t like it either, but apparently semi-structured data is the future. It’s proving easier to build systems that tolerate bad data and broken references than it is to get our data normalized and cleaned. I’m the guy on most teams pontificating about SQL and data-oriented code, but even I’ll concede that large scale live systems probably shouldn’t serve from an SQL database.
- koliber 3y agoDo they have only one huge table in the db that contains all of their entities? From the key structure that seems reasonable: shard id, type, object id. If that is the case, is a relational db still the right place to keep such data? It seems that if the goal is to store this kind of data in one or a few huge sharded tables, a different tool could be optimizing to do the job better. It seems that I’m missing something…
- rwultsch 3y agoA pin was a 1.2 KB json blob. There were other tables but pins was the big one. Why MySQL? It did not destroy data like the alternatives. how storage became efficient https://medium.com/pinterest-engineering/evolving-mysql-compression-part-1-7f8b09666589 https://medium.com/pinterest-engineering/evolving-mysql-comp... https://medium.com/pinterest-engineering/evolving-mysql-compression-part-2-2c3eb0101205 https://medium.com/pinterest-engineering/evolving-mysql-comp...
- UberFly 3y agoKagi Search's #1 user blocked site is... pinterest.com If they were once a good idea, those days have long passed.
- suddenclarity 3y agoDoesn't it just show that non-members dislike walls? In this case the requirement of an account. I think it would be a bad idea to rate Netflix and Discord based on how non-users experience them.
- zeusk 3y agoExcept for, netflix and discord aren't SEO spam and don't pollute search results.
- dfgfek 3y agoThey don't? If I google the name of a TV series to get more information about it, several streaming platforms that have it, including Netflix, take the first spots of the results, although I have no interest in that spam because I intend to pirate it.
- DrSiemer 3y agoNetflix and Discord don't tease people with content that appears to be available until you click anywhere near them and get all the black patterns vomited out over you. Google Image search is almost useless if you don't block out crap sites like this.
- t8sr 3y agoThis is over 10 years out of date. Pinterest today has literally 40x the number of users this article is talking about, and I’d bet the architecture is very different. It might still hold lessons for scaling today, but things sure have changed since 2012.
- engineercodex 3y agoCorrect. This is a retrospective of an artifact from the early 2010s. The architecture is much different now.
- exikyut 3y agoDo you have any straightforward leads or pointers on what the tech stack looks like today? It would probably be very interesting to compare then and now.
- poisonborz 3y agoHonestly the last 10 years of software made scaling in terms of serving requests easy, such feats are common. Real hurdles come with spammers, moderation, support.
- ochoseis 3y agoDid they switch from Django to Flask, and if so, why?
- stefanos82 3y agoWithout knowing their reasoning behind their decision, I would assume they did so, so they could have better microservices control than Django's monolithic approach; I could be wrong though, but that's the first thing that came in mind.
- ksec 3y ago>In January 2012, Pinterest hit 11.7 million monthly unique users with only 6 engineers. The thing to strive for 12 years later should be how to scale to 100M+ users with 6 engineers. NAND and CPU core has an order of magnitude reduction in price / performance. 1-2TB DRAM is obtainable. MySQL or Postgres has a decade of improvements made, along with simpler, battle tested solution for scaling. Both Python and Ruby, Rails or Django has all the lessons learned from real world. At least that is what should have happened.
- lifeisstillgood 3y agoThe interesting takeaway for me when I see "one engineer, two engineers" is not only "I could have built that stack" but "I did actually build that stack back then" Like millions of other devs, I have made a (decent) living out of this career, but you do wonder what might have been. I think there is a new stack - a new Dev Manual - of what to do and build to keep it simple today (Hadoop is involved :-)
- angarg12 3y agoThese days hardware and tech have come such a long way that building a service that supports millions of customers isn't hard at all. A colleague and I spent the last 3 years at work building a system that supports tens of millions of QPS, and we did 90% of the work ourselves. Of course this doesn't detract merit from the post. The devil is on the details, and there are many complexities other than raw scale.
- duckmysick 3y agoI might be misunderstanding something, but it seems like a hard problem if you spent 3 years solving it. Or are those 3 years mostly maintenance and the bulk of work was done in a few weeks at the start?
- liviux 3y agoInternet would be a better place without websites like Pinterest and Quora. And articles like this, about a presentation older than 10 years.
- impulser_ 3y agoMy theory is the majority of the bad tech decisions made at these companies is mostly due to the high turnover rate of employees in these company. The average tenure of a software engineer/developer is just 2 years. People come and go and the next thing you know you're using 6 different database with services written in 5 different languages. Instead if you kept the same 6 engineers around for decades you could probably scale to 100m with those same engineers. It one of the reasons why it seems like Google can never release a product. They can't keep people around long enough to see it through.
- psadri 3y agoWhatsApp scaled this way. -500m daily users with only -30 total engineers across 5 devices + backend.
- di4na 3y agoGoogle has longer tenure. The thing you seem to miss are the other common denominator. Huge amount of money and unreasonably far into the future expectations of returns. Means there is no short to medium term pressure to optimise for efficiency or returns, which means one of the fundamental element of good engineering environment is missing. These companies build in a vacuum of limitations in term of cost and a vacuum in term of goals.
- John23832 3y agoThe average tenure at Google has been widely reported for while now as roughly one year. https://stackoverflow.blog/2022/04/19/whats-the-average-tenure-of-an-engineer-at-a-big-tech-company-ep-434/ https://stackoverflow.blog/2022/04/19/whats-the-average-tenu...
- nielsole 3y agos/average/median/g according to the linked source(cnbc)
- 3y ago
- manusachi 3y agoThey also used to be "pretty jazzed about Elixir" and had it in their stack [1], at least while Steve Cohen was there (btw, Steve is also the author of lexical[2] - one of the newer Elixir LSPs out there) [1] https://web.archive.org/web/20160203120655/https://engineering.pinterest.com/blog/introducing-new-open-source-tools-elixir-community https://web.archive.org/web/20160203120655/https://engineeri... [2] https://github.com/lexical-lsp/lexical https://github.com/lexical-lsp/lexical
- pawelduda 3y agoYeh, the question is: are they still?
- mentos 3y agoQuestion I have for HN is if you had a weekend to create a Pinterest competitor for a hackathon that needed to scale to the same number of users, what would you use?
- tim333 3y ago10k users or 11 million?
- christophilus 3y agoWhatever I knew best. Node, Go, .NET, and Postgres in my case. Those can all handle a lot of load if properly engineered and adequately provisioned. Redis or some other caching layer might come into play if the scale really warranted it, but it certainly wouldn’t for a hackathon project.
- maccard 3y agoCompletely agree - go + myself + redis would be my go-to here. I'm of the camp that the minute you think you'll need redis, use redis. It's much easier to manage and deal with when you can turn it off than when you can't.
- pacifika 3y agoYou can’t create a Pinterest competitor in a weekend. Or the answer is enough money to purchase Pinterest.
- maccard 3y agoWhatever you know best. The tech stack is irrelevant. Assuming 11m MAU, we can finger wave to 200k peak active users. That's handleable on small hardware even if you use express. The hard part is finding 11 million users.
- qingcharles 3y agoI think everyone will answer this with their current stack-of-choice. If you have a hammer, you see every problem as a nail. I'd use .NET Razor pages, Dapper, Postgres, Kestrel on Linux? What's amazing is that 14 years later and the major browsers are still hoarding the Pinterest masonry layout behind a flag! https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_grid_layout/Masonry_layout https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_grid_la...
- smcleod 3y agoIsn’t Pinterest just a spam / advertising website? Most of the folks I’ve asked have it blocked in unlock/kagi/pihole etc…
- mfru 3y agoit is a nice tool for visual thinkers, also good for creating mood boards.
- tim333 3y agoLots of people like it. I've noticed female friends like it for interior design / fashion ideas / similar. It's now floated with a $25bn market cap, and 450 million users. According to Wikipedia they had 3,987 employees in 2022 so I guess they've bloated up a bit since 2012.
- graphe 3y agoIt is. It aggregstes other content and doesn't add anything of its own besides ads whichbice never seen since I AdBlock, worst thumbnails, and a walled garden for blogs.
- beacon294 3y agoCuration and curation tooling both add value, similar to how purifying chemicals or making food adds value. And that isn't the only dimension either
- graphe 3y agoThe only reason I know about Pinterest is because it ruined Google images and it destroyed curation on a much better platform.
- bratwurst3000 3y agoIf i want to visualize an idea I use it. Mostly interior design and such
- qingcharles 3y ago
- seper8 3y agoWhats a web engine?
- kramerger 3y agoAre we talking about actual , "willing", users or people who accidentally clicked on a Pinterest link in their search results then immediately clicked back? Because that's exactly how I use Pinterest.
- lolinder 3y agoMay I recommend Kagi? I had the exact same problem until I blocked Pinterest, and a lot of others had the same idea: https://kagi.com/stats?stat=leaderboard&k=-2 https://kagi.com/stats?stat=leaderboard&k=-2
- ipaddr 3y agoWish we could block the kagi advertising on here. If you visit 100 top tech sites it's great if you go outside of that bubble you get pretty thin results.
- deleted 3y ago[deleted]
- qingcharles 3y agoI've been a Pinterest user since launch. I couldn't imagine life without it. I love finding creative photos and videos and saving them all for inspiration.
- wlonkly 3y agoI understand the complaint, but I don't think that was a problem back in 2012, the timeframe of the article.
- 1023bytes 3y agoInteresting that they use both Redis and Memcache at the same time. Is there any particular reason for that?
- alberth 3y ago66 MySQL DBs + 66 secondaries 59 Redis Instances 51 Memcache Instances That’s interesting. For every 1 database, they also needed 1 redis cache + 1 memcache
- scottwick 3y agoNice to see that they're just using EC2 + S3 instead of a whole suite of niche cloud offerings that we see these days. Maybe this is mentioned in the linked presentation (https://www.infoq.com/presentations/Pinterest/ https://www.infoq.com/presentations/Pinterest/) but are they using managed databases? Or just running their own on EC2 instances as well? Curious to watch https://www.infoq.com/presentations/Pinterest/ https://www.infoq.com/presentations/Pinterest/ at some point.
- yeldarb 3y agoCool to read. Looks very familiar. I started an Easter Egg Hunt Facebook game out of my dorm room in 2008 called Hatchlings. We actually had them beat in the engineers to users ratio. When we had 5 million users and more pageviews/mo than the NY Times it was still just me (and I had hired my mom and sister part time to do customer support). It was running on a LAMP stack + memcached (first at DreamHost, then Joyent, and finally AWS). This article sparked in my mind a bit-twiddling hack I implemented to get a ton more scale out of our single giant MySQL box that survives in their code base to this day, much to every one of their dev’s chagrin (crazy to think that 15 years later the weekend project I launched still has tens of thousands of monthly subscribers). At Hatchlings, we had several hundred collections of Easter Eggs (usually 7-12 unique eggs in each collection). Each user would “open” and then hunt for a few collections at a time until they “finished” them. New collections would come out every week (to ensure there was new content daily). As we scaled both in users and number of collections I realized that the “normal” SQL layout you’d use to represent this (a users table, a collections table keyed on user_id, collection_id, and an eggs table keyed on user_id, collection_id, egg_id) was growing rapidly (each user was adding new rows every day) and needed multiple joins in our “hot” routes (when a user searches for an egg we need to know which collections they have access to, and when a user clicks an egg we need to know if it was new for them and if it newly completed that collection) which were being hit tens of thousands of times per second during peak times. Additionally, every egg collected increased the user’s score. So I implemented an optimization: store the collection unlocked and completed info compactly in the users table in bit fields. These were several 64 bit integers where eg collections_unlocked_1 being equal to 4 (binary 100) meant that the collection with ID 2 was unlocked. (collections_unlocked_2 equaling 1 would mean collection 64 was unlocked since there were 64 bits in 1). Your “active” collections were the bits that were 1 in an unlocked bit field but 0 in the parallel completed bit field. This did a few things: * Reduced the rows per user from total_collectionstotal_eggs to ~collections_in_progress (just in-flight collections) — about a 100x reduction in our total DB size * Reduced joins to 0 * Since all the “common” egg collections were already completed by all the most active users we didn’t have to check the eggs table for the vast majority of finds because if a collection was marked finished we knew you already had all the eggs in it * Made everything needed for the hot route compactly storable in a single memcached entry (which allowed us to greatly reduce writes because we would read scores and collections from memcached in the hot route and only write changes to the DB once/minute) It was a great speed and scaling optimization… But it was really tricky to deal with and reason about… and a binary arithmetic error could completely nuke months of game progress for people. And we also had to remember to add more columns to the users table every year or two or we’d run out of space for collections in our bit fields (we forgot about this a couple of times and it lead to downtime which tended to happen during the most peak times where we had hyped that we were going to release a ton of new content all at once).
- jdorfman 3y agoCirca 2013 I worked at a bootstrapped startup (MaxCDN). We got a trial request from Pinterest who put a small fraction of their traffic on dedicated edge servers and we couldn’t sustain it. They ended up going with EdgeCast that had the capacity and resources.
- Lamad1234 3y ago[dead]