10 ms·
How Instagram scaled to 14 million users with only 3 engineers
- raydev 3y agoWorth keeping in mind that during this period, there was only one frontend: the iOS app. It also offered a fraction of the functionality that modern IG/Twitter/FB/TikTok have now.
- acconrad 3y ago> 30 million monthly active users when it was acquired...[they] had just six generalist developers [1] So they got to 30mm users with 6 engineers...scaling linearly with 5mm customers/engineer. Incredible. [1] https://review.firstround.com/how-instagram-co-founder-mike-krieger-took-its-engineering-org-from-0-to-300-people https://review.firstround.com/how-instagram-co-founder-mike-...
- deleted 3y ago[deleted]
- Sparkyte 3y ago14 million doing not so complex things is an easy achievement. When you get into a lot of microservices providing tons of features your teams will balloon.
- Scarblac 3y agoThat's why it's important to keep the number of devs low, it makes it less likely that one starts talking about microservices.
- pestaa 3y agoIt's more important to keep the number of features low. Good devs talk about aligning the architecture to requirements, and that sometimes includes microservices.
- Sparkyte 3y agoThis is true. I completely agree with the minimized scope. However a dev needs to be careful to not let their service just sit unmaintained. New teams will always need engineers to maintain and improve.
- Sparkyte 3y agoYou talk as if monolithic apps are vastly superior. To be forward it depends entirely on the purpose and life of the application. It is about whatever shoe fits the design.
- Aeolun 3y agoSure. It’s just that 99% of applications work fine (or better) as a monolithic design.
- Sparkyte 3y agoDepends on the purpose of the application though. Monolithic is a good architecture when you have a few purposeful features and functions. But when your design relies on many services to provide a wide variety of features you need to break out this design to allow teams to operate independently. Mini monoliths are more popular today than traditional monoliths of the old.
- qaq 3y agoYou can split things up you don't have to though. Teams operating independently is fairly orthogonal to this.
- Sparkyte 3y agoYeah no I get you. You just want a monolith to be purposeful when you design one. Not multi-purposeful. This is also at the limitation of a programming language. I am kind of kubernetes guy, but I am dying because it relies heavily on a virtualize network distributed. It would drastically increase the performance if kubernetes clusters were built like monoliths and each kubernetes node handled traffic independently. So of like keeping it all in the same rack. Only leaving the rack if needed. But I keep seeing bad technology decisions repeated over and over. I stopped pushing because some person with a bigger title would say this is good design. Big kubernetes clusters eventually fail. Multiple small clusters survive.
- sleiben 3y agoWe use the microservices architecture as a single team and don’t have any issues with this for many years. The key is to have a monorepo and stay consistent by following strict coding guidelines. In my opinion it makes the backend way more resilient than a monolith. Don’t kill me for this opinion please ;)
- BoorishBears 3y agoHow does it make the backend more resilient than a monolith? Do you not realize you have multiple instances of a monolith or something?
- ahoka 3y agoBad input crashes app, monolith fails over, other instance crashes. Full outage. Assuming proper vertical separation, this risk can be reduced by microservices.
- elktown 3y agoIt might just as well be increased by microservices due to fragile dependency chains and just different people working async and what not.
- BoorishBears 3y agoThis is not what microservices solve. There'll always be critical microservices that keep your app running. It doesn't matter if all your other services are running if the one serving up core functionality goes down. If your engineering rigor is so poor that you can't get reliable failovers with a monolith, god help you keeping microservices running.
- devjab 3y agoIn two decades of development I've never once seen a monolithic architecture that with some form of shared database not be terrible for the business it powered. I certainly understand why it's very common, it's what's still being taught to most CS students in my country after all, and, it's frankly a lot easier to implement. The result, however, is always the same. It ends up being a mess where nobody can do anything because the data structures are so intertwined (and undocumented) that nobody knows how it's actually used. What happens is that monoliths become magnets for business logic, and then you bottleneck every change into requireing a select few members of your organisation. As time goes by, you end up with a giant turd that stagnates and directly hurts your business. Not by intention, but because that's exactly what happens when you make things complicated. It's important to keep in mind that this isn't a technical problem. It's an organisational problem. In fact, there is no technical reason why monoliths would be an anti-pattern, which is likely why they are still being taught as though they weren't at many universities where professors still naively think that the MBA's aren't going to cost-cut IT at every opportunity even though their entire organisation is made up of employees who spend 100% of their working time on IT devices of some form. Similarily, Microservices, aren't really the "technical" response to this. It's how IT and digitalisation had to evolve to keep up with business demands and better generate value. The simpler and more decoupled you keep things, the better you'll be able to respond to business needs. Sure, there are a gazillion different ways to do Microservices wrong, and if you do it wrong, then you'll likely be in the same mess that you would be with a monolith, only so much worse, because now you have 9 million tiny monoliths and shared databases. Luckily we still live in a world where everyone is somehow still OK with IT not working. We went to an appointment that isn't relevant the other day, and they had a tablet where you could register your license plate to avoid getting a parking ticket. It didn't work, so we talked with the receptionist who was like "yeah, it does that all the time, don't worry, if the systems are down then they can't give out tickets"... Fine for us, but think about that... It turned out the system was down in my entire city, which means that all those hundreds of employees who are out handing out tickets had nothing to do while their IT system was being fixed, hell, the entire company wasn't generating income for my city while their IT was down, and this was a regular occurrence? What my point with this is, is that you can do things really wrong, and still be a "successful" company, it's just that the companies who manage to generate value better (which is frankly always microservices of some form) tend to simply do better. But like I said. You can do "microservices" in a million different ways. Running two different django backends to handle different parts of Instagram could be considered having two microservices after all. The importance is how you deal with the needs of your organisaiton in a rapid fashion.
- qaq 3y agoWell it's not like you have have to get into microservices
- koonsolo 3y agoHonest question: what justifies microservices in the popular companies compared to instagram? Twitter: just sharing a bunch of text. Netflix: platform with static content, where sharing isn't even possible. TikToķ: Instagram for video, so basically just bigger files. WhatsApp: they also pulled it off with a small team. If you talk about Roblox, now there's a challenge! And they pulled it off with way less engineers than Twitter.
- the-smug-one 3y agoFor Twitter and TikTok, I bet that the reasons are B2B offerings and not their B2C offerings.
- siva7 3y agoCan someone confirm this story who has inside knowledge?
- engineercodex 3y agoSources are linked at the bottom of the article!
- kristopolous 3y ago"When you choose the right people you'll need only a few. When you don't, you'll need them all"
- quickthrower2 3y agoThe right idea too.
- Sparkyte 3y agoI think this idea hurts the industry. Don't get me wrong. I think it is important to hire the right people, but if you don't hire enough people this 3 man job is 80-100 hours a week per for months to years. Look how X has diminished in quality as Elon started slashing team sizes. Then when you design more features, security and other various systems to serve the customers it will creep in complexity. You can not escape that no matter what you do.
- frenchman99 3y agoI have noticed absolutely no difference with Twitter/X. I am a casual user, sure, but for me it seems to work well.
- andrewinardeer 3y agoThere was a bit of downtime and some bugs did make it into their prod env after the restructure. These bugs were rectified fairly quickly too. I generally agree with you on your comment. I am a casual user of X these days and the site seems to be humming along without any user facing engineering issues. With his products such as Tesla, X and Starlink touching millions of people daily Elon is an easy to reach punching bag. Even if he is somehow instrumental in solving the massive feat of putting humans safely on Mars there will always be people on the side lines having shots at him.
- rpastuszak 3y agoAnecdata for sure, but I use Twitter every day and features such as video upload just stopped working for me (failing silently), not to mention the site crashing or becoming completely unresponsive every week or so.
- deleted 3y ago[deleted]
- _xivi 3y ago> we can assume that Instagram was written using Objective-C and a combination of other things like UIKit. How can they know the internal infrastructure but have to assume the app language? Edit: So the entire piece is taken almost verbatim as-is from a couple old articles on instagram engineering blog. It might as well just redirect to: https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad https://instagram-engineering.com/what-powers-instagram-hund... This is against the guidelines: > Please submit the original source. If a post reports on something found on another site, submit the latter.
- rmetzler 3y agoInstagram engineers themself wrote a bit about their backend infrastructure. One of the more important topics was how they shard the data [0] and this is also linked to in this blog post. [0]: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c https://instagram-engineering.com/sharding-ids-at-instagram-...
- Sparkyte 3y agoIf it wasn't just an image site the potential for hotspotting would be insane! Size isn't a bad thing anymore since price has dropped exponentially since the inception of Instagram. I am positive they would use another modern technology today if it was present in the past. Fantastic read though.
- _3u10 3y agoThe potential for hotspotting decreases with the number of inserts per second. Like if you only did 1 insert per second and timed it right you could put all of those inserts on one server, but this would likely not overload the server. It's virtually impossible for anyone to hotspot in a meaningful way with this system.
- Sparkyte 3y agoNah this will totally be an issue. You can be on the extreme end of replication or the extreme end of sharding and experience performance problems. Sharding is more likely to hotspot depending on where hot data is consistent. The solution in most cases is a simple database that acts as a pointer database user db -> user's db. That is generated on the creation of a user. From here you create some simple cold storage models ( if user isn't active ) and some warm models which will scale out the db if the user's db grows it shards and replicates for more read access. But the last thing you want is to slow replication or have one DB that can't move to balance resource utilization. There are some new DB tech that does this without even sweating the deets.
- 0wis 3y agoInteresting I wonder if its seems easy because it’s explained simply or if it really is simple to put in place. I want to make a clone now, just to try. At least for this inspiration, this article was well worth the read. Thanks !
- rmetzler 3y agoInstagram engineers found some remarkable simple solutions to some hard problems. It's not easy to come up with these solutions. Designing the IDs for example is no small feat, but since this is now common knowledge it's probably not too hard to build a similar system. To get traction from users is the real challenge.
- koonsolo 3y ago> To get traction from users is the real challenge Unfortunately yes. Scaling is a problem I would love to have :D.
- BoorishBears 3y agoWhich makes things like Vercel so depressing to me. You've got a company paying off influential people in a space where people are looking for guidance, convincing them that they too have hard problems that cannot settle for simple solutions. Selling the narrative that developers need to be all in on the most irrelevant aspects of building a product, and ignoring the fact that if you instead focus on building simple, easy to maintain software, the fact your LCP isn't hyper optimized by some newly invented mental model for app development won't matter: Google (or any search engine for that matter) will not ignore the fact people just actually want your content. They do not care how great your web core vitals are if you waste a bunch of time bending over for some irrelevant bullshit problem instead of talking to users and iterating.
- wild_egg 3y agoThe fun part is often all that bullshit hyper optimized complexity can still be beaten on all the web core vitals by a humble LAMP server So not only are you wasting your time on all the wrong things, you're also gaining nearly no benefit from it I'm pretty convinced that all these shiny new hotness tools and frameworks of the past decade are actually just meant to sabotage competition and small companies
- luismedel 3y agoI have chills thinking about the oncall schedule of these 3 colleagues...
- IndySun 3y agorelated/mentioned - https://engineering.fb.com/2013/04/15/core-data/scaling-memcache-at-facebook/ https://engineering.fb.com/2013/04/15/core-data/scaling-memc...
- rich_sasha 3y agoStories like this kind of make sense to me. 3 people is very few but I guess they really knew what they are doing. Meanwhile, all these orgs with essentially a CRUD app, with 1,000s of engineers..? That I never understood.
- lovich 3y agoNot engineer driven orgs _and_ their product people don’t know how to drive the product or set requirements, or engineer driven by the type of engineers who love to tinker without making any real progress business wise
- onurcel 3y agoThese CRUD apps need complex business rules, requiring expertise in the domain and making them configurable on the application level for customer while trying to keep the app not bloated. Scaling is not the only challenge engineers face, but somehow it's the one that is mostly praised.
- throwaway290 3y agoThey also need to respond to customer requirements, which IG never needed to do while they had no actual customers. And as soon as fun was up and IG had actual customers (spoiler alert, advertisers) what a surprise 3 devs was not enough. They also need to quickly respond to downtime, because unlike IG if some of those CRUD apps go down in B2B world you are often losing customers actual money not just ad views
- sealeck 3y agoAd views translate to actual money??
- throwaway290 3y agoSure. If you are full on marketer you can say "not see an ad for service" is as bad as "service does not work". But anyway there was never a period where Instagram had a tiny 3 dev team and handled ads at the same time. 3 devs only worked back when there were no customers, no ads, no profits and no real responsibilities.
- prakhar897 3y agoI'm more interested in how they got 14 million users with only 4-5 people. what marketing tactics did they use? I assume those should be milked to death by now.
- dboreham 3y agoLuck.
- eviks 3y agoluck is being milked to death, but it only favors the living
- passion__desire 3y agoWhatsApp used to refresh its app by tweaking its name in early days on Appstore so that it appears new in the algorithm to be recommended. I guess that practice is now prohibited on both the stores.
- systemvoltage 3y agoI presume this architecture would look very different today? With supabase, spanner, cloudflare, etc? When scaling, the database bit is the least clear to me. How do you create an "isolated" database per customer?
- kgeist 3y agoI can't find where it says it's per customer. It says they have "a few physical shards". From what I understand, they had a few physical DB servers which stored their own subsets of data. When a request came in, the appropriate DB server was selected based on photo or user ID. P.S. Here's more information: https://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram https://instagram-engineering.tumblr.com/post/10853187575/sh...
- deleted 3y ago[deleted]
- Octabrain 3y agoI find the simplicity of the stack brilliant. It makes me to think now, to which extent, the industry nowadays simply suffers from a mix of lack of knowledge, CV driving design and big players in the game trying to sell you overkilling solutions and approaches for their own economical benefit. Have we perhaps, fell into a big enchant and, at the end of the day, 99% of companies out there could just use a classic LAMPish kinda stack managed by 3 or 4 dudes?
- ahoka 3y agoI just imagined trying to sell this architecture for a new product in an imaginary company, an amalgamation of every place I have ever worked: You have to change to Azure, because we are Microsoft partners and we have free credit. The credit is not too much though and we have to spend the same money on useless trainings so we keep being partners. 4 core and 8GB should be plenty for your dev VMs, that’s the largest we can run on free MSDN accounts. Have you tried the managed API gateway? Why not? You should use managed caching on the edge. We already have an on-prem SQL Server, use that to cut costs. Yeah it runs on Vmware and network storage. Do we have support contracts for Nginx, Ubuntu? We have RHEL licenses, so you should use that. Can we run this in our OpenShift cluster instead? To cut license cost it will be co-deployed with the developer envs of other product, but just set resource limits. Yeah, we only have NFS storage. S3? Just use a PVC, it’s the same. We decided on Datadog for unrelated product A and B, so you must use that. The license is expensive, so only log errors please. We use Kafka for the workqueue, but limited to 2 cpu cores to make it cheap, so please make sure not to send too many notifications. Python is not for production things, we will assign an offshore team to rewrite it in Spring Boot. We target Java 11, because our productivity increasing libraries are not yet updated. Minor change needed for deploy: every service should build it’s own RPM package in it’s dedicated git repository. You need to submit the architecture diagrams and service documents next week, thanks for the meeting. What did I miss?
- stock_toaster 3y ago> What did I miss? The security team is getting weird reports from their internal nmap scanner that is constantly scanning this network. No, they don't know which of their vendor automated scanners are doing it, no they can't turn it off, so please add code to specifically ignore those scans. Our VMware cluster has plenty of compute left, but it is out of storage space because there are hundreds of VMs from other teams and nobody knows what is still being used, so you can't get any more dev instances until we get more storage added to the NAS (in the next quarter's budget). Any unrelated director somehow ended up seeing this diagram and had some "suggestions". Please implement these changes asap.
- ksec 3y agoI wonder how far they could scale with modern technology with the same architecture and 3 engineers. Not only are Django, Postgres, Redis so much better now. The hardware per server is also at least 3-5 times faster. 100 million?
- rapnie 3y agoHere's recent insiders experience on developing Meta's Threads app on top of Instagram infra: https://newsletter.pragmaticengineer.com/p/building-metas-threads-app https://newsletter.pragmaticengineer.com/p/building-metas-th... > In July of this year, Meta launched its latest mobile app, Threads, a microblogging service and new rival to X, formerly Twitter. In the first five days following its launch, the app achieved 100M downloads – a new record for the company by some margin. Meta’s previous record for new app installs in the first 5 days after launch was 1M. Built with a slightly larger team, considered an agile team: 3 product managers, 3 designers, about 60 engineers.
- djtango 3y agoI tthink the reality of modern development, especially one plugged into the Meta ecosystem is that there is probably a tonne of integration work to be done. - monetization - finance - analytics - ad placement - ad bidding - android client - apple client - browser client Just off the top of my head in 2 mins that's a few of the extra concerns I can come up with... I remember meeting someone who was responsible for writing an ultra performant JS WhatsApp client for firefox OS. When you have 2B users, the long tail is long... OG Instagram was pre-monetisation. So yeah maybe a simple image hosting service needs fewer engineers but maybe a profitable business that can monetise the service needs more...
- JimDabell 3y agoRemember Instagram was iOS-only for a good long while. So no need for Android developers or front-end web developers.
- djtango 3y agoDidn't know that, I still don't use Instagram lol. So yeah exactly - it's easy to run a simple app with a few devs. Iirc in Singapore, Netflix has a reasonably sized team just dealing with payments. Presumably for a global business, payments is a pretty gnarly long tail of integrations (never worked with payments so pretty ignorant on the subject) not to mention I wonder whether they also are onboarding content distributors onto some kind of automated payment platform so they can very quickly churn content libraries or attribute some kind of viewing related payment... I guess tldr doing business is complex and needs lots of engineers vs making a small and focused app.
- tiffanyh 3y agoThreads I find it interesting that Meta choice largely this same tech stack for their newly created Threads service.
- JimDabell 3y agoIt’s not that they made the same choices again so much as they just re-used Instagram code and infrastructure.
- tiffanyh 3y agoThey had the choice of using FB/Hack stack or IG/Python stack, and they choose IG/Python … which is the interesting thing.
- JimDabell 3y agoThe product is basically Instagram but with text instead of images. I would expect that would be the primary motivating factor in choosing Instagram as a starting point rather than the language.
- tiffanyh 3y agoSource: https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad https://instagram-engineering.com/what-powers-instagram-hund...
- mcsoft 3y agoThe article says nothing about how they instantaneously updated millions of user feeds. It was the most challenging task, as it's way easier to scale reads than writes in distributed systems. Rumor has it early Twitter had a target of 5 sec to update everyone of 50M fan feeds when Justin Bieber touched a screen. I would love to hear some technical details on how they did it.
- pnt12 3y agoI remember reading a case study from their engineering blog about this a few years ago - I couldn't quickly find it but maybe it's still out there. It was some think about optimizing for read speed, because one write from a celebrity would cause thousands or millions of reads.
- unnouinceput 3y agoAlso this was in the days before it was acquired by FB. Hence why only 3 was need it.
- wucaworld 3y agoTerrible pop up on that site. Designed by an a hole.