10 ms·

Instagram was what, 12 employees when they got sold for a gazillion dollars? They all could fit into a van. Because they kept their system simple. It was (and s

• odiroot 7y ago
  I would really love to read their codebase.
  • pmarreck 7y ago
    What’s their stack?
    • grouchoboy 7y ago
      Primary I think is a Django project https://www.youtube.com/watch?v=lx5WQjXLlq8&t=10s https://www.youtube.com/watch?v=lx5WQjXLlq8&t=10s
      • pmarreck 7y ago
        I am straining to hide my lack of being impressed
  • papito 7y ago
    Read all about it: https://instagram-engineering.com/static-analysis-at-scale-an-instagram-story-8f498ab71a0c https://instagram-engineering.com/static-analysis-at-scale-a...
    • justaguyhere 7y ago
      typically deploying to production around a hundred times per day That is insane! We make a release once a week at work and things still go wrong sometimes. I am in awe how they are able to pull this off, especially at their scale.
      • snovv_crash 7y ago
        The more frequently you release, the less, or less serious, bugs there typically are, and the earlier you catch them. The faster development cycle also helps people invest in testing infrastructure more effectively.
        • K0SM0S 7y ago
          I'm trying to get a sense of magnitude, here. Let's say 10-20 commits per day per dev. Over 5~10 hours that's what, on the order of 1 commit-test-release cycle every 15 to 60 minutes? (subjectively for each dev) What do we actually write in that timeframe on average (thus including the ~90% of time we don't type code but think or read or test)? What's the "unit commit" here? So I'm thinking... let's take an example: today I'll refactor a few functions to update our model handling; I wish to reflect our latest custom types in the code. So it's a lot of in-place changes, e.g. from some list to tuple or dict; and the syntax that goes with it. No external logic change, but new methods mean slight variations in the details of implementation. - refactor one function: commit every testable change, like list to tuple? At least, I'm sure I'm not breaking other stuff by running the whole test suite every time it "works for me" in my isolated bubble. So I might commit every 5-10 minutes in that case. - Now I'm touching the API so I can't break promises to clients: I actually need to test more rigorously anyway. I'm probably taking closer to 20-40 minutes per commit, it's more tedious. Assuming I commit every update of the model, even insignificant, I get immediate feedback (e.g. performance dump), so I know when to stop, backtrack, try again? And it's always just one "tiny" step? - Later I review some code and have to go through all these changes. I assume it's easier to spot elementary mistakes; but what of the big picture? Sure I can show a diff over the whole process — I assume you'd learn to play with git with such an "extreme" approach. Am I on the right track, here? I totally get your comment but I'm trying to get a feel for how it works. I typically commit-test-release prod 3-4 times a day at most (on simple projects), and typically more like once every 2-3 days, 2-3 times a week. Which is "agile" enough I reckon... So I'm genuinely interested here. I feel there's untapped power in the method I'm just beginning to grasp.
          • snovv_crash 7y ago
• aguyfromnb 7y ago
  Sure,but the value of Instagram came from the fact it was a good idea, well executed,at the right moment in time. I'm not sure they managed to solve a ton of complexity with a small team.
  • dandigangi 7y ago
    Has to be true. They were dealing with major scale pre-purchase. Also... talented set of engineers.
• ignoramous 7y ago
  WhatsApp, similarly, had 30+ employees when they got acquired [0]. They built the fastest IM on the market with 450M+ users sending 1B+ messages everyday, and at one point surpassed Facebook in terms of number of images uploaded. The engs they had were world-class, so really, I think, saying microservices (or latest-fad) get in the way etc is disingenuous since you also require world-class talent to begin with (if you're going to keep the team-size small and yet be able to manage crazy scale) and a competitor willing to pay through the nose for the acquisition. [0] https://www.sequoiacap.com/article/four-numbers-that-explain/ https://www.sequoiacap.com/article/four-numbers-that-explain...
  • deleted 7y ago
    [deleted]
  • nickbauman 7y ago
    They chose Erlang. A language built for communication and managing wire protocols at scale. Which describes WhatsApp itself. That was probably the biggest impact single decision for WhatsApp technically.
    • ignoramous 7y ago
      I am not sure if WhatsApp engs chose ejabberd because it was written in Erlang or because ejabberd was the defacto implementation of XMPP. They stumbled upon and fixed bugs in BEAM/OTP at their scale [0][1][2]. They also ran FreeBSD (for its superior networking?) on bare-metal hosts running customised system-images [3] and employed networking experts at some point. [0] https://www.youtube-nocookie.com/embed/c12cYAUTXXs https://www.youtube-nocookie.com/embed/c12cYAUTXXs [1] https://www.youtube-nocookie.com/embed/wDk6l3tPBuw https://www.youtube-nocookie.com/embed/wDk6l3tPBuw [2] https://www.youtube-nocookie.com/embed/93MA0VUWP9w https://www.youtube-nocookie.com/embed/93MA0VUWP9w [3] https://www.youtube-nocookie.com/embed/TneLO5TdW_M https://www.youtube-nocookie.com/embed/TneLO5TdW_M
      • dnautics 7y ago
        I could be wrong but I think freebsd allowed multiplexing connections per port before Linux? Please correct me I am mistaken about this.
• nickbauman 7y ago
  But microservices are an example of simpler systems. Each microservice does far less than the whole monolith does. You can read all the code in ~15 minutes. I've worked at companies that have monoliths that are 50x more difficult to work on because of the size. Some of them millions of lines of code. Nobody really knows how they work anymore.
  • krab 7y ago
    The complexity is still somewhere. You can structure your monolith in a way that you can read each module's code in ~15 minutes. Each module also can be developed and tested in isolation. However, if the organization is not capable of structuring the monolith, why should it be successful with microservices? Such organization may lead to sharing data stores and custom libraries between microservices and that's when the real fun begins. Maybe even trying to deploy all of the services atomically to not worry about API compatibility.
  • darkerside 7y ago
    Underrated point, although incomplete from my perspective. 2 simple systems > 1 complex system. But 1 semi-complex system > 10 simple systems. Especially when you consider the points of integration between those systems increases geometrically with the number of systems.
  • dspillett 7y ago
    Microservices are simple components of what could be a simple or a complex system. If things are overly broken down then unnecessary complexity could easily be added.
  • hopia 7y ago
    Microservices make you architect your whole service differently. The communication is essentially asynchronous message passing which may or may not make things more difficult.
  • LargeWu 7y ago
    Microservices, often, is just another word for "distributed monolith". Sure you can read the code of a single service quickly, but often in practice it's not possible to just make changes to one service. There are usually both explicit and implicit dependencies that span many service layers. What you gain in readability I think you often lose more in maintenance overhead.
• krab 7y ago
  I'm now working with a system where we add microservices. In my experience, they allow you to split a team that then owns its whole deployment cycle. This allows for easier hot-fixing and dealing with database schema changes. There are now almost fifty developers and we already have a third (micro-)service. ;-) I agree completely that starting with a monolith is a win for feature development, performance and operations. But if your team grows, your release cycle will slow down unless you decentralize it.
  • nradov 7y ago
    That has nothing to do with the release cycle. A large monolith can still do continuous delivery and release multiple times per day. What tends to slow down is the feature delivery cycle.
    • krab 7y ago
      Would you care to explain your reasoning? This topic genuinely interests me. My experience is quite the opposite. To deliver a feature, you still need to integrate changes in multiple services. To do it in a monolith is IMHO easier. You have a single artifact you can test, the probability you have the right automation is higher. Also, the devs will run bigger part of the system in the development version. What I was talking about was the latency of smaller changes. Think of bug fixes. Somebody has to notice it, triage, fix, (wait for automated tests) and deploy. Often, a bug fix affects just one service. With smaller services, this chain is simpler.
      • nradov 7y ago
        There are no hard rules. In general a microservices architecture tends to enforce segmentation which makes it faster to iterate on features that only impact a single service. But coordinating larger changes across multiple services owned by separate teams can certainly be slower. Monolith architectures often degenerate into the proverbial "big ball of mud" over time. But if the team has the discipline to maintain proper design through continuous refactoring then they can retain the ability to deliver features with short cycle times.
• serbrech 7y ago
  The thinking is backwards, if you are only 10 people, there I no need to shape the system in a microservicr fashion. You just extract the pieces that need scaling. If you are 60 devs, you need to split the system so that everyone can work on it without walking on each other.
  • therealdrag0 7y ago
    Also you can have so few devs when your value is in your network. Most other businesses' value is in their features. We have customers constantly begging for features, so we have to have more engineers to produce more value to our customers.
  • papito 7y ago
    As someone said, microservices is a technical solution to a people problem. Devs don't want to talk to each other so they wall off behind their own API. Boom, no need to talk to each other. Ever. Or is there?
    • barrkel 7y ago
      Requiring everyone to talk to everyone so everyone has global context isn't just "devs don't want to talk to each other"; it is actually an information dissemination and coordination problem which scales non-linearly (at least n^2), and needs some kind of modularity to be tractable to normal humans. Microservices are like modules but for SaaS rather than shrinkwrap, and are where you end up when you follow SLAs, encapsulation of resource consumption, etc. to their logical conclusion.
      • majormajor 7y ago
        "Microservices" don't guarantee that everyone doesn't have to talk to everyone. Good thoughtful design is necessary regardless of how you're building/organizing/deploying/operating the code.
      • vikiomega9 7y ago
        > scales non-linearly Indeed, https://en.wikipedia.org/wiki/The_Nature_of_the_Firm https://en.wikipedia.org/wiki/The_Nature_of_the_Firm
• jjtheblunt 7y ago
  Excellently said
• jayd16 7y ago
  In my experience you do end up having external dependencies and more than one service. You do end up breaking out some code into special instance types, (high ram for video processing or what have you). These are problems you do have and do have to solve so you might as well come up with a plan. Deploying microservices really isn't that hard once you make it routine, imo. But what do I know? I would not have expected Instagram to run things like user login on the same instance as photo upload and processing.
  • chrismarlow9 7y ago
    microservice vs monolithic is a shade of grey. what if all the code is in one codebase, but distributed to varying instance types that use some config flag to say the role it plays in that context. is that monolithic or microservice? I'd say the code is monolithic and the architecture is microservices. So it's some hybrid version. what you're talking about (isolation of responsibilities) can be done and still be considered monolithic. you can also use the exact same instance type for everything and still be considered microservice. I think what we're really talking about is containers vs machine images. And personally I think containers right now are suffering from the same abuse/hype that datastores suffered like redis/mongo/couch etc. Sure they have an application and solve problems, but they're being over used to the point of causing technical debt.
  • papito 7y ago
    It doesn't mean that they do. Their system is partitioned to a point, and something like image processing is very easy to offload and expose via HTTP, for sure.