10 ms·
How we ended up with microservices
- jdlshore 11y agoGreat read. I found it interesting that they basically replicated the XP playbook: cross-functional teams, continuous code review with pairing, collective team ownership of code and results, bounded contexts. It makes me wonder if the "microservice" part of it was necessary. What if they had produced "microlibraries" rather than "microservices?"
- numbsafari 11y agoI think you have to read to the part where he talks about the deployment impacts. A key part of what happened was they needed deployment flexibility as well. From their perspective, they would need to implement the same basic infrastructure to achieve that flexibility: But even if everything went smoothly, we knew that the current code for the monolith had to be refactored anyway. The code had suffered a lot during the past few years, tech debt everywhere. Besides the mess we made ourselves, we still had to update it from Rails 2.x to 3, and this is a big migration effort in itself. This was probably critical their ability to adopt new technologies, like Clojure and later Scala.
- crdoconnor 11y ago>It makes me wonder if the "microservice" part of it was necessary. What if they had produced "microlibraries" rather than "microservices?" I've wondered that every single time somebody has touted the benefits of microservices. The people who have really 'succeeded' in it seem to conflate the benefits of looser coupling between dependent software systems (which is always a good thing) and making those systems talk to one another over a network socket (which isn't necessarily a good thing).
- nulltype 11y agoYeah I'm pretty skeptical of the benefits. It seems like better isolation within your monolithic application solves a lot of these issues, no need for a network socket. If you can't isolate things well enough for some reason, maybe it makes sense to have separate services (maybe run them all on the same machine, deployed at the same time and talk over a local socket?), but even then I suspect you just need normalservices not microservices.
- Already__Taken 11y agoMaybe adding the network socket makes the isolation within the code a requirement as opposed to a best practice. By this mechanism maintaining isolation is a requirement and not something someone can bypass "just this once" but fix it later. Perhaps you're both right.
- crdoconnor 11y ago>Maybe adding the network socket makes the isolation within the code a requirement I've worked with microservice applications that had extremely tightly coupled services that were all highly dependent upon one another. Adding the network socket layer just magnified the problems caused by the tight coupling. So yeah, if you want to make your life even more miserable, split up a tightly coupled "macroservice" into a series of tightly coupled microservices.
- nulltype 11y agoSo I guess you're saying that microservices are pointless, the real problem is tight coupling?
- crdoconnor 11y agoKind of. There's a flood of blog posts (including this one I think) that have conflated the two, probably unintentionally. I'm happy that decoupling their systems worked out well for them. I'm not so happy this is fomenting a new fashion for creating unnecessary network API end points. That isn't to say that you should always combine your services into one big mega-service. Just that dividing up services should be something you do only when it becomes obviously necessary and for good reasons unrelated to coupling.
- tomchristie 11y agoThe benefits of microservices aren't architectural. The point re. library APIs vs services APIs is completely correct, because that's actually not what the microservice style addresses. The point is rather organizational/operational - Being able to independently deploy, scale, monitor and manage different parts of the system. That allows you to distribute ownership of the system, and also allows partitioning, so that a failure in one component need not effect the whole system.
- qznc 11y agoorganizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations —M. Conway
- laredo312 11y agoCould you give an example of an architectural benefit vs operational benefit?
- tomchristie 11y agoWhen I say 'architectural' there I mean that as in the underlying design. How the system is modeled and and what different objects and interfaces exist in that model. Eg. Say you have a billing component in your system. You may have the same underlying billing component in either a monolith or as microservices. It may be a nicely decoupled in both cases (either existing as an independent module/library in the monolith case, or as an independent service in the microservices case.) The (potential) benefit that's afforded from the microservices approach is not that there's a better underlying design there, but that a single team or developer can properly take ownership of deploying, scaling, monitoring and managing the service, independently of the rest of the application. (Plus scope for partitioning and graceful partial failures vs whole system failures)
- rafaelferreira 11y agoI've tried that in the past. Even if modules aren't tightly coupled, deployment is, so different teams need to synchronize at deployment time. Resource isolation is also a big problem, if a module update introduces a performance bug, it will affect everything else. Yet another problem is keeping shared libraries in sync; if you want to update a core lib for component X, it will need to be updated (and tested) for everything else.
- crdoconnor 11y ago>I've tried that in the past. Even if modules aren't tightly coupled, deployment is, so different teams need to synchronize at deployment time. No they don't. There's no reason why two different teams can't schedule an upgrade of the same service at different times. The riskiness of this is entirely dependent upon how good your integration test suite is. >if a module update introduces a performance bug, it will affect everything else. The module will still affect everything that is dependent upon it if it is rebuilt as a microservice. You're just moving the performance problem from one place to another. >if you want to update a core lib for component X, it will need to be updated (and tested) for everything else. Ok, so upgrade the library and run the full set of integration tests.
- tomchristie 11y ago> The module will still affect everything that is dependent upon it if it is rebuilt as a microservice. Your services are running on different servers (or containers) to each other, so they're partitioned. If one service has a bug that introduces a catastrophic error that takes all the server resources you'll either: Monolith: Bring down the service completely. Microservices/SOA: Timeouts to part of the system, and partial loss of functionality. (Assuming you've done a decent job of engineering for partial failure)
- ken47 11y agoThere's absolutely no reason why you can't timeout within your monolith.
- deleted 11y ago[deleted]
- jrochkind1 11y ago>It makes me wonder if the "microservice" part of it was necessary. What if they had produced "microlibraries" rather than "microservices?" So the OP did discuss considering micro-libraries (perhaps via Rails engines), but they decided not to. > We discussed using Rails engines and various other tools to implement this... At the deployment side, we would need to make sure that a feature can be deployed in isolation. Pushing a change to module to production should not require a new deployment of unrelated modules, and if such deployment went bad and production was broken the only feature impacted should be the one that suffered the change... It goes on a bit. I think the reasons against this approach for them aren't entirely clear in the discussion, it would be good to hear more. Although as a Rails dev myself, this one rings true: > The code had suffered a lot during the past few years, tech debt everywhere. Besides the mess we made ourselves, we still had to update it from Rails 2.x to 3, and this is a big migration effort in itself The ability to migrate from Rails 2 to 3 one service at a time is actually a pretty huge benefit, since that migration was monstrous. This is probably generalizable. One other thing their ultimate microservice approach got them was the ability to write different services in different languages, and thus gradually transition to clojure/scala. I don't know if that was part of the original analysis; or if everyone would consider this a benefit. :) But it worked out for them. Lately the reason microservices are a _pain_ is pretty clear to me; it is good to get an essay like the OP grounded in very specific experience on how microservices worked out very well for them. It does seem to make sense by the end. As the OP also says at the beginning, this is as much for organizational reasons as technical reasons. I suspect you need a fairly large team, where the microservices can be divided up amongst different developers, as in OP, before the benefits can start to outweigh the costs.
- jacques_chester 11y agoIt's easy and obvious to set up queues and hand offs. It seems efficient and logical and it looks good on a whiteboard. The throughput of work in flight is of course amazing. Then, much later, you realise latency sucks. People don't care that you have 100 features coming in 2 years. They care that you have the killer feature now. So that's what counts here: latency to customer value. The time between picking the next most valuable feature and putting it in front of paying customers, is the one loop that needs to be optimised. I'm lucky to work Pivotal Labs. Our archetypal team is one product manager, one designer, two engineers from us and two engineers from the client. Currently we are experimenting with building teams with embedded data scientists as well. It works because we can, as an autonomous unit, do all the work that needs to be done to take a feature from ideation to production. When you have that, it's amazing what you can get done. I know I sound all-knowing and clever and stuff. But if I'd been setting up a software project before working at Labs, I'd have done a lot of the same things as this article describes as the starting point. [edited to try and remove the impression that I would've Done It Perfectly From The Beginning, which is the opposite of what I was trying to say]
- marktangotango 11y ago>> The time between picking the next most valuable feature and putting it in front of paying customers, is the one loop that needs to be optimised. Well said, BUT how to determine that next most valuable feature? I worked on a product once that required quiet involved sales engineering support, feedback from the sales engineers doing POC on customer sites was, to me, the most valuable feedback of all. But for other low touch or consumer targeted products, I'm baffled how to do this.
- jacques_chester 11y agoSometimes you do it by user interviews, sometimes by paper prototypes, there's a huge grab bag of options. My peers who focus on product and design would be better at telling you their magic, but from what I've seen it's deep and broad and the results are often surprising. We've saved customers a lot of money by discovering that wanted to build something that their customers didn't want to buy. From the engineering side, my role is to make development fast enough that product and design can find out whether they were right by having something in production quickly. This obviously varies according to project. For a highly gatekept ops culture, I could have something in an acceptance environment in hours that takes days, or weeks, or months, to reach true production. But sometimes we get true CD and a team of designers and engineers can put tested code in front of users the same day, and have meaningful metrics shortly afterwards. On my first project we had a hard, immovable 6 week deadline. We had a path to production after 3.5 weeks of pushing to get through complicated deployment hoops. Once we had the path to production our product manager and designer were able to perform laser-focused surgery on our backlog in the final weeks. They knew, sometimes within hours, what changes were being used and what had not worked. It was a real eye-opener for me. Still a standard I hope for on any project.
- pimlottc 11y agoMight be a clearer as "How Soundcloud ended up with microservices". I was expecting a piece on the general trend within the industry.
- huckyaus 11y agoThe title originally contained [about SoundCloud] at the end, which is the only reason I clicked the link (I'm a front end dev and frequent SC user). Not sure why it was removed.
- masklinn 11y agoSome HN mods are anal about not editing original titles, even when it makes for nonsensical HN titles.
- cam- 11y agoFashion.
- pc86 11y agoOh absolutely, $THING is fashionable and is therefore ridiculous and nobody should use it. It couldn't possibly be that $THING is incredibly useful under a specific set of circumstances, and that caused $THING to be fashionable. Just because something is fashionable doesn't mean that's the reason people decide to use it.
- cam- 11y agoSoftware suffers from fashion. If you look at the core java libraries they are a history of software engineering fashion. I would argue that fashion drives most of our choices. We are not immune to it, if anything we are about the same level as teenagers IMO. Tech for whatever reason is very susceptible to fashion.
- amirmc 11y agoTo be fair to the article, it should be obvious that it's about SoundCloud by the end of the first paragraph (& thus the 'we' refers to 'us at SoundClound'). I do agree that the title here is out of context but it seems any modification of titles is seen as editorialising.
- deleted 11y ago[deleted]
- curtis 11y agoThe Reddit post has some pretty good comments too: https://www.reddit.com/r/programming/comments/3k8sb2/how_we_ended_up_with_microservices/ https://www.reddit.com/r/programming/comments/3k8sb2/how_we_...
- ianpurton 11y agoI've never been convinced about splitting developers into front and back end. I see no reason why a decent developer can't build the front and the back end for each piece of new functionality. This removes all the communication issues.
- radicalbyte 11y agoFirst: it's all about reducing latency. Split the work, implement in parallel then you'll ship in "half" the time. Second: Good developers are hard to find. Backend requires both good CS background (algorithms, security) as well as development skills. Frontend requires some feeling for design and can tolerate poor development skills.
- gkoberger 11y agoFirst: Read the Mythical Man Month Second: Don't insult front-end developers. I do both back and front, and I find front-end to require much more skill. Nobody notices if a backend feels slightly off, but it's very obvious when an interaction was built by someone who isn't a great front-end programmer. Both sides are necessary and difficult in their own way; they just require different skillsets.
- StavrosK 11y ago> Nobody notices if a backend feels slightly off Oh you'll notice it in three months, when you have to write that new feature.
- gkoberger 11y agoI was talking about end users, although having to rewrite bad code is true for both front- and back-end.
- madeofpalk 11y ago> First: Read the Mythical Man Month Actually, that refers to adding more resources to _an already late project_. It explicitly mentions that starting out with more resources can speed up development. In saying that, 9 women can't make a baby in one month.
- vosper 11y agoThis is really great. The micro services debate has just kicked off on my teams, and people are starting to make their cases. While the details of the article differ from our systems (less web, for one thing) the problems they experienced sound pretty familiar to me right now. I'll be sending this around the office tomorrow.
- dzhiurgis 11y agoOk, so when are you going to fix search though? There is not way to search by tags & filter by length. Consider this query: https://soundcloud.com/search/sounds?q=*&filter.duration=epic&filter.genre_or_tag=dub%20techno https://soundcloud.com/search/sounds?q=*&filter.duration=epi... It includes shows tracks that are clearly short. Alternative is this: https://soundcloud.com/search/sounds?q=dub%20techno&filter.duration=epic&filter.genre_or_tag=dub%20techno https://soundcloud.com/search/sounds?q=dub%20techno&filter.d... But then not all dub techno tracks have it in the title. Also, default grid view when viewing your likes is super annoying. Any chances to change into list?