3 ms·

“A typical organization in the 2020s has: Hundreds of microservices” Can people with expertise in this situation speak about how typical it is? I guess if you

by aidos 6y ago

“A typical organization in the 2020s has: Hundreds of microservices” Can people with expertise in this situation speak about how typical it is? I guess if you have all those services then you need your IDE to help - but surely this isn’t super common? If it is, what sort of size of micro service are we talking here? I’ve worked in orgs with 3M line codebases, and I’d struggle to find a hundred ways to partition the code naturally.

blovescoffee 6y ago
I would say this is somewhat accurate at my company (thousands of engineers). It depends on what you quality as a microservice.
closeparen 6y ago
I work in a large microservices environment. It’s very important that IDEs don’t make it easy to reach inside other services. The whole point is to create cognitive barriers at the public APIs. The pain of coordinating changes across services is a feature: it is telling you that your architecture is wrong.
- aidos 6y ago
  And what qualifies as a micro service in your environment? Can you give a few examples? How many people work on each micro service? Etc etc. Would love to hear more details so I can build a better mental picture.
  - reissbaker 6y ago
    I'm not the OP, but I can give you some examples from when I worked at Airbnb a few years ago: * Each team (let's call it eight engineers per team on average) typically managed a few services. * Services were typically ~thousands of lines of code to ~tens of thousands. Rarely, there were very small services that were hundreds of lines of code. There was a giant service -- the original Rails app -- called "Monorail" that had millions of lines of code, but was actively being dismantled. My understanding is that there is now a code freeze on Monorail, and the only changes allowed are changes that remove code from it and move it to a smaller service (either an existing one or a new one). * Services communicated with each other over HTTP or Kafka, typically using Thrift, although also sometimes using JSON. Airbnb had an in-house "service mesh" (developed before the term service mesh existed) that abstracted away individual machines and was capable of operating at the HTTP or TCP layer. A few classic examples of mostly-single-responsibility services: * Prometheus (which I wrote) abstracted away reverse geocoding and correctly caching and manipulating results from geocoder APIs. * Medusa abstracted away ranking and filtering listings by date/amenities/etc. * Micasa abstracted away basic CRUD operations on individual listings. Airbnb intentionally used codenames for services rather than calling them by what they did, which I wasn't a huge fan of; I forget most of the codenames now (and hardly anyone knew all of the codenames offhand even at the time). Some other example without the cute names were the pricing recommendation service (used to recommend prices to new hosts), the calendar management service, the experience CRUD service, etc.
  - closeparen 6y ago
    Without being too specific, the gateways I have in the codebases in front of me: Data: - Canonical data repositories for each of: order, transaction, customer, feedback, geography, privacy opt-in/out. - Hadoop SQL facade. - Support contact analytical service for a particular type of issue. Business operations: - Core order fulfillment service (orders populate into repository when complete). - Support contact operational services: ingestion, routing, and auto-resolution. Product infrastructure: - Machine learning model serving engine. - Map drawing service. - Translations service. - Dynamic configuration service. - A/B test service. - Server-Sent-Events hub service for our mobile apps. - Facades for each of: voice, SMS, mobile native pushes, email. - Marketing campaign service (dumber but more convenient facade to the previous). Basic infrastructure: - Column store. - Secrets service. - Cache service. - Blob storage service. - Workflow engine. - Timer service (technically a subset of a workflow engine). - Message queue. Each one of these is backed by a team of 2-10 people. I have been inside 2 or 3 of them to fix bugs that affected me.
brundolf 6y ago
It depends on how you define "typical". I'd bet it's a narrow minority of companies, but most of those companies are very large, so then what percentage of all engineers are working at companies like that? Maybe not a majority, but a significant number
thirdlamp 6y ago
While we're not in the hundreds (and I don't know our loc), what I've seen happen is that on top of having a large number of services you end up with "meta-repos" that contain infrastructure-as-code or you end up having a repo from which you build the company wide Jenkins image with its configuration and so on.
akiselev 6y ago
It's pretty common. There's the extreme end like Uber (>3000 microservices) and countless cargo culting startups (I see a few dozen microservices for a team of half a dozen or fewer engineers in the wild frequently, which is big enough to desire cross-repo refactoring) but the 800 pound gorillas like Microsoft, Amazon, Apple, Facebook, etc. are at that scale by necessity. I doubt however that anyone is doing any sort of refactoring that touches a significant fraction of those and if they do, it's an organized manual team effort. Microservices didn't become popular because the code could naturally be refactored that way, but because they enforced an interface boundary that worked better for mid to large sized organizations with multiple teams of engineers and levels of management where the communication overhead is much higher.
rkalla 6y ago
In corp this is certainly true - 100s and 100s of services - but YOUR work is scoped to one domain and I wouldn't see you needing to check out the world. Each team has developed it's own way to work with it's subset of services - I doubt every team across every org could figure out how to run everyone elses domains to even make checking out the world a useful exercise. As a long term corporate exercise, maybe that would "be awesome", but not trivial to do and PAINFUL to support and stay on top of over time. We had a hard enough time keeping test suites up to date - I can't imagine trying to keep our portfolio "runnable" for everyone else.
- closeparen 6y ago
  That exercise is monorepo migration. Every target has to work with nothing more than "buck build" or "bazel test" or whatever, or you can't check it in. And anything that runs in production must be checked in. I'm not sure whether it was really worth it for us, but it did solve that problem.
tabtab 6y ago
Microservices are helpful only if the shape your teams match them. Conway's Law. Machine performance is not the main reason for them, but rather division of labor.

by aidos 6y ago

blovescoffee 6y ago

I would say this is somewhat accurate at my company (thousands of engineers). It depends on what you quality as a microservice.

closeparen 6y ago

I work in a large microservices environment. It’s very important that IDEs don’t make it easy to reach inside other services. The whole point is to create cognitive barriers at the public APIs. The pain of coordinating changes across services is a feature: it is telling you that your architecture is wrong.

aidos 6y ago
And what qualifies as a micro service in your environment? Can you give a few examples? How many people work on each micro service? Etc etc. Would love to hear more details so I can build a better mental picture.
- reissbaker 6y ago
  I'm not the OP, but I can give you some examples from when I worked at Airbnb a few years ago: * Each team (let's call it eight engineers per team on average) typically managed a few services. * Services were typically ~thousands of lines of code to ~tens of thousands. Rarely, there were very small services that were hundreds of lines of code. There was a giant service -- the original Rails app -- called "Monorail" that had millions of lines of code, but was actively being dismantled. My understanding is that there is now a code freeze on Monorail, and the only changes allowed are changes that remove code from it and move it to a smaller service (either an existing one or a new one). * Services communicated with each other over HTTP or Kafka, typically using Thrift, although also sometimes using JSON. Airbnb had an in-house "service mesh" (developed before the term service mesh existed) that abstracted away individual machines and was capable of operating at the HTTP or TCP layer. A few classic examples of mostly-single-responsibility services: * Prometheus (which I wrote) abstracted away reverse geocoding and correctly caching and manipulating results from geocoder APIs. * Medusa abstracted away ranking and filtering listings by date/amenities/etc. * Micasa abstracted away basic CRUD operations on individual listings. Airbnb intentionally used codenames for services rather than calling them by what they did, which I wasn't a huge fan of; I forget most of the codenames now (and hardly anyone knew all of the codenames offhand even at the time). Some other example without the cute names were the pricing recommendation service (used to recommend prices to new hosts), the calendar management service, the experience CRUD service, etc.
- closeparen 6y ago
  Without being too specific, the gateways I have in the codebases in front of me: Data: - Canonical data repositories for each of: order, transaction, customer, feedback, geography, privacy opt-in/out. - Hadoop SQL facade. - Support contact analytical service for a particular type of issue. Business operations: - Core order fulfillment service (orders populate into repository when complete). - Support contact operational services: ingestion, routing, and auto-resolution. Product infrastructure: - Machine learning model serving engine. - Map drawing service. - Translations service. - Dynamic configuration service. - A/B test service. - Server-Sent-Events hub service for our mobile apps. - Facades for each of: voice, SMS, mobile native pushes, email. - Marketing campaign service (dumber but more convenient facade to the previous). Basic infrastructure: - Column store. - Secrets service. - Cache service. - Blob storage service. - Workflow engine. - Timer service (technically a subset of a workflow engine). - Message queue. Each one of these is backed by a team of 2-10 people. I have been inside 2 or 3 of them to fix bugs that affected me.

brundolf 6y ago

It depends on how you define "typical". I'd bet it's a narrow minority of companies, but most of those companies are very large, so then what percentage of all engineers are working at companies like that? Maybe not a majority, but a significant number

thirdlamp 6y ago

While we're not in the hundreds (and I don't know our loc), what I've seen happen is that on top of having a large number of services you end up with "meta-repos" that contain infrastructure-as-code or you end up having a repo from which you build the company wide Jenkins image with its configuration and so on.

akiselev 6y ago

It's pretty common. There's the extreme end like Uber (>3000 microservices) and countless cargo culting startups (I see a few dozen microservices for a team of half a dozen or fewer engineers in the wild frequently, which is big enough to desire cross-repo refactoring) but the 800 pound gorillas like Microsoft, Amazon, Apple, Facebook, etc. are at that scale by necessity. I doubt however that anyone is doing any sort of refactoring that touches a significant fraction of those and if they do, it's an organized manual team effort. Microservices didn't become popular because the code could naturally be refactored that way, but because they enforced an interface boundary that worked better for mid to large sized organizations with multiple teams of engineers and levels of management where the communication overhead is much higher.

rkalla 6y ago

In corp this is certainly true - 100s and 100s of services - but YOUR work is scoped to one domain and I wouldn't see you needing to check out the world. Each team has developed it's own way to work with it's subset of services - I doubt every team across every org could figure out how to run everyone elses domains to even make checking out the world a useful exercise. As a long term corporate exercise, maybe that would "be awesome", but not trivial to do and PAINFUL to support and stay on top of over time. We had a hard enough time keeping test suites up to date - I can't imagine trying to keep our portfolio "runnable" for everyone else.

closeparen 6y ago
That exercise is monorepo migration. Every target has to work with nothing more than "buck build" or "bazel test" or whatever, or you can't check it in. And anything that runs in production must be checked in. I'm not sure whether it was really worth it for us, but it did solve that problem.

tabtab 6y ago

Microservices are helpful only if the shape your teams match them. Conway's Law. Machine performance is not the main reason for them, but rather division of labor.