13 ms·

I'm not sure what they've been left with is a monolith after all. I would say they just have a new service, which is the size of what they should have originall

I'm not sure what they've been left with is a monolith after all. I would say they just have a new service, which is the size of what they should have originally attempted before splitting. In particular, as to their original problem, the shared library seems to be the main source of pain and that isn't technically solved by a monolith, along with not following the basic rule of services "put together first, split later". I feel prematurely splitting services like that is bound to have issues unless they have 100 developers for 100 services. The claim of "1 superstar" is misleading too, this service doesn't include the logic for their API, Admin, Billing, User storage etc etc, it's still a service, one of a few that make up Segment in totality.

athenot 8y ago
Agreed. Reading about their setup and comparing with some truly large scale services I work with, I'm left with the idea that Segment's service is roughly the size of one microservice on our end. Perhaps the takeaway is don't go overboard with fragmenting services when they conceptually fulfill the same business role. And regardless of the architecture of the system, there are hard state problems to deal with in association with service availability.
- tomnipotent 8y ago
  > Segment's service is roughly the size of one microservice on our end. This gave me a good chuckle. Dunning–Kruger in full effect.
- testplzignore 8y ago
  The most telling fact is that it "took milliseconds to complete running the tests for all 140+ of our destinations". I've never worked on a single service whose tests ran that fast, given that the time spent by the overhead of the test framework and any other one-time initialization can take a few seconds just itself. It's great to have tests that run fast, but that's a bit ridiculous. Some rules of thumb I just came up with: Number of repos should not exceed number of developers. Number of tests divided by number of developers should be at least 100. Number of lines of code divided by number of repos should be at least 5000. Your tests should not run faster than the time it takes to read this sentence. A single person should not be able to memorize the entire contents of a single repo, unless that person is Rain Man.
  - mpweiher 8y ago
    > never worked on a single service whose tests ran that fast I'd say you've never had good tests. I have a test-suite for a bunch of my frameworks that dates to the mid 90s, with tests added regularly with new functionality. It currently takes 4 seconds total for 6 separate frameworks and 1000 individual tests. Which is actually a bit slower than it should be, it used to take around 1-2 seconds, so might have to dig a little to see what's up. With tests this fast, they become a fixed part of the build-process, so every build runs the tests, and a test failure is essentially treated the same as a compiler error: the project fails to build. The difference goes beyond quantitative to qualitative, and hard to communicate. Testing becomes much less of a distinct activity but simple an inextricable part of writing code. So I would posit: Your tests should not run slower than the time it takes to read this sentence.
    - sieabahlpark 8y ago
      Not all domains have libraries which can run tests that fast?
      - mpweiher 8y ago
        Sure. I'd suggest that the domains that limit testing speed are exceedingly rare. Much, much, much more common are technical issues. Either way, the idea of considering slow tests a feature was novel to me.
    - deleted 8y ago
      [deleted]
    - mekazu 8y ago
      Unit tests that don’t read or write to disk and don’t try thousands of repetitions of things should be bleeding fast, but the most useful integration tests that actually help find faults (usually with your assumptions about the associated APIs) often need interaction with your disk or database or external service and tend to take a bit more than a few seconds. I find you need both.
hinkley 8y ago
unless they have 100 developers for 100 services. That cure is worse than the disease. Every service works differently and 80% of them are just wrong, and there’s nothing you can do because Tim owns that bit.
- topicseed 8y ago
  But if you can explain to the team or the CTO why Tim is doing it wrong and how it is impacting X, Y and Z, then Tim will fix or be sent else where, no?
  - kofejnik 8y ago
    but the CTO is a nice guy who just had a management training, so instead, both you and Tim are sent to a mandatory conflict resolution class
  - hinkley 8y ago
    Tim accuses everyone else of being lazy or stupid. Tom (real guy) was too busy all the time to do anything other than the 80/20 rule. He was too busy because he didn't share. So of course he was a fixture of the company...
  - 0xEFF 8y ago
    No, not unless you’re somehow able to make the team, the CTO and Tim feel good at the same time. If you figure that part out let me know.
    - topicseed 8y ago
      But if Tim's work is so bad that you can prove it..... How can a boss dodge the proof? I guess Tim should be fired second, the boss should go first...
      - flukus 8y ago
        > How can a boss dodge the proof? This is a business decision, reality has no influence here. That's why they invented the term "Business reality".
techinformed56 8y ago
+1 I felt this article is more about how to use microservices right way vs butchering the idea. It is not right to characterize this as microservices vs monolith service. Initial version of their attempt went too far by spinning up a service for each destination. This is taking microservices to extreme which caused organizational and maintenance issue once number of destinations increased. I am surprised they did not foresee this. The final solution is also microservice architecture with a better separation of concerns/functionalities. One service for managing in bound queue of events and other service for interacting with all destinations.
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- tabtab 8y ago
  Age-old truth: "Use the right tool for the right job".
tomnipotent 8y ago
They never said the entire company was left with 1, only than 100s were condensed to "1 superstar".
matwood 8y ago
> along with not following the basic rule of services "put together first, split later". Agreed. I treat services like an amoeba. Let your monolith grow until you see the obvious split points. The first one I typically see is authentication, but YMMV. Notice I also do not say 'microservices'. I don't care about micro as much as functional grouping.
nikon 8y ago
+1 Changing one "shared library" shouldn't mean deploying 140 services immediately. They had one service to begin with forked for each destination. Of course that was a nightmare to maintain!
- Cthulhu_ 8y ago
  I can see their reasoning though; most of those services are pretty straightforward I think (common data model in -> transform -> specific outbound API data out -> convert result back to common data model). The challenge they had is that a lot of the logic in each of those services could be reused (http, data transformation, probably logging / monitoring / etc), so shared libraries and such.
kohlerm 8y ago
It looks to me that the shared library issue got solved by the monorepo approach. They could have gone the monorepo way and still have microservices. Managing a lot of repos and keeping them consistent with regards do dependencies is not easy. In reality you do not want everyone to use a different version of a dependency. You might allow deviations but ultimately you wand to minimize them. They also just might have had too many repos.
- Cthulhu_ 8y ago
  Next to a monorepo they would also need a deployment strategy allowing them to deploy multiple services (e.g. every service that was affected by the library change) simultaneously, so that after deploying they can still talk to one another. For a single service this is doable enough (start up, wait for green health, route requests to new service instance), but it increases in complexity when there's >1 service. I'm sure the process can be repeated and automated etc, but it will be more complex. Doing zero-downtime deployments for a single service is hard enough.
inetknght 8y ago
> the basic rule of services "put together first, split later" Is this rule mentioned or discussed somewhere? A quick google search links to a bunch of dating suggestions about splitting the bill. Searching for the basic rule of services "put together first, split later" reveals nothing useful.

athenot 8y ago

Agreed. Reading about their setup and comparing with some truly large scale services I work with, I'm left with the idea that Segment's service is roughly the size of one microservice on our end. Perhaps the takeaway is don't go overboard with fragmenting services when they conceptually fulfill the same business role. And regardless of the architecture of the system, there are hard state problems to deal with in association with service availability.

tomnipotent 8y ago
> Segment's service is roughly the size of one microservice on our end. This gave me a good chuckle. Dunning–Kruger in full effect.
testplzignore 8y ago
The most telling fact is that it "took milliseconds to complete running the tests for all 140+ of our destinations". I've never worked on a single service whose tests ran that fast, given that the time spent by the overhead of the test framework and any other one-time initialization can take a few seconds just itself. It's great to have tests that run fast, but that's a bit ridiculous. Some rules of thumb I just came up with: Number of repos should not exceed number of developers. Number of tests divided by number of developers should be at least 100. Number of lines of code divided by number of repos should be at least 5000. Your tests should not run faster than the time it takes to read this sentence. A single person should not be able to memorize the entire contents of a single repo, unless that person is Rain Man.
- mpweiher 8y ago
  > never worked on a single service whose tests ran that fast I'd say you've never had good tests. I have a test-suite for a bunch of my frameworks that dates to the mid 90s, with tests added regularly with new functionality. It currently takes 4 seconds total for 6 separate frameworks and 1000 individual tests. Which is actually a bit slower than it should be, it used to take around 1-2 seconds, so might have to dig a little to see what's up. With tests this fast, they become a fixed part of the build-process, so every build runs the tests, and a test failure is essentially treated the same as a compiler error: the project fails to build. The difference goes beyond quantitative to qualitative, and hard to communicate. Testing becomes much less of a distinct activity but simple an inextricable part of writing code. So I would posit: Your tests should not run slower than the time it takes to read this sentence.
  - sieabahlpark 8y ago
    Not all domains have libraries which can run tests that fast?
    - mpweiher 8y ago
      Sure. I'd suggest that the domains that limit testing speed are exceedingly rare. Much, much, much more common are technical issues. Either way, the idea of considering slow tests a feature was novel to me.
  - deleted 8y ago
    [deleted]
  - mekazu 8y ago
    Unit tests that don’t read or write to disk and don’t try thousands of repetitions of things should be bleeding fast, but the most useful integration tests that actually help find faults (usually with your assumptions about the associated APIs) often need interaction with your disk or database or external service and tend to take a bit more than a few seconds. I find you need both.

hinkley 8y ago

unless they have 100 developers for 100 services. That cure is worse than the disease. Every service works differently and 80% of them are just wrong, and there’s nothing you can do because Tim owns that bit.

topicseed 8y ago
But if you can explain to the team or the CTO why Tim is doing it wrong and how it is impacting X, Y and Z, then Tim will fix or be sent else where, no?
- kofejnik 8y ago
  but the CTO is a nice guy who just had a management training, so instead, both you and Tim are sent to a mandatory conflict resolution class
- hinkley 8y ago
  Tim accuses everyone else of being lazy or stupid. Tom (real guy) was too busy all the time to do anything other than the 80/20 rule. He was too busy because he didn't share. So of course he was a fixture of the company...
- 0xEFF 8y ago
  No, not unless you’re somehow able to make the team, the CTO and Tim feel good at the same time. If you figure that part out let me know.
  - topicseed 8y ago
    But if Tim's work is so bad that you can prove it..... How can a boss dodge the proof? I guess Tim should be fired second, the boss should go first...
    - flukus 8y ago
      > How can a boss dodge the proof? This is a business decision, reality has no influence here. That's why they invented the term "Business reality".

techinformed56 8y ago

+1 I felt this article is more about how to use microservices right way vs butchering the idea. It is not right to characterize this as microservices vs monolith service. Initial version of their attempt went too far by spinning up a service for each destination. This is taking microservices to extreme which caused organizational and maintenance issue once number of destinations increased. I am surprised they did not foresee this. The final solution is also microservice architecture with a better separation of concerns/functionalities. One service for managing in bound queue of events and other service for interacting with all destinations.

deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
tabtab 8y ago
Age-old truth: "Use the right tool for the right job".

tomnipotent 8y ago

They never said the entire company was left with 1, only than 100s were condensed to "1 superstar".

matwood 8y ago

> along with not following the basic rule of services "put together first, split later". Agreed. I treat services like an amoeba. Let your monolith grow until you see the obvious split points. The first one I typically see is authentication, but YMMV. Notice I also do not say 'microservices'. I don't care about micro as much as functional grouping.

nikon 8y ago

+1 Changing one "shared library" shouldn't mean deploying 140 services immediately. They had one service to begin with forked for each destination. Of course that was a nightmare to maintain!

Cthulhu_ 8y ago
I can see their reasoning though; most of those services are pretty straightforward I think (common data model in -> transform -> specific outbound API data out -> convert result back to common data model). The challenge they had is that a lot of the logic in each of those services could be reused (http, data transformation, probably logging / monitoring / etc), so shared libraries and such.

kohlerm 8y ago

It looks to me that the shared library issue got solved by the monorepo approach. They could have gone the monorepo way and still have microservices. Managing a lot of repos and keeping them consistent with regards do dependencies is not easy. In reality you do not want everyone to use a different version of a dependency. You might allow deviations but ultimately you wand to minimize them. They also just might have had too many repos.

Cthulhu_ 8y ago
Next to a monorepo they would also need a deployment strategy allowing them to deploy multiple services (e.g. every service that was affected by the library change) simultaneously, so that after deploying they can still talk to one another. For a single service this is doable enough (start up, wait for green health, route requests to new service instance), but it increases in complexity when there's >1 service. I'm sure the process can be repeated and automated etc, but it will be more complex. Doing zero-downtime deployments for a single service is hard enough.

inetknght 8y ago

> the basic rule of services "put together first, split later" Is this rule mentioned or discussed somewhere? A quick google search links to a bunch of dating suggestions about splitting the bill. Searching for the basic rule of services "put together first, split later" reveals nothing useful.