6 ms·

Yeah, our (mostly) ban on shared code is a major reason we end up with a lot of services; there's even service.business-day which tells you if a day is a workin

Yeah, our (mostly) ban on shared code is a major reason we end up with a lot of services; there's even service.business-day which tells you if a day is a working day

d4nt 7y ago
I’m interested in the term “distributed monolith” and why shared libraries are bad, can you elaborate? (I’m an experienced dev who’s never worked on micro services). Say a service that consumed 6 other services became a service that consumed 3 other services and 3 libraries. The interfaces were the same, the libraries are consumed via package management and have good test coverage. Why is that worse? I’m actually not surprised you have a service for handling working days. The list of UK bank holidays should ideally be stored in one place, with everything else consuming that. But not all of the 1500 services have their own data stores, do they?
- p10jkle 7y ago
  The common idea is that if you have to regularly roll out all your services, then you still basically have a monolith. So if you have a lot of shared code in libaries, there are no benefits to microservices. In our case, maybe we do have core libaries like to read from Cassandra, but you don't necessarily have to roll out everything if you improve them. Some pods could be even a year old All the services have their own keyspace in Cassandra, there is near-zero cross service data access
  - virgilp 7y ago
    That's not what makes a distributed monolith, IMO. Or to put it differently, it's not what makes a "monolith" be equal to "bad" in people minds. You want to avoid the badness, don't want to avoid the label. So, what is bad about a monolith? The complexity. Coupling of unrelated things; because it's simple to do so, because it's right there next to you and you know how it's implemented and what it performs, so you do thing A in component X knowing/relying on the fact that component Y will do the thing B, because that's how it's implemented (not because it's inherent unavoidable business logic). How do you end up with a distributed monolith? E.g. assume that microservice B will call microservice C only after you (i.e. microservice A) had a chance to call C first. Yes your code is distributed, but you don't get the benefit of logic segregation between the modules - you just get all the downsides of distributed systems. Do note that my scenario requires no shared code or library. Not sharing code doesn't save you from the distributed monolith - careful thought does.
    - jacquesm 7y ago
      Amen. Lots of micro service engineering is at the level of cargo cults and label avoidance. There are companies that do it well but relatively few of those. Whether TFA is a good or a bad example can not be told from the article contents, it would need a lot more information.
- dsirola 7y ago
  I think the more common term is distributed ball of mud
- trollied 7y ago
  It seems crazy to me that you'd have to do a HTTP(S) request to find out whether a day was a working day. A HTTP(S) call which then probably does a DB lookup over the network. Just seems like an extra layer of overhead, when you could just have a library instead. I've never worked on microservices, so I may just have a fundamental misunderstanding of how granular you're supposed to be & also don't know if this is the norm. Would be nice to hear some other opinions on this.
  - tonyedgecombe 7y ago
    I must admit I'm struggling to see what problem they are trying to solve with this. I'm not aware of a single platform where you can't modularise your code.
    - rtpg 7y ago
      Though I’m against micro services for lots of reasons, there is truth to the “library update” problem If you update the business day library, then you also need to bump the version of this lib used in your 100+ “real services”, or risk some services seeing differences on this front. Imagine a bug from different logic due to version differences between libs in a distributed system. Would be a mess to debug. If you say “oh well services should just install latest” now you have the distributed monolith problem. A new version of a library could get pushed out but have a regression bringing down half your services
      - marcosdumay 7y ago
        How do you deal with incompatible updates of the services? I fail to see how they won't have the exact same problems.
        tempay 7y ago
        You just roll back that one service and every other service picks up the change automatically. Doing the same thing with shared libraries requires redeploying everything that is dependent on the bad library (i.e. hundreds of containers instead of two or three). It's not necessarily that one model is better than the other just that using a hybrid approach often means having the issues of both with only a few (if any!) of the benefits.
viraptor 7y ago
How do you deal with passing common data between services? For example, does every service checking the holiday endpoint need to write its own access/deserialisation/mapping starting from http or some other RPC layer? If yes, how do you keep that in sync/compatible? Do you have an explosion of versions on each endpoint? This is likely trivial for the holidays endpoint, but I'm interested in the more complex ones.
- lwb 7y ago
  This is what I want to know too -- seems like you have to have at least one common library, that is, the library that lets you write a service call in one line instead of twenty lines. IME Go can be quite verbose, especially with >3 lines of error checking after every function call.
- lossolo 7y ago
  They are probably using grpc and automatically generate clients libraries.
ealexhudson 7y ago
Why (mostly) ban shared code? If you have that many microservices the deployment has to be well automated; so rolling out updated services on library update can't be difficult, and the interface to a compiled library is about as controllable as the interface to a network service. Is there a separate reason to those? Do you worry about the additional latency of additional service calls (for some trivial functions that doesn't feel terribly worthwhile)?
- cameronbrown 7y ago
  I can imagine these shared libraries being in an inconsistent state at one time or the risks of updating thousands of services at once, would be two factors.
marcinzm 7y ago
I'm curious why the investment in so many services instead of investing in improved CI/CD/build tools. For example, automatically deploying services when the libraries they depend on are updated. Monitoring to automatically check if a deployment is having issues. I mean, due to security concerns you still will need to re-deploy services every so often so having great CI/CD seems a more worthwhile investment. edit: Also, for the same reason chaos engineering, short lived certificates and periodic backup tests make sense; constantly testing out the deployment of all services makes sense to me. If you do something like deploying a service rarely then the chance of something going wrong (forgot a step, radical external changes, etc.) is decently higher.

Yeah, our (mostly) ban on shared code is a major reason we end up with a lot of services; there's even service.business-day which tells you if a day is a working day

d4nt 7y ago

I’m interested in the term “distributed monolith” and why shared libraries are bad, can you elaborate? (I’m an experienced dev who’s never worked on micro services). Say a service that consumed 6 other services became a service that consumed 3 other services and 3 libraries. The interfaces were the same, the libraries are consumed via package management and have good test coverage. Why is that worse? I’m actually not surprised you have a service for handling working days. The list of UK bank holidays should ideally be stored in one place, with everything else consuming that. But not all of the 1500 services have their own data stores, do they?

p10jkle 7y ago
The common idea is that if you have to regularly roll out all your services, then you still basically have a monolith. So if you have a lot of shared code in libaries, there are no benefits to microservices. In our case, maybe we do have core libaries like to read from Cassandra, but you don't necessarily have to roll out everything if you improve them. Some pods could be even a year old All the services have their own keyspace in Cassandra, there is near-zero cross service data access
- virgilp 7y ago
  That's not what makes a distributed monolith, IMO. Or to put it differently, it's not what makes a "monolith" be equal to "bad" in people minds. You want to avoid the badness, don't want to avoid the label. So, what is bad about a monolith? The complexity. Coupling of unrelated things; because it's simple to do so, because it's right there next to you and you know how it's implemented and what it performs, so you do thing A in component X knowing/relying on the fact that component Y will do the thing B, because that's how it's implemented (not because it's inherent unavoidable business logic). How do you end up with a distributed monolith? E.g. assume that microservice B will call microservice C only after you (i.e. microservice A) had a chance to call C first. Yes your code is distributed, but you don't get the benefit of logic segregation between the modules - you just get all the downsides of distributed systems. Do note that my scenario requires no shared code or library. Not sharing code doesn't save you from the distributed monolith - careful thought does.
  - jacquesm 7y ago
    Amen. Lots of micro service engineering is at the level of cargo cults and label avoidance. There are companies that do it well but relatively few of those. Whether TFA is a good or a bad example can not be told from the article contents, it would need a lot more information.
dsirola 7y ago
I think the more common term is distributed ball of mud
trollied 7y ago
It seems crazy to me that you'd have to do a HTTP(S) request to find out whether a day was a working day. A HTTP(S) call which then probably does a DB lookup over the network. Just seems like an extra layer of overhead, when you could just have a library instead. I've never worked on microservices, so I may just have a fundamental misunderstanding of how granular you're supposed to be & also don't know if this is the norm. Would be nice to hear some other opinions on this.
- tonyedgecombe 7y ago
  I must admit I'm struggling to see what problem they are trying to solve with this. I'm not aware of a single platform where you can't modularise your code.
  - rtpg 7y ago
    Though I’m against micro services for lots of reasons, there is truth to the “library update” problem If you update the business day library, then you also need to bump the version of this lib used in your 100+ “real services”, or risk some services seeing differences on this front. Imagine a bug from different logic due to version differences between libs in a distributed system. Would be a mess to debug. If you say “oh well services should just install latest” now you have the distributed monolith problem. A new version of a library could get pushed out but have a regression bringing down half your services
    - marcosdumay 7y ago
      How do you deal with incompatible updates of the services? I fail to see how they won't have the exact same problems.
      - tempay 7y ago
        You just roll back that one service and every other service picks up the change automatically. Doing the same thing with shared libraries requires redeploying everything that is dependent on the bad library (i.e. hundreds of containers instead of two or three). It's not necessarily that one model is better than the other just that using a hybrid approach often means having the issues of both with only a few (if any!) of the benefits.

viraptor 7y ago

How do you deal with passing common data between services? For example, does every service checking the holiday endpoint need to write its own access/deserialisation/mapping starting from http or some other RPC layer? If yes, how do you keep that in sync/compatible? Do you have an explosion of versions on each endpoint? This is likely trivial for the holidays endpoint, but I'm interested in the more complex ones.

lwb 7y ago
This is what I want to know too -- seems like you have to have at least one common library, that is, the library that lets you write a service call in one line instead of twenty lines. IME Go can be quite verbose, especially with >3 lines of error checking after every function call.
lossolo 7y ago
They are probably using grpc and automatically generate clients libraries.

ealexhudson 7y ago

Why (mostly) ban shared code? If you have that many microservices the deployment has to be well automated; so rolling out updated services on library update can't be difficult, and the interface to a compiled library is about as controllable as the interface to a network service. Is there a separate reason to those? Do you worry about the additional latency of additional service calls (for some trivial functions that doesn't feel terribly worthwhile)?

cameronbrown 7y ago
I can imagine these shared libraries being in an inconsistent state at one time or the risks of updating thousands of services at once, would be two factors.

marcinzm 7y ago

I'm curious why the investment in so many services instead of investing in improved CI/CD/build tools. For example, automatically deploying services when the libraries they depend on are updated. Monitoring to automatically check if a deployment is having issues. I mean, due to security concerns you still will need to re-deploy services every so often so having great CI/CD seems a more worthwhile investment. edit: Also, for the same reason chaos engineering, short lived certificates and periodic backup tests make sense; constantly testing out the deployment of all services makes sense to me. If you do something like deploying a service rarely then the chance of something going wrong (forgot a step, radical external changes, etc.) is decently higher.