5 ms·

> Shared data structures? Common library? Services have an API. If you are sharing data structures and a common "core" library across services, yes, you are off

> Shared data structures? Common library? Services have an API. If you are sharing data structures and a common "core" library across services, yes, you are officially doing it wrong. Maybe I've been doing this wrong all my life then. Imagine I want to fetch user data by id from Galactus, the user data service. I know what shape of data I want from it, and Galactus knows what shape of data it provides. Should both my service and Galactus have full, individual copies of this data structure? Alternatively, if Galactus has a published schema that I refer to, isn't that a shared data structure? By extracting the user data shape to a common repository (library, header, schema, whatever), not only we deduplicate code, but we also centralize documentation. Maybe the `address` field is cached and frequently outdated, and someone from Galactus noted this down along with where to find the golden source. Once you factor in other sources of redundancies (username regex, recommended logging, business-specific snippets), that naturally becomes a common library. I get that it sacrifices some service independence, but is this piece of independence always so worth it, that common libraries are "officially doing it wrong"?

jedberg 3y ago
The ideal way this works is the Galactus team publishes a library that consumes their API. They build their service, you then use their library to access it. But the key here is that their own library still goes through their API, not directly to their database. So that if you don't want to use their library you don't have to.
- BoppreH 3y ago
  Ah, that's exactly what I've been doing, and unconsciously assumed here. I can attest it works very well, hence my confusion.
- zten 3y ago
  The way to do this wrong (by forcing the use of the "blessed" client or common platform dependencies, instead of leaving it open to the consumer as you suggest) is discussed in this talk: https://www.microservices.com/talks/dont-build-a-distributed-monolith/ https://www.microservices.com/talks/dont-build-a-distributed...
- cogman10 3y ago
  The thing to watch out for with the client is that the dependencies of the client aren't shared with the server dependencies. Client libraries should be minimal and optional when working with an API. The failing pattern we've seen at my company is a client library, a server library, and a shared "model" library. The issue that has happened time and time again is the model library ends up including classes and dependencies that either the server or client needs. This in turn leads to a frustrating experience of dependency wrangling when someone decides to throw in lombok, or kafka, or JavaFx into the model library for that 1 sweat `Pair` class the server is using internally. Or worse, they depend on a different model library because they need that one sweat enum. (Which in turn depends on other model classes, or server classes, or whatever). If you do a client library, it should depend on practically nothing. The transport library (http client), authentication, and the data marshalling library. Nothing more. It should also be viewed as an example of how to work with your API and not the only way to work with the API. (Don't reinvent graphql... please!)
- preseinger 3y ago
  the whole point of an API is to define an interface in terms of a runtime protocol, invariant to any specific language, code, library, SDK, etc. in fact the galactus team should actively _not_ publish a library for their API doing so introduces a source-level coupling between consumers and producer, which the API was (presumably) created specifically to avoid
  - jedberg 3y ago
    It's basically a reference implementation. Their library should not do any work other than marshal data. Kind of like how Amazon produces a command line tool to interact with their API, but you don't have to use it. It's just a lot easier.
    - preseinger 3y ago
      the problem is that when an producer publishes a library like this, it's very tempting for them to assume that all access will go thru that library, and maybe even that all access will go thru the most recent version of that library it takes an enormous amount of discipline to ensure that this kind of software is treated as "a reference implementation" rather than the canonical path if an internal service API is too difficult to use without an API library, then this is a pretty good signal that something is wrong and needs to be fixed, imo
cogman10 3y ago
> Should both my service and Galactus have full, individual copies of this data structure? Yes! Maintaining that structure should be painful. Why? Because it's an exposed public API and there are ramifications to updating it that need to be thought out. Further, it can be super easy to pollute that structure with information only relevant to the server or client. Using something like JSON schema, openapi, or grpc/protobufs can make generating that second structure easy, but sharing it is something that has caused my org tremendous headaches.
- mattmanser 3y ago
  Ok, so this answer, and the other answers in this comment thread thoroughly convince me you're all batshit insane. How can you get any work done like this? Are you all working at massive corps that develop at a glacial pace? Sounds like you all spend your time shuffling papers rather than doing anything meaningful.
  - beebmam 3y ago
    Do you want a serious answer, or are you just here to insult people? If you edit your comment to remove the insults, I'd be glad to share my thoughts.
    - mattmanser 3y ago
      Look, I've never seen microservices done well. I'm negative about it because people are implementing this architecture in applications that as far as I can tell should never even consider it. Maybe they're just doing it badly. But as far as far as I can tell it's a pretty awful architecture pattern. To implement microservices, one takes what is a service in a normal application, a handful of code files in a normal application. Maybe some model files, a service, validators and a repo file. A slice of an application. One creates a new project file, build files, etc. Maybe a new repo, maybe not. You then wrap that simple service in a bunch of boilerplate plumbing code so it can actually work on its own. So, basically, a ton of extra code, right off the bat. Each time. Then to do it right according to this thread, you duplicate the definition files between your services, multiple times, add JSON schema files that you didn't have to maintain before, and, someone else has mentioned, create an extra library on top of all this so your colleagues can implement it as if it were just a normal method call. Even more code! And that's your microservice. A lot of extra work. Busy work as far as I can see, no benefits. Just to do exactly what it used to do. But, worse still, it has huge drawbacks, including: 1. Very slow "method" calls. Normal method calls are obviously orders of magnitude faster than whatever you're doing. L1 Cache is always going to be massively faster. 2. Poor debugging 3. Complicated devops requirement 4. Hidden complexity in the interaction between services that is impossible to see I just don't get it. Never have. I tried to play along, but I personally think the emperor has no clothes. If a client is going to insist on microservices, so be it, but it's a massive waste of time and money in my opinion.
Misdicorl 3y ago
Another answer: > Imagine I want to fetch user data by id from Galactus, the user data service. I know what shape of data I want from it, and Galactus knows what shape of data it provides. Should both my service and Galactus have full, individual copies of this data structure? No. Galactus should have the full shape of the "user object" but should never put that on the wire or expose it. The wire API should expose sensible things like "what is the billing and shipping address(es) of this user id". Galactus is responsible for maintaining the mapping between the "user object" and whatever the relevant return format is for those data. Edit: This allows you to run API migrations and data migrations independently. Which is crucial for any service that will last longer than 1 year.
- mjr00 3y ago
  Agreed with all of this. I consider the API structure (which might be defined by protobuf/Avro/Thrift/etc) to be separate from anything that is used by either the server or client services. The API structure is the only thing that should be shared. When you start out, it will feel very repetitive because your client object structure matches your API structure matches your server object structure. You might even think "DRY!" and want to combine these, but IMO, resist the temptation. You will eventually have a case where these need to evolve independently. For the Galactus case, imagine that for compliance (or whatever) reason, you need to start soft-deleting users, and you do that via the simple case of an `IsDeleted` flag on the User table. You'll need that flag on the User object on the Galactus side, but you don't want to add this to the API and expose it to clients. > Edit: This allows you to run API migrations and data migrations independently. Which is crucial for any service that will last longer than 1 year. Totally agreed here too.
  - bcrosby95 3y ago
    Is it normal for people to just splat their database across the wire? Because that seems pretty terrible. We wrap every API response in a DTO and are deliberate about picking and choosing which fields we want to expose. Anything else sounds like a recipe for bad times.
    - 3y ago
codethief 3y ago
> Imagine I want to fetch user data by id from Galactus, the user data service. Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
pjot 3y ago
Well yeah, but Omega Star still doesn’t support fucking ISO timestamps like they said they would over a month ago. And so Galactus wont be able to find the Birthday Boy provider which mean WingMan wont know how to talk to anybody…
- peteradio 3y ago
  I'm fucking blocked and I'll never find love.
  - pjot 3y ago
    You’ll never know my pain - or Galctus’s pain - you stupid, pathetic little product manager!
simplotek 3y ago
> Should both my service and Galactus have full, individual copies of this data structure? Yes. I recommend you read "Domain-driven design" by Eric Evans, specially the part explaining the concept of a bounded context. > Alternatively, if Galactus has a published schema that I refer to, isn't that a shared data structure? No, that's the interface. > not only we deduplicate code You don't. You just needlessly add a constraint for no reason at all, and in the process make your life a mess.

jedberg 3y ago

The ideal way this works is the Galactus team publishes a library that consumes their API. They build their service, you then use their library to access it. But the key here is that their own library still goes through their API, not directly to their database. So that if you don't want to use their library you don't have to.

BoppreH 3y ago
Ah, that's exactly what I've been doing, and unconsciously assumed here. I can attest it works very well, hence my confusion.
zten 3y ago
The way to do this wrong (by forcing the use of the "blessed" client or common platform dependencies, instead of leaving it open to the consumer as you suggest) is discussed in this talk: https://www.microservices.com/talks/dont-build-a-distributed-monolith/ https://www.microservices.com/talks/dont-build-a-distributed...
cogman10 3y ago
The thing to watch out for with the client is that the dependencies of the client aren't shared with the server dependencies. Client libraries should be minimal and optional when working with an API. The failing pattern we've seen at my company is a client library, a server library, and a shared "model" library. The issue that has happened time and time again is the model library ends up including classes and dependencies that either the server or client needs. This in turn leads to a frustrating experience of dependency wrangling when someone decides to throw in lombok, or kafka, or JavaFx into the model library for that 1 sweat `Pair` class the server is using internally. Or worse, they depend on a different model library because they need that one sweat enum. (Which in turn depends on other model classes, or server classes, or whatever). If you do a client library, it should depend on practically nothing. The transport library (http client), authentication, and the data marshalling library. Nothing more. It should also be viewed as an example of how to work with your API and not the only way to work with the API. (Don't reinvent graphql... please!)
preseinger 3y ago
the whole point of an API is to define an interface in terms of a runtime protocol, invariant to any specific language, code, library, SDK, etc. in fact the galactus team should actively _not_ publish a library for their API doing so introduces a source-level coupling between consumers and producer, which the API was (presumably) created specifically to avoid
- jedberg 3y ago
  It's basically a reference implementation. Their library should not do any work other than marshal data. Kind of like how Amazon produces a command line tool to interact with their API, but you don't have to use it. It's just a lot easier.
  - preseinger 3y ago
    the problem is that when an producer publishes a library like this, it's very tempting for them to assume that all access will go thru that library, and maybe even that all access will go thru the most recent version of that library it takes an enormous amount of discipline to ensure that this kind of software is treated as "a reference implementation" rather than the canonical path if an internal service API is too difficult to use without an API library, then this is a pretty good signal that something is wrong and needs to be fixed, imo

cogman10 3y ago

> Should both my service and Galactus have full, individual copies of this data structure? Yes! Maintaining that structure should be painful. Why? Because it's an exposed public API and there are ramifications to updating it that need to be thought out. Further, it can be super easy to pollute that structure with information only relevant to the server or client. Using something like JSON schema, openapi, or grpc/protobufs can make generating that second structure easy, but sharing it is something that has caused my org tremendous headaches.

mattmanser 3y ago
Ok, so this answer, and the other answers in this comment thread thoroughly convince me you're all batshit insane. How can you get any work done like this? Are you all working at massive corps that develop at a glacial pace? Sounds like you all spend your time shuffling papers rather than doing anything meaningful.
- beebmam 3y ago
  Do you want a serious answer, or are you just here to insult people? If you edit your comment to remove the insults, I'd be glad to share my thoughts.
  - mattmanser 3y ago
    Look, I've never seen microservices done well. I'm negative about it because people are implementing this architecture in applications that as far as I can tell should never even consider it. Maybe they're just doing it badly. But as far as far as I can tell it's a pretty awful architecture pattern. To implement microservices, one takes what is a service in a normal application, a handful of code files in a normal application. Maybe some model files, a service, validators and a repo file. A slice of an application. One creates a new project file, build files, etc. Maybe a new repo, maybe not. You then wrap that simple service in a bunch of boilerplate plumbing code so it can actually work on its own. So, basically, a ton of extra code, right off the bat. Each time. Then to do it right according to this thread, you duplicate the definition files between your services, multiple times, add JSON schema files that you didn't have to maintain before, and, someone else has mentioned, create an extra library on top of all this so your colleagues can implement it as if it were just a normal method call. Even more code! And that's your microservice. A lot of extra work. Busy work as far as I can see, no benefits. Just to do exactly what it used to do. But, worse still, it has huge drawbacks, including: 1. Very slow "method" calls. Normal method calls are obviously orders of magnitude faster than whatever you're doing. L1 Cache is always going to be massively faster. 2. Poor debugging 3. Complicated devops requirement 4. Hidden complexity in the interaction between services that is impossible to see I just don't get it. Never have. I tried to play along, but I personally think the emperor has no clothes. If a client is going to insist on microservices, so be it, but it's a massive waste of time and money in my opinion.

Misdicorl 3y ago

Another answer: > Imagine I want to fetch user data by id from Galactus, the user data service. I know what shape of data I want from it, and Galactus knows what shape of data it provides. Should both my service and Galactus have full, individual copies of this data structure? No. Galactus should have the full shape of the "user object" but should never put that on the wire or expose it. The wire API should expose sensible things like "what is the billing and shipping address(es) of this user id". Galactus is responsible for maintaining the mapping between the "user object" and whatever the relevant return format is for those data. Edit: This allows you to run API migrations and data migrations independently. Which is crucial for any service that will last longer than 1 year.

mjr00 3y ago
Agreed with all of this. I consider the API structure (which might be defined by protobuf/Avro/Thrift/etc) to be separate from anything that is used by either the server or client services. The API structure is the only thing that should be shared. When you start out, it will feel very repetitive because your client object structure matches your API structure matches your server object structure. You might even think "DRY!" and want to combine these, but IMO, resist the temptation. You will eventually have a case where these need to evolve independently. For the Galactus case, imagine that for compliance (or whatever) reason, you need to start soft-deleting users, and you do that via the simple case of an `IsDeleted` flag on the User table. You'll need that flag on the User object on the Galactus side, but you don't want to add this to the API and expose it to clients. > Edit: This allows you to run API migrations and data migrations independently. Which is crucial for any service that will last longer than 1 year. Totally agreed here too.
- bcrosby95 3y ago
  Is it normal for people to just splat their database across the wire? Because that seems pretty terrible. We wrap every API response in a DTO and are deliberate about picking and choosing which fields we want to expose. Anything else sounds like a recipe for bad times.
  - 3y ago

codethief 3y ago

> Imagine I want to fetch user data by id from Galactus, the user data service. Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ

pjot 3y ago

Well yeah, but Omega Star still doesn’t support fucking ISO timestamps like they said they would over a month ago. And so Galactus wont be able to find the Birthday Boy provider which mean WingMan wont know how to talk to anybody…

peteradio 3y ago
I'm fucking blocked and I'll never find love.
- pjot 3y ago
  You’ll never know my pain - or Galctus’s pain - you stupid, pathetic little product manager!

simplotek 3y ago

> Should both my service and Galactus have full, individual copies of this data structure? Yes. I recommend you read "Domain-driven design" by Eric Evans, specially the part explaining the concept of a bounded context. > Alternatively, if Galactus has a published schema that I refer to, isn't that a shared data structure? No, that's the interface. > not only we deduplicate code You don't. You just needlessly add a constraint for no reason at all, and in the process make your life a mess.