8 ms·

I don't have much ideology behind going with microservices vs. monolith, but what we've done on some recent projects is organize our code into modules that only

I don't have much ideology behind going with microservices vs. monolith, but what we've done on some recent projects is organize our code into modules that only communicate with each other through a narrow and well defined boundary layer. If we need to split a module out into a separate service, then it isn't nearly as much work to split it out later. One of the practical issues we've had with microservices that need to interact with each other in real time is ensuring a consistent state across systems. For example, let's say I need to change the status of an object and afterwards, call a separate service to change state there as well. What happens if the call fails in some way? You can't just run all of this inside a single database transaction anymore. Now you have to design your code to deal with several potential failure points and edge cases, which adds complexity. The other consideration is all calls to a service should be idempotent if possible. It makes coding from the client side a lot easier if you can just fire off a call multiple times (in case of local or remote failure) and not have to worry about state. Just some of my thoughts, since this stuff has been on my plate recently.

SanderMak 10y ago
Great observation, and exactly what I meant with the modular alternative in this article. When you can 'get away' with such an architecture, it makes a lot of things simpler.
- aaronblohowiak 10y ago
  If you have transactions that have to span multiple modules, I'd argue they are too commingled
pm90 10y ago
One solution is to use a distributed key value store (like etcd) to coordinate tasks. Kubernetes does that (AFAIK); you should definitely check that out.
brandur 10y ago
Transactions certainly make it easier to maintain global consistency, but one possible contradiction with the above is that if your modules are sharing transactions, then your boundaries are no longer narrow and well defined. By definition, your entire database and all its internal workings are now part of the interface. This is one problem that I've observed with all the monoliths I've worked on. Because modules are colocated and sharing a database is easy, eventually somebody will do it (even if it wasn't originally intended), and you get lots of ostensibly modular code intertwined with other modules in non-obvious and subtly problematic ways.
- adrianratnapala 10y ago
  I agree about this danger. But then again, nests of services can also grow tangled dependencies, now in the form of RPC calls. It's a general problem in software: adding a dependency without cleaning up (or even becoming aware of) it's effects on the dep-graph is often the quickest way to solve today's problem. You then pay for it over the rest of the life of the project.
  - threeseed 10y ago
    Sure but they are tangled at the macro level. What happens with monoliths is they tangle at all levels. How many times have you seen a giant "Utils" module which initially contained stateless StringUtils and similar classes then devolved into a dumping ground for stateful business logic. Or especially with JVM applications how many times do library dependencies for one of part of the codebase end up causing issues with another. That's a big bonus with microservices i.e. being able to manage third-party dependencies better.
- platz 10y ago
  For a process that is inherently sequentially dependent on previous results, how is this not a transaction other than declaring it not so or 'dropping the outcome all over the floor' if there's a problem? It's kind of like saying, "You're not allowed to have this problem, you be better off if you had some other problem like the one I have here" Unless you're just saying to "grow the boundary" until all parts of the sequentially dependent process is inside the boundary? (this may be tricky to deal with the more external systems there are that cannot be "internalized") * I am not saying you need distributed transactions - just that some processes cannot easily be encapsulated as "atomic" operations.
lobster_johnson 10y ago
We've used microservices for around 6-7 years now. One thing we realized quite early was that letting each microservice store state "silos" independently was a bad idea. You run into the synchronization issue you describe. Instead, we've moved the state to a central, distributed store that everyone talks to. This allows you to do atomic transactions. Our store also handles fine-grained permissions, so your auth token decides what you're allowed to read and write. One non-obvious consequence is that some microservices now can be eliminated entirely, because their API was previously entirely about CRUD. Or they can be reduced to a mere policy callback -- for example, let's say the app is a comment system that allows editing your comment, but only within 5 minutes. ACLs cannot express this, so to accomplish this we have the store invoke a callback to the "owner" microservice, which can then accept or reject the change. Another consequence is that by turning the data store into a first-class service, many APIs can be expressed as data, similar to the command pattern. For example, imagine a job system. Clients request work to be done by creating jobs. This would previously be done by POSTing a job to something like /api/jobs. Instead, in the new scheme a client just creates a job in the data store. Then the job system simply watches the store for new job objects. Of course, this way of doing things comes with its own challenges. For example, how do you query the data, and how do you enforce schemas? We solved some of these things in a rather ad hoc way that we were not entirely happy with. For example, we didn't have joins, or a schema language. So about a year ago we went back to the drawing board and started building our next-generation data store, which builds in and codifies a bunch of the patterns we have figured out while using our previous store. It has schemas (optional/gradual typing), joins, permissions, changefeeds and lots of other goodies. It's looking extremely promising, and already forms the foundation of a commercial SaaS product. This new store will be open source. Please feel free to drop me an email if you're interested in being notified when it's generally available.
- sheeshkebab 10y ago
  and how that central 'data store service' is different than a single 'database service' (rdbms or nosql - CRUD) that all microservices connect to and run there select/insert/update/delete/crud ops? Other than api - rest vs whatever binary rpc protocol, it sounds very much like a standard database...
vojant 10y ago
That's a good point, transactions are hard in the micro-service world. In my experience usually It's possible to re-design architecture to encapsulate transaction inside one micro-service. If you have transaction across multiple micro-service are they are really decoupled? Similar problem is with doing asynchronous requests e.g. using RabbitMQ it is possible only with well designed boundaries as it's hard to control state of request if you do everything asynchronously. Anyway micro-services is not perfect solution for everything, but even despite these problems I love working with them!
adrianratnapala 10y ago
Hmm, my ideology doesn't say whether microservices are better than "monoliths". But it does roll its eyes when it sees people mistake the encapsulating things into modules for some particular technology helping you do that. I mean when OOP was new people talked as if (a) no one had been trying to seperate out modules before OOP, and (b) the class was the natural boundary between modules. Both are false. BTW: what does that article mean about that [in] "Java 9 a native module system is added..."; presumably this is something distinct from the package system it always had. What are the differences?
- empthought 10y ago
  Java packages are namespaces for code. Java modules are deployable bundles of code. The units of deployment (JAR, WAR, EAR files) pre-Java 9 do not have a consistent model of specifying dependencies and exports needed at runtime. In other word, packages help the compiler, but give no information to the runtime or deployment as far as how, when, and from where to deploy or load code.
- SanderMak 10y ago
  Author here. Java 9 adds a new module system where module descriptors are introduced to explicitly demarcate the public API of a module, and to express its dependencies on other modules. Example of a module descriptor: module mymodule { exports mymodule.pkga; exports mymodule.pkgb; requires someothermodule; } What happens is that every package except the ones exported are accessible to other modules. Non-exported packages are encapsulated, not even reflection can break through that barrier. The requires statements are used by the Java compiler and runtime to verify the current configuration of modules resolves correctly. Obviously there's lots of more detail to go into. Of course I recommend you check out my upcoming book (early release available) for that: http://shop.oreilly.com/product/0636920049494.do http://shop.oreilly.com/product/0636920049494.do In short, Java makes a great step forward wrt. modularity. When regular JARs transition to modular JARs (adding a module descriptor), many more checks and balances are in place than are currently possible with the classpath.
  - gravity_123 10y ago
teacpde 10y ago
Idempotency would be nice, but it is often impossible to have at all layers. Eventually at some point, you deal with stateful microservices and distributed transactions. Depending on how long the transactions take, either two-phase commits or compensation transactions are needed to rollback or restore states when failures happen. And that is not trivial to implement and complicates your system further. Stable and well-defined interfaces between microservices are another luxury hard to have in reality, especially when business and application logics constantly evolve. More often than not, it's inevitable to juggle multiple services to fulfill the need, which takes much more time, effort and risk than monolith.
- dastbe 10y ago
  While there are plenty of stateful services out there (somebody has to store your data or spin up your VM, after all), I think you'd be surprised at how few if any of them require distributed transactions. a lot of problems that appear to require distributed transactions can be solved by more efficient routing as well as more thoughtful approaches to how you approach state. I would also say its easier to define good, growable, and sustainable APIs when you really think about what the primitives of your service are. Try and avoid baking a lot of opinion into your APIs, which lets your consumers own most of their business and application logic. If you do have a need to embed new business logic into your APIs, think about how you can preserve the current default as well as provide extension points instead of one-off changes.

SanderMak 10y ago

Great observation, and exactly what I meant with the modular alternative in this article. When you can 'get away' with such an architecture, it makes a lot of things simpler.

aaronblohowiak 10y ago
If you have transactions that have to span multiple modules, I'd argue they are too commingled

pm90 10y ago

One solution is to use a distributed key value store (like etcd) to coordinate tasks. Kubernetes does that (AFAIK); you should definitely check that out.

brandur 10y ago

Transactions certainly make it easier to maintain global consistency, but one possible contradiction with the above is that if your modules are sharing transactions, then your boundaries are no longer narrow and well defined. By definition, your entire database and all its internal workings are now part of the interface. This is one problem that I've observed with all the monoliths I've worked on. Because modules are colocated and sharing a database is easy, eventually somebody will do it (even if it wasn't originally intended), and you get lots of ostensibly modular code intertwined with other modules in non-obvious and subtly problematic ways.

adrianratnapala 10y ago
I agree about this danger. But then again, nests of services can also grow tangled dependencies, now in the form of RPC calls. It's a general problem in software: adding a dependency without cleaning up (or even becoming aware of) it's effects on the dep-graph is often the quickest way to solve today's problem. You then pay for it over the rest of the life of the project.
- threeseed 10y ago
  Sure but they are tangled at the macro level. What happens with monoliths is they tangle at all levels. How many times have you seen a giant "Utils" module which initially contained stateless StringUtils and similar classes then devolved into a dumping ground for stateful business logic. Or especially with JVM applications how many times do library dependencies for one of part of the codebase end up causing issues with another. That's a big bonus with microservices i.e. being able to manage third-party dependencies better.
platz 10y ago
For a process that is inherently sequentially dependent on previous results, how is this not a transaction other than declaring it not so or 'dropping the outcome all over the floor' if there's a problem? It's kind of like saying, "You're not allowed to have this problem, you be better off if you had some other problem like the one I have here" Unless you're just saying to "grow the boundary" until all parts of the sequentially dependent process is inside the boundary? (this may be tricky to deal with the more external systems there are that cannot be "internalized") * I am not saying you need distributed transactions - just that some processes cannot easily be encapsulated as "atomic" operations.

lobster_johnson 10y ago

We've used microservices for around 6-7 years now. One thing we realized quite early was that letting each microservice store state "silos" independently was a bad idea. You run into the synchronization issue you describe. Instead, we've moved the state to a central, distributed store that everyone talks to. This allows you to do atomic transactions. Our store also handles fine-grained permissions, so your auth token decides what you're allowed to read and write. One non-obvious consequence is that some microservices now can be eliminated entirely, because their API was previously entirely about CRUD. Or they can be reduced to a mere policy callback -- for example, let's say the app is a comment system that allows editing your comment, but only within 5 minutes. ACLs cannot express this, so to accomplish this we have the store invoke a callback to the "owner" microservice, which can then accept or reject the change. Another consequence is that by turning the data store into a first-class service, many APIs can be expressed as data, similar to the command pattern. For example, imagine a job system. Clients request work to be done by creating jobs. This would previously be done by POSTing a job to something like /api/jobs. Instead, in the new scheme a client just creates a job in the data store. Then the job system simply watches the store for new job objects. Of course, this way of doing things comes with its own challenges. For example, how do you query the data, and how do you enforce schemas? We solved some of these things in a rather ad hoc way that we were not entirely happy with. For example, we didn't have joins, or a schema language. So about a year ago we went back to the drawing board and started building our next-generation data store, which builds in and codifies a bunch of the patterns we have figured out while using our previous store. It has schemas (optional/gradual typing), joins, permissions, changefeeds and lots of other goodies. It's looking extremely promising, and already forms the foundation of a commercial SaaS product. This new store will be open source. Please feel free to drop me an email if you're interested in being notified when it's generally available.

sheeshkebab 10y ago
and how that central 'data store service' is different than a single 'database service' (rdbms or nosql - CRUD) that all microservices connect to and run there select/insert/update/delete/crud ops? Other than api - rest vs whatever binary rpc protocol, it sounds very much like a standard database...

vojant 10y ago

That's a good point, transactions are hard in the micro-service world. In my experience usually It's possible to re-design architecture to encapsulate transaction inside one micro-service. If you have transaction across multiple micro-service are they are really decoupled? Similar problem is with doing asynchronous requests e.g. using RabbitMQ it is possible only with well designed boundaries as it's hard to control state of request if you do everything asynchronously. Anyway micro-services is not perfect solution for everything, but even despite these problems I love working with them!

adrianratnapala 10y ago

Hmm, my ideology doesn't say whether microservices are better than "monoliths". But it does roll its eyes when it sees people mistake the encapsulating things into modules for some particular technology helping you do that. I mean when OOP was new people talked as if (a) no one had been trying to seperate out modules before OOP, and (b) the class was the natural boundary between modules. Both are false. BTW: what does that article mean about that [in] "Java 9 a native module system is added..."; presumably this is something distinct from the package system it always had. What are the differences?

empthought 10y ago
Java packages are namespaces for code. Java modules are deployable bundles of code. The units of deployment (JAR, WAR, EAR files) pre-Java 9 do not have a consistent model of specifying dependencies and exports needed at runtime. In other word, packages help the compiler, but give no information to the runtime or deployment as far as how, when, and from where to deploy or load code.
SanderMak 10y ago
Author here. Java 9 adds a new module system where module descriptors are introduced to explicitly demarcate the public API of a module, and to express its dependencies on other modules. Example of a module descriptor: module mymodule { exports mymodule.pkga; exports mymodule.pkgb; requires someothermodule; } What happens is that every package except the ones exported are accessible to other modules. Non-exported packages are encapsulated, not even reflection can break through that barrier. The requires statements are used by the Java compiler and runtime to verify the current configuration of modules resolves correctly. Obviously there's lots of more detail to go into. Of course I recommend you check out my upcoming book (early release available) for that: http://shop.oreilly.com/product/0636920049494.do http://shop.oreilly.com/product/0636920049494.do In short, Java makes a great step forward wrt. modularity. When regular JARs transition to modular JARs (adding a module descriptor), many more checks and balances are in place than are currently possible with the classpath.
- gravity_123 10y ago

teacpde 10y ago

Idempotency would be nice, but it is often impossible to have at all layers. Eventually at some point, you deal with stateful microservices and distributed transactions. Depending on how long the transactions take, either two-phase commits or compensation transactions are needed to rollback or restore states when failures happen. And that is not trivial to implement and complicates your system further. Stable and well-defined interfaces between microservices are another luxury hard to have in reality, especially when business and application logics constantly evolve. More often than not, it's inevitable to juggle multiple services to fulfill the need, which takes much more time, effort and risk than monolith.

dastbe 10y ago
While there are plenty of stateful services out there (somebody has to store your data or spin up your VM, after all), I think you'd be surprised at how few if any of them require distributed transactions. a lot of problems that appear to require distributed transactions can be solved by more efficient routing as well as more thoughtful approaches to how you approach state. I would also say its easier to define good, growable, and sustainable APIs when you really think about what the primitives of your service are. Try and avoid baking a lot of opinion into your APIs, which lets your consumers own most of their business and application logic. If you do have a need to embed new business logic into your APIs, think about how you can preserve the current default as well as provide extension points instead of one-off changes.