5 ms·

From my own experience building a platform of products on top of microservices, I find that the most difficult part about this architecture (and the one that no

by sakopov 10y ago

From my own experience building a platform of products on top of microservices, I find that the most difficult part about this architecture (and the one that nobody ever talks about) is how to share data between microservices via message bus instead of direct requests. If you can get this down the rest, in my opinion, is just pure bliss and a piece of cake.

ben_jones 10y ago
Redis. Redis all the things. It can't be understated the peace of mind knowing your operations are somewhat atomic.
- rashkov 10y ago
  I think redis definitely has a place as a pubsub channel in a micro services architecture, and that in this capacity it solves the issue of sharing information among services. I'm not really seeing your point about atomic operations though. Would be curious to hear more about it
- devonkim 10y ago
  Wait, peace of mind knowing your operations are "somewhat" atomic? Am I not understanding something but isn't that something that would normally be disturbing when it comes to distributed transactions (what microservices oftentimes structure)?
chuhnk 10y ago
Event sourcing is the thing you're looking for and something that will probably be the focus of many building microservice platforms.
- notduncansmith 10y ago
  This. About 2 years ago, I started to notice the incredible usefulness of event sourcing and its applicability at various layers of the stack. From the React/Flux frontend paradigm, to the "everything is a stream" philosophy of Unix systems, to Erlang actors, to event sourcing for application backends, to databases like Datomic and CouchDB. This is why I'm so excited about Elm right now - it explicitly puts this at the core of every application, and gives you a really nice language and type system for encoding your domain logic and manipulating data. Time to write a blog post, I guess.
- Terr_ 10y ago
  I think the real network-level win comes from rigorous events, period. Event sourcing, on the other hand, is an internal implementation detail of a particular service... Which happens to get you a good stream of events as a side-effect.
pm90 10y ago
OpenStack actually does this really well, using RabbitMQ as the messaging tool. https://wiki.openstack.org/wiki/Oslo/Messaging https://wiki.openstack.org/wiki/Oslo/Messaging
devonkim 10y ago
Message buses and queues are far, far easier to manage than API endpoints across a bunch of loci of control, so this may be why it's not discussed as much as, say, how the lifecycle of REST services work. Topics and queues with different segmentation features are much more powerful and fine-grained in control compared to a service that's tied to HTTP transport than, say, a REST or SOAP API. For object serialization / marshaling, you can use a lot more stuff now all of a sudden like Protocol Buffers, Capn Proto, Thrift, Avro. The recommended advice comes down to what your bus / queue is (each one offers vastly different best practices and features to implement them) as well as how you want to scale out and up. But perhaps a general set of terminologies / jargon may be worth pursuing as a community for the problem. The question I really have is how people have managed to upgrade their message bus and to keep availability high with rollback contingencies while upgrading, say, RabbitMQ or Kafka across all your connected services. I really don't hear about that as much as how to do rolling restarts / upgrades of nodes for a service endpoint using some feedback from monitoring and metrics.
- rando289 10y ago
  > Topics and queues with different segmentation features are much more powerful and fine-grained in control compared to a service that's tied to HTTP transport than, say, a REST or SOAP API. "than" and beyond is an illogical statement. Like Y is Z than X.
  - devonkim 10y ago
    Funky parse as I wrote it out, but you're correct. Despite the problem, I think the point being made is understandable. You can replace "than" with "like."
- moonlighter 10y ago
  Can't answer about RabbitMQ or Kafka, but if you're using a managed service like AWS SQS it's a non-issue. And if you need to test or make changes to a queue, it usually just means having to create another queue in parallel while the old one still works, and switch over once ready. The guys from Shazam gave a great talk about this at the AWS:reinvent conference back in 2013: https://www.youtube.com/watch?v=k_54Jcmi4zM https://www.youtube.com/watch?v=k_54Jcmi4zM
eva1984 10y ago
You need to further define 'share data'? Usually it could be done, to connect multiple different services through a central data storage, let it be redis/cassandra/RDBMS/HBase... If what you mean by sharing data is about dependency between calls to different services, e.g. A -> B -> C..., I would go with Storm and other stream processing frameworks, cause that is why those frameworks are built...
- sakopov 10y ago
  I'm simply talking about scenarios where one API needs to talk to another to get a resource it needs to act upon in some way. This can get vary hairy very quickly especially when each microservice ends up with copies of resources owned by other microservices that it now has to update asynchronously via pub/sub any time there is a change. This was the biggest challenge. And I think any teams willing to switch to this type of architecture need to look at it from this angle because monitoring and troubleshooting such systems is quite an undertaking.
devishard 10y ago
Have you tried Erlang? It's basically designed around solving that problem. Many people use RabbitMQ to solve that problem and it's no coincidence that RabbitMQ is written in Erlang. Elixir is possibly a similar option (don't know, haven't tried it).
- greenleafjacob 10y ago
  Erlang's raison d'etre is not message passing but fault tolerance. From Joe Armstrong[1]: > The work described in this thesis is the result of a research program started in 1981 to find better ways of programming Telecom applications. These applications are large programs which despite careful testing will probably contain many errors when the program is put into service. We assume that such programs do contain errors, and investigate methods for building reliable systems despite such errors. The research has resulted in the development of a new programming language (called Erlang), together with a design methodology, and set of libraries for building robust systems (called OTP). [1]: http://ftp.nsysu.edu.tw/FreeBSD/ports/distfiles/erlang/armstrong_thesis_2003.pdf http://ftp.nsysu.edu.tw/FreeBSD/ports/distfiles/erlang/armst...
  - devishard 10y ago
    I don't think you can talk about fault tolerance and message passing as different in Erlang. Sure, one is the how, the other is the why, but that's just semantics.

by sakopov 10y ago

ben_jones 10y ago

Redis. Redis all the things. It can't be understated the peace of mind knowing your operations are somewhat atomic.

rashkov 10y ago
I think redis definitely has a place as a pubsub channel in a micro services architecture, and that in this capacity it solves the issue of sharing information among services. I'm not really seeing your point about atomic operations though. Would be curious to hear more about it
devonkim 10y ago
Wait, peace of mind knowing your operations are "somewhat" atomic? Am I not understanding something but isn't that something that would normally be disturbing when it comes to distributed transactions (what microservices oftentimes structure)?

chuhnk 10y ago

Event sourcing is the thing you're looking for and something that will probably be the focus of many building microservice platforms.

notduncansmith 10y ago
This. About 2 years ago, I started to notice the incredible usefulness of event sourcing and its applicability at various layers of the stack. From the React/Flux frontend paradigm, to the "everything is a stream" philosophy of Unix systems, to Erlang actors, to event sourcing for application backends, to databases like Datomic and CouchDB. This is why I'm so excited about Elm right now - it explicitly puts this at the core of every application, and gives you a really nice language and type system for encoding your domain logic and manipulating data. Time to write a blog post, I guess.
Terr_ 10y ago
I think the real network-level win comes from rigorous events, period. Event sourcing, on the other hand, is an internal implementation detail of a particular service... Which happens to get you a good stream of events as a side-effect.

pm90 10y ago

OpenStack actually does this really well, using RabbitMQ as the messaging tool. https://wiki.openstack.org/wiki/Oslo/Messaging https://wiki.openstack.org/wiki/Oslo/Messaging

devonkim 10y ago

Message buses and queues are far, far easier to manage than API endpoints across a bunch of loci of control, so this may be why it's not discussed as much as, say, how the lifecycle of REST services work. Topics and queues with different segmentation features are much more powerful and fine-grained in control compared to a service that's tied to HTTP transport than, say, a REST or SOAP API. For object serialization / marshaling, you can use a lot more stuff now all of a sudden like Protocol Buffers, Capn Proto, Thrift, Avro. The recommended advice comes down to what your bus / queue is (each one offers vastly different best practices and features to implement them) as well as how you want to scale out and up. But perhaps a general set of terminologies / jargon may be worth pursuing as a community for the problem. The question I really have is how people have managed to upgrade their message bus and to keep availability high with rollback contingencies while upgrading, say, RabbitMQ or Kafka across all your connected services. I really don't hear about that as much as how to do rolling restarts / upgrades of nodes for a service endpoint using some feedback from monitoring and metrics.

rando289 10y ago
> Topics and queues with different segmentation features are much more powerful and fine-grained in control compared to a service that's tied to HTTP transport than, say, a REST or SOAP API. "than" and beyond is an illogical statement. Like Y is Z than X.
- devonkim 10y ago
  Funky parse as I wrote it out, but you're correct. Despite the problem, I think the point being made is understandable. You can replace "than" with "like."
moonlighter 10y ago
Can't answer about RabbitMQ or Kafka, but if you're using a managed service like AWS SQS it's a non-issue. And if you need to test or make changes to a queue, it usually just means having to create another queue in parallel while the old one still works, and switch over once ready. The guys from Shazam gave a great talk about this at the AWS:reinvent conference back in 2013: https://www.youtube.com/watch?v=k_54Jcmi4zM https://www.youtube.com/watch?v=k_54Jcmi4zM

eva1984 10y ago

You need to further define 'share data'? Usually it could be done, to connect multiple different services through a central data storage, let it be redis/cassandra/RDBMS/HBase... If what you mean by sharing data is about dependency between calls to different services, e.g. A -> B -> C..., I would go with Storm and other stream processing frameworks, cause that is why those frameworks are built...

sakopov 10y ago
I'm simply talking about scenarios where one API needs to talk to another to get a resource it needs to act upon in some way. This can get vary hairy very quickly especially when each microservice ends up with copies of resources owned by other microservices that it now has to update asynchronously via pub/sub any time there is a change. This was the biggest challenge. And I think any teams willing to switch to this type of architecture need to look at it from this angle because monitoring and troubleshooting such systems is quite an undertaking.

devishard 10y ago

Have you tried Erlang? It's basically designed around solving that problem. Many people use RabbitMQ to solve that problem and it's no coincidence that RabbitMQ is written in Erlang. Elixir is possibly a similar option (don't know, haven't tried it).

greenleafjacob 10y ago
Erlang's raison d'etre is not message passing but fault tolerance. From Joe Armstrong[1]: > The work described in this thesis is the result of a research program started in 1981 to find better ways of programming Telecom applications. These applications are large programs which despite careful testing will probably contain many errors when the program is put into service. We assume that such programs do contain errors, and investigate methods for building reliable systems despite such errors. The research has resulted in the development of a new programming language (called Erlang), together with a design methodology, and set of libraries for building robust systems (called OTP). [1]: http://ftp.nsysu.edu.tw/FreeBSD/ports/distfiles/erlang/armstrong_thesis_2003.pdf http://ftp.nsysu.edu.tw/FreeBSD/ports/distfiles/erlang/armst...
- devishard 10y ago
  I don't think you can talk about fault tolerance and message passing as different in Erlang. Sure, one is the how, the other is the why, but that's just semantics.