22 ms·

Async message-oriented systems vs. REST for inter-microservice communications

213 points by stolsvik 4y ago

deleted 4y ago
[deleted]
mrkeen 4y ago
> Transactionality: Each endpoint has either processed a message, done its work (possibly including changing something in a database), and sent a message, or none of it. Sounds too good to be true. Would love to hear more.
- stolsvik 4y ago
  Well, okay, you're right! It is nearly true, though. :) I've written a bit about it here: https://mats3.io/using-mats/transactions-and-redelivery/ https://mats3.io/using-mats/transactions-and-redelivery/
weatherlight 4y ago
chuckles in erlang
- drkrab 4y ago
  Yep
- stolsvik 4y ago
  :-) As mentioned a few places, the actor model is actually part of the inspiration behind Mats: https://mats3.io/background/what-is-mats/#attempt-at-a-condensed-explanation https://mats3.io/background/what-is-mats/#attempt-at-a-conde...
mdaniel 4y ago
Making your own license is the new JavaScript framework, I guess: https://github.com/centiservice/mats3/blob/v0.19.4-2023-02-10/LICENSE.md#polyform-perimeter-license-100 https://github.com/centiservice/mats3/blob/v0.19.4-2023-02-1...
- Pet_Ant 4y ago
  Not open source. > Noncompete > Any purpose is a permitted purpose, except for providing to others any product that competes with the software.
  - stolsvik 4y ago
    I do not say that it is Open Source either. From the front page: "Free to use, source on github. Noncompete licensed - PolyForm Perimeter." Feel free to comment on this: Is this a complete deal breaker for all potential users?
    - tinco 4y ago
      Well yeah of course, it's a direct contradiction of the rising tide lifts all boats principle. Do you think Kubernetes would have any traction at all if it had a clause that it couldn't be used on AWS? If it can't be adopted by the industry as a whole, then it can't be considered an industry standard. It wouldn't fly at my organization anyway, not even looking up what PolyForm is.
      - stavros 4y ago
        As far as I can see, this doesn't say it can't be used on AWS, it only says Amazon can't launch its own service that uses this software to compete with itself. It's too short to really tell what "compete" entails, though.
        stolsvik 4y ago
        You are correct, this is meant as an AWS/GCP/Azure preventor. ElasticSearch situation. That is, AFAIU, the intention of the license I adopted. The "examples" part spells it pretty directly out, as I also try to do here: https://centiservice.com/license/ https://centiservice.com/license/ You may definitely use it anywhere you like.
        4y ago
derefr 4y ago
> Back-pressure (e.g. slowing down the entry-points) can easily be introduced if queues becomes too large. ...which presumably includes load-shedding to stop misbehaving components from overloading the queues; at which point, unless you want clients to just lose track of the things they wanted done when they get a "we're too busy to handle this right now" response, you've essentially circled back around to clients having to use a client with REST-like "synchronous/blocking requests with retry/backpressure" semantics — just where the requests that are being synchronously-blocked on are "register this as a work-item and give me an ID to check on its status" rather than "do this entire job and tell me the result." And if you're doing that, why force the client to think in terms of async messaging at all? Just let them do REST, and hide the queue under the API layer of the receiver.
- tass 4y ago
  This is where I always end up. You can have queues which give you certain benefits, but there’a a lot of stuff to be built on top to make it as operationally simple as http.
  - stolsvik 4y ago
    I will argue that this simplicity is exactly what Mats provides. At least that is the intention.
    - revskill 4y ago
      I don't see the code on webpage to explain things. Simplicity means you can explain complex things with simple code. Because English is ambigous and subjective. Just use code ?
      - stolsvik 4y ago
        There is code here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/ .. and here: https://mats3.io/docs/mats-flow-initiation/ https://mats3.io/docs/mats-flow-initiation/ .. and here: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ .. and here: https://mats3.io/docs/springconfig/ https://mats3.io/docs/springconfig/ .. and here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ .. and here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ .. and on the github page here: https://github.com/centiservice/mats3/blob/main/README.md https://github.com/centiservice/mats3/blob/main/README.md .. and you are advised to explore the code here: https://mats3.io/docs/explore/ https://mats3.io/docs/explore/
ngrilly 4y ago
How is it different from NATS.io that solves most of the problems listed? (except the transactional aspect but I’m not convinced it’s a good thing to have the same tool do everything)
- stolsvik 4y ago
  I see references to NATS multiple times, but I fail to see how it solves what Mats aims to solve? Mats could be implemented on top of NATS, i.e. use NATS as a backend, instead of JMS. (We use ActiveMQ as the broker)
eterps 4y ago
"better than RPC" would be a more accurate title.
- stolsvik 4y ago
  Well, I actually call Mats "Message-Oriented Async RPC".
eikenberry 4y ago
Reminds me of the old Protocol vs. API debate. http://wiki.c2.com/?ApiVsProtocol= http://wiki.c2.com/?ApiVsProtocol=
- stolsvik 4y ago
  Isn't that more of the difference between JMS and AMQP?
nitwit005 4y ago
> Messaging naturally provides high availability, scalability, location transparency, prioritization, stage transactionality, fault tolerance, great monitoring, simple error handling, and efficient and flexible resource management. What is "stage transactionality"? If I do a Google search for it, I just find this page.
- stolsvik 4y ago
  Hehe, okay. It was meant to mean "Each stage is processed in a transaction". Kinda hard to get down into a list. But my wording evidently didn't make anything clearer! If you read a few more pages, then it should hopefully become clearer. This page is specifically talking about it: https://mats3.io/using-mats/transactions-and-redelivery/ https://mats3.io/using-mats/transactions-and-redelivery/ - but as it is one of the primary points of why I made Mats, it is mentioned multiple places, e.g. here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ This is not Mats-specific - it is directly using functionality provided by the message broker, via JMS.
revskill 4y ago
In our production apps, all network issues is resolved by simple rate-limiter.
- tatersolid 4y ago
  That rate limiter doesn’t solve anything for your users.
guhcampos 4y ago
Oh yes, another "this thing I sell is an actual silver bullet" post. Message busses are great. RPC is too. There are use cases for both. Saying one is "better" than the other is silly, and in this case, a shame. There are loads of message passing libraries out there, based on all kinds of backend, from RabbitMQ, to NATS, to Redis, to Kafka. This does not innovate over anything, it's just shameless marketing.
- stolsvik 4y ago
  This is unfair. I made Mats so that I could use messaging in a simpler form. Nothing else. Mats is an API that can be implemented on top of any queue-based message broker - which excludes Kafka. But it definitely includes ApacheMQ (which is what we use), Artemis and hence RedHat's MQ (which the tests runs), and RabbitMQ (whose JMS implementation is too limited to directly be used, but I do hope to implement Mats on top of it at some point). Probably also NATS. Probably also Apache Pulsar, which I just recently realized have a JMS client. You could even implement it on top of ZeroMQ, and implement it on top of any database, particularly Postgres since it has those "queue extensions" NOTIFY and SKIP LOCKED. edit: I actually have an feature-issue exploring such an implementation: https://github.com/centiservice/mats3/issues/15 https://github.com/centiservice/mats3/issues/15
  - latchkey 4y ago
    > ApacheMQ I hope ActiveMQ Artemis is better than the 'classic' version and that is what you're using. The last time I used it probably a decade ago now, there were so many issues with it, that it was a complete train wreck at scale. I would be very hesitant to pick that one up again.
jeffbee 4y ago
If you pretend that your message bus has zero producer impedance and costs nothing then this analysis makes great sense. If you have ever operated or paid for this type of scheme in the real world then you will have some doubts.
- stolsvik 4y ago
  I guess you'd say the same about cloud functions and lambdas, then? To which I agree. Paying per message would require the message cost to be pretty small. Might want to evaluate setting up a broker yourself if the cost starts getting high.
eBombzor 4y ago
Why is this better over Kafka?
- stolsvik 4y ago
  As far as I understand, Kafka is positioning itself to be the leading Event Sourcing solution. I view event sourcing to be fundamentally different from message passing. For a long time I tried to love event sourcing, but I see way to many problems with it. The primary problem I see is that you then end up with a massive source of events, which any service can subscribe to as they see fit. How is this different from having one gigantic spaghetti database? Also, event migrations over time. RPC and messaging feels to me to be much clearer separated: I own the Accounts, and you own the Orders. We explicitly communicate when we need to. I see benefits on both sides, but have firmly landed on not event sourcing.
samsquire 4y ago
Thanks for this. I love the idea of breaking up a flow into separately scheduled but still linear message flow. I wrote about a similar idea in ideas2 https://github.com/samsquire/ideas2#84-communication-code-slices https://github.com/samsquire/ideas2#84-communication-code-sl... The idea is that I enrich my code with comments and a transpiler schedules different parts of the code to different machines and inserts communication between blocks. I read about how Zookeeper algorithm for transactionality and robustness to messages being dropped, which is interesting reading. https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals.html https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals.... How does Mats compare? LMAX disruptor has a pattern where you split up each side of an IO request into two events, to avoid blocking in an handler. So you would always insert a new event to handle an IO response.
robertlagrant 4y ago
Having done a reasonable amount of messaging code in my time, I would say the final form of this sort of thing might look more like Cadence[0] than anything like this. [0] https://github.com/uber/cadence https://github.com/uber/cadence
- stolsvik 4y ago
  Cadence is a workflow management system. As is Temporal, Apache Beam, Airbnb Airflow, Netflix Conductor, Spotify Luigi, and even things like Github Actions, Google Cloud Workflows, Azure Service Fabric, AWS SWF, Power Automate. A primary difference is that those are external systems, where you define the flows inside that system - the system then "calling out" to get pieces of the flow done. Mats is an "internal" system: You code your flows inside the service. It is meant to directly replace synchronously calling out to REST services, instead enabling async messaging but with the added bonus of being as simple as using REST services. But yes, I see the point.
  - MuffinFlavored 4y ago
    Is GitHub Actions really similar enough to Temporal/Cadence to be included in the list?
    - stolsvik 4y ago
      Hmm. Maybe not. But they sure have much in common: You define a set of things that should be done, triggered by something - either a schedule, an event (oftentimes a repository event, but it doesn't have to), or from another Github action.
      - robertlagrant 4y ago
        I think that definition could encompass everything from CPU interrupts to having a human secretary : - )
    - lorendsr 4y ago
      You could call them both workflow management systems. One of the differences is Temporal/Cadence uses code to define workflows instead of YAML. It's a large enough difference that Temporal is defining their category as "durable execution": https://temporal.io/blog/building-reliable-distributed-systems-in-node https://temporal.io/blog/building-reliable-distributed-syste...
lp4vn 4y ago
I think this article is kind of misleading. You use messaging for asynchronous communication and REST for synchronous communication. The article makes me believe that using REST for synchronous communication is a kind of deprecated alternative in front of message passing.
- stolsvik 4y ago
  Not sure when you saw the HN post, but the original title was "Why messaging is much better than REST for inter-microservice communications" - @dang / HN changed it. The article specifically points out that this is for inter-service communication. However, you can also use the same system for synchronous communication out the your end user clients, then using the MatsFuturizer: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/
hbrn 4y ago
1. Anything that is connected to user interface should be synchronous by default. 2. You can't predict which parts of your system will be connected to user interface. 3. Here's the worst part: async messaging is viral. A service that depends on async service becomes async too. You should be very cautions introducing async messaging to your systems. The only parts that should be allowed to be async are the ones that can afford to fail. I spend good amount of time trying to work around these dumb enterprise patterns when building products on top of async APIs. You are literally forced to build inferior products just because someone thought that async messaging is so great. It's great for everybody, except the final user. Async processing is not a virtue, it's a necessity for high load/high throughput systems. The reason SOA failed many years ago is precisely the async message bus.
- stolsvik 4y ago
  We clearly do not agree. Wrt. sync processing when using Mats: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ But my better solution is instead to pull the async-ness all the way out to the client: https://matssocket.io/ https://matssocket.io/ Also, I have another take on the SOA failure, mentioned here: https://mats3.io/about/ https://mats3.io/about/ It was definitely not because of async, at least as I remember it.
  - SpaghettiX 4y ago
    I appreciate some events can be asynchronous for clients, for example: actions taken by other users, or events generated by the system. However, I do think implementation details (using async in the server) should be encapsulated from clients: when users save a new document, it's much easier for the client to receive a useful albeit delayed response, rather than "event submitted", wait for the result on a stream. Of course, other relevant clients may need to hear about that event too. The service architecture should not affect / make-life-harder for clients. Therefore I think disagree with both parent and grandparent comments. Use each when they make sense, not "synchronous by default" (grandparent comment, though I do think there are good points made), or "asynchronous based on service architecture" (parent comment). > But my better solution is to pull the async-ness all the way out to the client: https://matssocket.io/ https://matssocket.io/ Is that a solution that you use? I took a look at matssocket https://www.npmjs.com/package/matssocket https://www.npmjs.com/package/matssocket, it currently has 2 weekly downloads. :thinking:.
adamckay 4y ago
The articles notes about async messaging architectures being superior to REST-based systems seems rather disingenuous, in my opinion, as it's seemingly only considering the most basic REST API deployed on a single node as the alternative. For example: > High Availability: For each queue, you can have listeners on several service instances on different physical servers, so that if one service instance or one server goes down, the others are still handling messages. This is negated in a REST-based system with the use of an API gateway / simple load balancer and multiple upstream nodes. > Location Transparency [Elastic systems need to be adaptive and continuously react to changes in demand, they need to gracefully and efficiently increase and decrease scale.]: Service Location Discovery is avoided, as messages only targets the logical queue name, without needing information about which nodes are currently consuming from that queue. Fair enough, service discovery is another challenge, but it's not hugely complex with modern API gateways and arguably no more complex with running and maintaining a message queue with associated workers. You've also got a risk in a distributed messaging system used by multiple teams that one service publishes messages into a queue that has been deprecated and has no consumers listening anymore. > Scalability / Elasticity: It is easy to increase the number of nodes (or listeners per node) for a queue, thereby increasing throughput, without any clients needing reconfiguration. This can be done runtime, thus you get elasticity where the cluster grows or shrinks based on the load, e.g. by checking the size of queues. Same as HA, solved with a load balancer. > Transactionality: Each endpoint has either processed a message, done its work (possibly including changing something in a database), and sent a message, or none of it. > Resiliency / Fault Tolerance: If a node goes down mid-way in processing, the transactional aspect kicks in and rolls back the processing, and another node picks up. Due to the automatic retry-mechanism you get in a message based system, you also get fault tolerance: If you get a temporary failure (database is restarted, network is reconfigured), or you get a transient error (e.g. a concurrency situation in the database), both the database change and the message reception is rolled back, and the message broker will retry the message. These seem to be arguing the same point, and perhaps this is solved in the Mats library but as a general advantage of async message queues over synchronous REST calls, the message broker retrying the message or messages being lost isn't a given - they're difficult to get entirely right in both architectures. > Monitoring: All messages pass by the Message Broker, and can be logged and recorded, and made statistics on, to whatever degree one wants. >Debugging: The messages between different parts typically share a common format (e.g. strings and JSON), and can be inspected centrally on the Message Broker. Centralising via an API gateway can also offer these.
zmmmmm 4y ago
This reminds me more of Apache Camel[0] than other things it's being compared to. > The process initiator puts a message on a queue, and another processor picks that up (probably on a different service, on a different host, and in different code base) - does some processing, and puts its (intermediate) result on another queue This is almost exactly the definition of message routing (ie: Camel). I'm a bit doubtful about the pitch because the solution is presented as enabling you to maintain synchronous style programming while achieving benefits of async processing. This just isn't true, these are fundamental tradeoffs. If you need a synchronous answer back then no amount of queuing, routing, prioritisation, etc etc will save you when the fundamental resource providing that is unavailable, and the ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. If you go into this ad hoc, and build in a leaky abstraction that asynchronous things are are actually synchronous and vice versa, before you know it you are going to have unstable behaviour or even worse, deadlocks all over your system and the worst part - the true state of the system is now hidden in which messages are pending in transient message queues everywhere. What really matters here is to fundamentally design things from the start with patterns that allow you to be very explicit about what needs to be synchronous vs async (building on principles of idempotency, immutability, coherence, to maximise the cases where async is the answer). The notion of Apache Camel is to make all these decisions a first class elements of your framework and then to extract out the routing layer as a dedicated construct. The fact it generalises beyond message queues (treating literally anything that can provide a piece of data as a message provider) is a bonus. [0] https://camel.apache.org/ https://camel.apache.org/
- hummus_bae 4y ago
  > The ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. Async frameworks don't eliminate the possibility of long running processes, that continue to process long after responding a request - this is still possible with specific libraries/frameworks, they'll only take away the synchronous interface and provide as asychonous one instead. It is also important to note that error handling will be different between these 2 paradigms and it's important, whatever the most suitable one is, to acknowledge this since it forces us (developers) to handle the potential errors differently depending on the approach we choose.
rkangel 4y ago
If I understand this right, this is basically the Erlang/Elixir OTP programming model, but across microservices rather than across a single (potentially distributed) VM. To be clear - that is a good thing. One of the core concepts of OTP (effectively the Erlang standard library) is the GenServer. A GenServer processes incoming messages, mutates state if appropriate and sends responses. The OTP machinery means that this "send a message and wait for a response" is just a straight function call with return value to the caller. OTP takes care of all the edge cases (like when the process at the other end goes away half way through). This means that your code is just a straight series of synchronous function calls, which may be sending messages underneath to do things or get data, but you don't have to care. It's a lovely system to work in, and makes complicated systems feel simple. The elements communicating are in Erlang terminology 'processes' - but not OS processes, they are instead lightweight userspace schduled things - very lightweight to create. Erlang has built in distribution that allows you to connect multiple running machines, and then the same message passing works across network boundaries. You're still limited to the BEAM VM though. This is the 'full' microservice version of that.
- stolsvik 4y ago
  Thanks, I'll take that Erlang-comparison as praise! (I actually mention the actor model as inspiration here: https://mats3.io/background/what-is-mats/#attempt-at-a-condensed-explanation https://mats3.io/background/what-is-mats/#attempt-at-a-conde...)
- fy20 4y ago
  I'm pretty sure there should be a meme here, that any new async messaging system ends up reinventing Erlang/OTP.
  - stolsvik 4y ago
    There is: https://news.ycombinator.com/item?id=34767858 https://news.ycombinator.com/item?id=34767858
ay 4y ago
Makes me think of https://grugbrain.dev/ https://grugbrain.dev/: Microservices grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too seem very confusing to grug
weatherlight 4y ago
“Virding's First Rule of Programming: Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” ― Robert Virding
- stolsvik 4y ago
  :-) I actually mention the actor model as inspiration for Mats here: https://mats3.io/background/what-is-mats/#attempt-at-a-condensed-explanation https://mats3.io/background/what-is-mats/#attempt-at-a-conde...
awill88 4y ago
Clickbait imho because there are less charged ways to express the point of what is essentially a sales pitch comparing a generic synchronous web service. Why risk the cognitive dissonance by contriving the term “synchronous REST-based systems” What? Lol.
- stolsvik 4y ago
  The original title was "Why messaging is much better than REST for inter-microservice communications" - @dang / HN changed it. I feel the change is wrong in at least two ways: The linked article literally argues that messaging is much better ("superior"), and it is specifically arguing for the use in inter-service communications: "The arguments here are for service-mesh internal IPC/RPC communications. For e.g. REST endpoints facing the user/client, there is a “synchronous-2-Mats-bridge” called the MatsFuturizer" You can read about that bridge here: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/
deleted 4y ago
[deleted]
SNosTrAnDbLe 4y ago
How is this different from a standard enterprise service bus style architecture using ActiveMQ or Kafka ?
- stolsvik 4y ago
  I believe this is detailed a few places on that website, e.g. here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/ “Messaging with a call stack”, “invokable, message-based async endpoints”
captainbeef 4y ago
How does Mats compare with dapr?
- stolsvik 4y ago
  I am not entirely sure. I had bunched dapr together with Istio, as a "service-mesh" layer utilizing side-car pods to mesh the different pieces of your service. Just browsed it again, and there seems to be a bit more tooling in place. But I fail to see something like Mats in its feature list, though. Mats is about endpoints which can invoke other service's endpoints. It is "messaging with a call stack", simplifying how you can code messaging-based inter-service communications. Pub/sub is not the same, and (synchronous) REST/gRPC is what Mats explicitly is an alternative to.
deleted 4y ago
[deleted]
Animats 4y ago
OK, so they are using an "async" system to simulate a thread-oriented system with blocking. You can do that, but why? The primitive they've created is roughly equivalent to QNX messaging. That's a synchronous send message and wait for reply system, like REST. It just has a lot less overhead, and can be used both on the same machine and remotely. QNX lets you have multiple clients talking to the same server, and multiple servers serving the same connection port. People keep re-inventing this, but not improving it much. For this to really work well, the message passing has to be integrated with the CPU dispatcher, so that every interprocess call doesn't put you back to the end of the line for the CPU. That's what nobody seems to get right.
- vore 4y ago
  What happens when a high priority process RPCs to a low priority process to avoid priority inversion? Does the low priority process inherit the caller's priority for the duration of the call?
  - Animats 4y ago
    In QNX, yes, In this new thing, I don't know. That's a key feature of QNX, because it's a hard real time system. Priorities are taken very seriously. Process priority is also RPC priority, so higher priority processes get priority on services, such as file I/O.
  - stolsvik 4y ago
    Mats has two levels of priority: "ordinary" and "interactive". JavaDoc for interactive: <https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiator.MatsInitiate.html#interactive() https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiat...> Notice that this flag follows the entire Mats Flow, so that the initiated flow gets a "cut the line"-card for all queues it goes through.
- stolsvik 4y ago
  > they are using an "async" system to simulate a thread-oriented system with blocking. You can do that, but why? The "simulation" is purely visual, or rather "cognitive load reduction"-wise. It is explained multiple time throughout the pages om https://mats3.io https://mats3.io, for example here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/, and here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ (with multiple examples, both pure Java and Spring-based definitions, also comparing to an ordinary REST controller). The clue is to be able to use the simple mental model of sequential steps, with "RPC calls" intertwined. However, this is only on a superficial level, to make the developer's reasoning simple and familiar - what really happens is that your Mats Endpoint is really multiple stages, where each stage is an independent little message processor. To achieve this, Mats implements "Messaging with a Call Stack", where you get a state object which "magically" follows you through the stages, simulating the stack variables you'd have if it actually was a proper method. It works surprisingly well. > That's a synchronous send message and wait for reply system, like REST. This you get if you employ the MatsFuturizer: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ This is a "tack on"-tool to the otherwise fully async nature of Mats/messaging. > For this to really work well, the message passing has to be integrated with the CPU dispatcher It sounds like you are 100% set on speed. This is not really what Mats is after - it is meant as a inter-service communcation system, and IO will be your limiting factor at any rate. Mats sacrifices a bit of speed for developer ergonomics - the idea is that by easily enabling fully async development of ISC in a complex microservice system, you gain back that potential loss from a) actually being able to use fully async processing (!), and b) the inherent speed of messaging (it is at least as fast as HTTP, and you avoid the overhead of HTTP headers etc. It is mentioned here, "What Mats is not": https://github.com/centiservice/mats3#what-mats-is-not https://github.com/centiservice/mats3#what-mats-is-not
rajeshp1986 4y ago
The author lost me at the first few lines : > In a microservice architecture one needs to communicate between the different services. The golden hammer for such communication is REST-ish services employing JSON over HTTP. Are people really relying on REST-ish inter-services communications? I have seen very large systems and most of the systems are async/promised based or have an underlying message-queue implementation or pub-sub like Kafka.
- tsimionescu 4y ago
  Yes, REST or something like gRPC are probably the most common ways to do this. For example, the Kubernetes API server that sits at the center of Kubernetes' communication model is a REST server.
- stolsvik 4y ago
  Yes, as far as I have come to understand, REST and similar protocols like gRPC are definitely the most common inter-service communications solutions. Witness e.g. Netflix's stack with Hystrix, and the popular Kubernetes add-on Istio service mesh with its Envoy service proxy. And as the sibling comment points out, Kubernetes' own control layer is REST. Messaging is an alternative, which I believe is underused due to its completely different mental model which is harder to reason about. Mats tries to make that simple - by "emulating" that you are working with something like REST, by introducing "Messaging with a call stack" and the state object. More here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/, here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ and much more here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ edit: Btw: "async/promised based" - this is not the point. It is true that you can code REST with "async/await" or Promise-semantics. However, you have still then bound the process to a piece of memory on a server. If the server goes down, all the processes it held, be it blocking in a thread, or in some Promise object, goes down with it. With Mats, the process "lives in the message". As long as you got it onto the message broker (i.e., the initiation went OK), it will never disappear: It will either go to completion, or it will pop out into a DLQ (Dead Letter Queue), where it can be inspected, you can jump to your logging system to find its processing, and you can - importantly - reissue the message and thus restart the flow from where it stopped, if the underlying problem is now cleared (e.g. a database that was down, or some data that (due to a bug) was in some erroneous state which now is fixed) I write about that here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/, search for "A comment about the meaning of asynchronous". You can read about the MatsBrokerMonitor, a Mats-specific tool for inspecting the message broker: https://mats3.io/docs/matsbrokermonitor/ https://mats3.io/docs/matsbrokermonitor/. It gives more details about Mats flows than e.g. ActiveMQ's standard inspection GUI gives. I should however really work on that text, and throw in some images. You can clone and run the development-server if you want to see it in action: https://github.com/centiservice/matsbrokermonitor/blob/main/matsbrokermonitor-htmlgui/src/test/java/io/mats3/matsbrokermonitor/htmlgui/MatsBrokerMonitor_TestJettyServer.java https://github.com/centiservice/matsbrokermonitor/blob/main/...
lowbloodsugar 4y ago
> Service Location Discovery is avoided, as messages only targets the logical queue name, without needing information about which nodes are currently consuming from that queue. Something is talking to something over a network. They need to get sent somewhere so the “where” must be known. If the “where” is “a load balancer” then that works for REST too. If the “where” is a mesh discovery protocol then that works for REST too. Ultimately you service is going to send a packet. The network card requires a destination IP.
- stolsvik 4y ago
  In a message broker architecture, it is typically the clients connecting to the broker. Thus, as you correctly point out, they need to know where to connect. But that is the same value for every single client. Using the failover transport of ActiveMQ, we have defined amq1, amq2 and amq3 - so it'll cycle through these until one of them answers. Once you're in, you only refer to the queue names. Which in Mats effectively are direct references to the endpoint name, typically in a Java-style class.method way, e.g. "OrderService.placeOrder".
mangatmodi 4y ago
Not sure about the claim that async has better Transactionality then REST. >Transactionality: Each endpoint has either processed a message, done its work (possibly including changing something in a database), and sent a message, or none of it.
- stolsvik 4y ago
  The big point with messaging is that you have rollback, and retries. Mats leverages this. If Stage N in the total process has picked up a message, starts to process it, and then something (temporarily) fails (or the node crashes), then it will roll back whatever DB operations you have done up till then, and roll back that it has picked up the message. The MQ will now reissue the same message, and it will be picked up again. This time, things work out, a new message is produced, and the entire processing of this stage is committed. So, either you have received a message, done your DB-stuff, and sent an outgoing message, or you have done none of that. I do not see you easily do that with REST. Or at least you will have to code quite a bit to get such nice semantics. With messaging and Mats, you get it entirely for free. To be fair, it is not entirely true. There are two separate transactions going here, and it can fail in an annoying way. I write about this here: https://mats3.io/using-mats/transactions-and-redelivery/ https://mats3.io/using-mats/transactions-and-redelivery/
  - lnenad 4y ago
    But rollback and retries are specific to the systems executing the commands, not the interface that is invoking them. If you have an RPC based transaction (command1, command2, command3) that fills a list with rollback commands and in case of a failure run through the list of rollback commands that does the same thing as putting them in a queue. If you don't have rollbacks in the system (ie stuff was written in the db during the first command) the queue isn't gonna help with that.
    - stolsvik 4y ago
      I might not have gotten across clearly wrt. how this works. It is each stage that is transactional. If the stage processing fails (or the node crashes), both the DB transactiona, and the messaging transaction, is rolled back. It is then retried. ActiveMQ has a default of 1 delivery, and 6 redelivery attempts. If those 7 fails, the message is assumed "poison", and is put on the DLQ. But this means that a Mats Flow is a series of transactions, each is individually handled. As you probably allude to, you cannot roll back the entire flow - it is just the particular step that is rolled back. Thus, if the message ends up on the DLQ, you have a mid-way process, where the steps in front are already done and committed, while this stage, and any downstream, are not yet done. The message is however on the DLQ. If the problem was e.g. an temporary database failure, which now is resolved, you can now just reissue the DLQ (move it from the DLQ back to its queue), and the process will continue as if nothing had happened.
volf_ 4y ago
I made a very similar project in Rust that seems to mimic this idea: https://github.com/volfco/boxcar https://github.com/volfco/boxcar The core idea I had was to decouple the connection from the execution of the RPC. Mats3 looks to be doing a lot more than what I've done so far, but it's nice to see similar ideas out there to take inspiration from.

213 points by stolsvik 4y ago

deleted 4y ago

[deleted]

> Transactionality: Each endpoint has either processed a message, done its work (possibly including changing something in a database), and sent a message, or none of it. Sounds too good to be true. Would love to hear more.

stolsvik 4y ago
Well, okay, you're right! It is nearly true, though. :) I've written a bit about it here: https://mats3.io/using-mats/transactions-and-redelivery/ https://mats3.io/using-mats/transactions-and-redelivery/

weatherlight 4y ago

chuckles in erlang

drkrab 4y ago
Yep
stolsvik 4y ago
:-) As mentioned a few places, the actor model is actually part of the inspiration behind Mats: https://mats3.io/background/what-is-mats/#attempt-at-a-condensed-explanation https://mats3.io/background/what-is-mats/#attempt-at-a-conde...

mdaniel 4y ago

Making your own license is the new JavaScript framework, I guess: https://github.com/centiservice/mats3/blob/v0.19.4-2023-02-10/LICENSE.md#polyform-perimeter-license-100 https://github.com/centiservice/mats3/blob/v0.19.4-2023-02-1...

Pet_Ant 4y ago
Not open source. > Noncompete > Any purpose is a permitted purpose, except for providing to others any product that competes with the software.
- stolsvik 4y ago
  I do not say that it is Open Source either. From the front page: "Free to use, source on github. Noncompete licensed - PolyForm Perimeter." Feel free to comment on this: Is this a complete deal breaker for all potential users?
  - tinco 4y ago
    Well yeah of course, it's a direct contradiction of the rising tide lifts all boats principle. Do you think Kubernetes would have any traction at all if it had a clause that it couldn't be used on AWS? If it can't be adopted by the industry as a whole, then it can't be considered an industry standard. It wouldn't fly at my organization anyway, not even looking up what PolyForm is.
    - stavros 4y ago
      As far as I can see, this doesn't say it can't be used on AWS, it only says Amazon can't launch its own service that uses this software to compete with itself. It's too short to really tell what "compete" entails, though.
      - stolsvik 4y ago
        You are correct, this is meant as an AWS/GCP/Azure preventor. ElasticSearch situation. That is, AFAIU, the intention of the license I adopted. The "examples" part spells it pretty directly out, as I also try to do here: https://centiservice.com/license/ https://centiservice.com/license/ You may definitely use it anywhere you like.
        4y ago

derefr 4y ago

> Back-pressure (e.g. slowing down the entry-points) can easily be introduced if queues becomes too large. ...which presumably includes load-shedding to stop misbehaving components from overloading the queues; at which point, unless you want clients to just lose track of the things they wanted done when they get a "we're too busy to handle this right now" response, you've essentially circled back around to clients having to use a client with REST-like "synchronous/blocking requests with retry/backpressure" semantics — just where the requests that are being synchronously-blocked on are "register this as a work-item and give me an ID to check on its status" rather than "do this entire job and tell me the result." And if you're doing that, why force the client to think in terms of async messaging at all? Just let them do REST, and hide the queue under the API layer of the receiver.

tass 4y ago
This is where I always end up. You can have queues which give you certain benefits, but there’a a lot of stuff to be built on top to make it as operationally simple as http.
- stolsvik 4y ago
  I will argue that this simplicity is exactly what Mats provides. At least that is the intention.
  - revskill 4y ago
    I don't see the code on webpage to explain things. Simplicity means you can explain complex things with simple code. Because English is ambigous and subjective. Just use code ?
    - stolsvik 4y ago
      There is code here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/ .. and here: https://mats3.io/docs/mats-flow-initiation/ https://mats3.io/docs/mats-flow-initiation/ .. and here: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ .. and here: https://mats3.io/docs/springconfig/ https://mats3.io/docs/springconfig/ .. and here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ .. and here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ .. and on the github page here: https://github.com/centiservice/mats3/blob/main/README.md https://github.com/centiservice/mats3/blob/main/README.md .. and you are advised to explore the code here: https://mats3.io/docs/explore/ https://mats3.io/docs/explore/

ngrilly 4y ago

How is it different from NATS.io that solves most of the problems listed? (except the transactional aspect but I’m not convinced it’s a good thing to have the same tool do everything)

stolsvik 4y ago
I see references to NATS multiple times, but I fail to see how it solves what Mats aims to solve? Mats could be implemented on top of NATS, i.e. use NATS as a backend, instead of JMS. (We use ActiveMQ as the broker)

eterps 4y ago

"better than RPC" would be a more accurate title.

stolsvik 4y ago
Well, I actually call Mats "Message-Oriented Async RPC".

eikenberry 4y ago

Reminds me of the old Protocol vs. API debate. http://wiki.c2.com/?ApiVsProtocol= http://wiki.c2.com/?ApiVsProtocol=

stolsvik 4y ago
Isn't that more of the difference between JMS and AMQP?

nitwit005 4y ago

> Messaging naturally provides high availability, scalability, location transparency, prioritization, stage transactionality, fault tolerance, great monitoring, simple error handling, and efficient and flexible resource management. What is "stage transactionality"? If I do a Google search for it, I just find this page.

stolsvik 4y ago
Hehe, okay. It was meant to mean "Each stage is processed in a transaction". Kinda hard to get down into a list. But my wording evidently didn't make anything clearer! If you read a few more pages, then it should hopefully become clearer. This page is specifically talking about it: https://mats3.io/using-mats/transactions-and-redelivery/ https://mats3.io/using-mats/transactions-and-redelivery/ - but as it is one of the primary points of why I made Mats, it is mentioned multiple places, e.g. here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ This is not Mats-specific - it is directly using functionality provided by the message broker, via JMS.

revskill 4y ago

In our production apps, all network issues is resolved by simple rate-limiter.

tatersolid 4y ago
That rate limiter doesn’t solve anything for your users.

guhcampos 4y ago

Oh yes, another "this thing I sell is an actual silver bullet" post. Message busses are great. RPC is too. There are use cases for both. Saying one is "better" than the other is silly, and in this case, a shame. There are loads of message passing libraries out there, based on all kinds of backend, from RabbitMQ, to NATS, to Redis, to Kafka. This does not innovate over anything, it's just shameless marketing.

stolsvik 4y ago
This is unfair. I made Mats so that I could use messaging in a simpler form. Nothing else. Mats is an API that can be implemented on top of any queue-based message broker - which excludes Kafka. But it definitely includes ApacheMQ (which is what we use), Artemis and hence RedHat's MQ (which the tests runs), and RabbitMQ (whose JMS implementation is too limited to directly be used, but I do hope to implement Mats on top of it at some point). Probably also NATS. Probably also Apache Pulsar, which I just recently realized have a JMS client. You could even implement it on top of ZeroMQ, and implement it on top of any database, particularly Postgres since it has those "queue extensions" NOTIFY and SKIP LOCKED. edit: I actually have an feature-issue exploring such an implementation: https://github.com/centiservice/mats3/issues/15 https://github.com/centiservice/mats3/issues/15
- latchkey 4y ago
  > ApacheMQ I hope ActiveMQ Artemis is better than the 'classic' version and that is what you're using. The last time I used it probably a decade ago now, there were so many issues with it, that it was a complete train wreck at scale. I would be very hesitant to pick that one up again.

jeffbee 4y ago

If you pretend that your message bus has zero producer impedance and costs nothing then this analysis makes great sense. If you have ever operated or paid for this type of scheme in the real world then you will have some doubts.

stolsvik 4y ago
I guess you'd say the same about cloud functions and lambdas, then? To which I agree. Paying per message would require the message cost to be pretty small. Might want to evaluate setting up a broker yourself if the cost starts getting high.

eBombzor 4y ago

Why is this better over Kafka?

stolsvik 4y ago
As far as I understand, Kafka is positioning itself to be the leading Event Sourcing solution. I view event sourcing to be fundamentally different from message passing. For a long time I tried to love event sourcing, but I see way to many problems with it. The primary problem I see is that you then end up with a massive source of events, which any service can subscribe to as they see fit. How is this different from having one gigantic spaghetti database? Also, event migrations over time. RPC and messaging feels to me to be much clearer separated: I own the Accounts, and you own the Orders. We explicitly communicate when we need to. I see benefits on both sides, but have firmly landed on not event sourcing.

samsquire 4y ago

Thanks for this. I love the idea of breaking up a flow into separately scheduled but still linear message flow. I wrote about a similar idea in ideas2 https://github.com/samsquire/ideas2#84-communication-code-slices https://github.com/samsquire/ideas2#84-communication-code-sl... The idea is that I enrich my code with comments and a transpiler schedules different parts of the code to different machines and inserts communication between blocks. I read about how Zookeeper algorithm for transactionality and robustness to messages being dropped, which is interesting reading. https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals.html https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals.... How does Mats compare? LMAX disruptor has a pattern where you split up each side of an IO request into two events, to avoid blocking in an handler. So you would always insert a new event to handle an IO response.

robertlagrant 4y ago

Having done a reasonable amount of messaging code in my time, I would say the final form of this sort of thing might look more like Cadence[0] than anything like this. [0] https://github.com/uber/cadence https://github.com/uber/cadence

stolsvik 4y ago
Cadence is a workflow management system. As is Temporal, Apache Beam, Airbnb Airflow, Netflix Conductor, Spotify Luigi, and even things like Github Actions, Google Cloud Workflows, Azure Service Fabric, AWS SWF, Power Automate. A primary difference is that those are external systems, where you define the flows inside that system - the system then "calling out" to get pieces of the flow done. Mats is an "internal" system: You code your flows inside the service. It is meant to directly replace synchronously calling out to REST services, instead enabling async messaging but with the added bonus of being as simple as using REST services. But yes, I see the point.
- MuffinFlavored 4y ago
  Is GitHub Actions really similar enough to Temporal/Cadence to be included in the list?
  - stolsvik 4y ago
    Hmm. Maybe not. But they sure have much in common: You define a set of things that should be done, triggered by something - either a schedule, an event (oftentimes a repository event, but it doesn't have to), or from another Github action.
    - robertlagrant 4y ago
      I think that definition could encompass everything from CPU interrupts to having a human secretary : - )
  - lorendsr 4y ago
    You could call them both workflow management systems. One of the differences is Temporal/Cadence uses code to define workflows instead of YAML. It's a large enough difference that Temporal is defining their category as "durable execution": https://temporal.io/blog/building-reliable-distributed-systems-in-node https://temporal.io/blog/building-reliable-distributed-syste...

lp4vn 4y ago

I think this article is kind of misleading. You use messaging for asynchronous communication and REST for synchronous communication. The article makes me believe that using REST for synchronous communication is a kind of deprecated alternative in front of message passing.

stolsvik 4y ago
Not sure when you saw the HN post, but the original title was "Why messaging is much better than REST for inter-microservice communications" - @dang / HN changed it. The article specifically points out that this is for inter-service communication. However, you can also use the same system for synchronous communication out the your end user clients, then using the MatsFuturizer: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/

hbrn 4y ago

1. Anything that is connected to user interface should be synchronous by default. 2. You can't predict which parts of your system will be connected to user interface. 3. Here's the worst part: async messaging is viral. A service that depends on async service becomes async too. You should be very cautions introducing async messaging to your systems. The only parts that should be allowed to be async are the ones that can afford to fail. I spend good amount of time trying to work around these dumb enterprise patterns when building products on top of async APIs. You are literally forced to build inferior products just because someone thought that async messaging is so great. It's great for everybody, except the final user. Async processing is not a virtue, it's a necessity for high load/high throughput systems. The reason SOA failed many years ago is precisely the async message bus.

stolsvik 4y ago
We clearly do not agree. Wrt. sync processing when using Mats: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ But my better solution is instead to pull the async-ness all the way out to the client: https://matssocket.io/ https://matssocket.io/ Also, I have another take on the SOA failure, mentioned here: https://mats3.io/about/ https://mats3.io/about/ It was definitely not because of async, at least as I remember it.
- SpaghettiX 4y ago
  I appreciate some events can be asynchronous for clients, for example: actions taken by other users, or events generated by the system. However, I do think implementation details (using async in the server) should be encapsulated from clients: when users save a new document, it's much easier for the client to receive a useful albeit delayed response, rather than "event submitted", wait for the result on a stream. Of course, other relevant clients may need to hear about that event too. The service architecture should not affect / make-life-harder for clients. Therefore I think disagree with both parent and grandparent comments. Use each when they make sense, not "synchronous by default" (grandparent comment, though I do think there are good points made), or "asynchronous based on service architecture" (parent comment). > But my better solution is to pull the async-ness all the way out to the client: https://matssocket.io/ https://matssocket.io/ Is that a solution that you use? I took a look at matssocket https://www.npmjs.com/package/matssocket https://www.npmjs.com/package/matssocket, it currently has 2 weekly downloads. :thinking:.

adamckay 4y ago

The articles notes about async messaging architectures being superior to REST-based systems seems rather disingenuous, in my opinion, as it's seemingly only considering the most basic REST API deployed on a single node as the alternative. For example: > High Availability: For each queue, you can have listeners on several service instances on different physical servers, so that if one service instance or one server goes down, the others are still handling messages. This is negated in a REST-based system with the use of an API gateway / simple load balancer and multiple upstream nodes. > Location Transparency [Elastic systems need to be adaptive and continuously react to changes in demand, they need to gracefully and efficiently increase and decrease scale.]: Service Location Discovery is avoided, as messages only targets the logical queue name, without needing information about which nodes are currently consuming from that queue. Fair enough, service discovery is another challenge, but it's not hugely complex with modern API gateways and arguably no more complex with running and maintaining a message queue with associated workers. You've also got a risk in a distributed messaging system used by multiple teams that one service publishes messages into a queue that has been deprecated and has no consumers listening anymore. > Scalability / Elasticity: It is easy to increase the number of nodes (or listeners per node) for a queue, thereby increasing throughput, without any clients needing reconfiguration. This can be done runtime, thus you get elasticity where the cluster grows or shrinks based on the load, e.g. by checking the size of queues. Same as HA, solved with a load balancer. > Transactionality: Each endpoint has either processed a message, done its work (possibly including changing something in a database), and sent a message, or none of it. > Resiliency / Fault Tolerance: If a node goes down mid-way in processing, the transactional aspect kicks in and rolls back the processing, and another node picks up. Due to the automatic retry-mechanism you get in a message based system, you also get fault tolerance: If you get a temporary failure (database is restarted, network is reconfigured), or you get a transient error (e.g. a concurrency situation in the database), both the database change and the message reception is rolled back, and the message broker will retry the message. These seem to be arguing the same point, and perhaps this is solved in the Mats library but as a general advantage of async message queues over synchronous REST calls, the message broker retrying the message or messages being lost isn't a given - they're difficult to get entirely right in both architectures. > Monitoring: All messages pass by the Message Broker, and can be logged and recorded, and made statistics on, to whatever degree one wants. >Debugging: The messages between different parts typically share a common format (e.g. strings and JSON), and can be inspected centrally on the Message Broker. Centralising via an API gateway can also offer these.

zmmmmm 4y ago

This reminds me more of Apache Camel[0] than other things it's being compared to. > The process initiator puts a message on a queue, and another processor picks that up (probably on a different service, on a different host, and in different code base) - does some processing, and puts its (intermediate) result on another queue This is almost exactly the definition of message routing (ie: Camel). I'm a bit doubtful about the pitch because the solution is presented as enabling you to maintain synchronous style programming while achieving benefits of async processing. This just isn't true, these are fundamental tradeoffs. If you need a synchronous answer back then no amount of queuing, routing, prioritisation, etc etc will save you when the fundamental resource providing that is unavailable, and the ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. If you go into this ad hoc, and build in a leaky abstraction that asynchronous things are are actually synchronous and vice versa, before you know it you are going to have unstable behaviour or even worse, deadlocks all over your system and the worst part - the true state of the system is now hidden in which messages are pending in transient message queues everywhere. What really matters here is to fundamentally design things from the start with patterns that allow you to be very explicit about what needs to be synchronous vs async (building on principles of idempotency, immutability, coherence, to maximise the cases where async is the answer). The notion of Apache Camel is to make all these decisions a first class elements of your framework and then to extract out the routing layer as a dedicated construct. The fact it generalises beyond message queues (treating literally anything that can provide a piece of data as a message provider) is a bonus. [0] https://camel.apache.org/ https://camel.apache.org/

hummus_bae 4y ago
> The ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. Async frameworks don't eliminate the possibility of long running processes, that continue to process long after responding a request - this is still possible with specific libraries/frameworks, they'll only take away the synchronous interface and provide as asychonous one instead. It is also important to note that error handling will be different between these 2 paradigms and it's important, whatever the most suitable one is, to acknowledge this since it forces us (developers) to handle the potential errors differently depending on the approach we choose.

rkangel 4y ago

If I understand this right, this is basically the Erlang/Elixir OTP programming model, but across microservices rather than across a single (potentially distributed) VM. To be clear - that is a good thing. One of the core concepts of OTP (effectively the Erlang standard library) is the GenServer. A GenServer processes incoming messages, mutates state if appropriate and sends responses. The OTP machinery means that this "send a message and wait for a response" is just a straight function call with return value to the caller. OTP takes care of all the edge cases (like when the process at the other end goes away half way through). This means that your code is just a straight series of synchronous function calls, which may be sending messages underneath to do things or get data, but you don't have to care. It's a lovely system to work in, and makes complicated systems feel simple. The elements communicating are in Erlang terminology 'processes' - but not OS processes, they are instead lightweight userspace schduled things - very lightweight to create. Erlang has built in distribution that allows you to connect multiple running machines, and then the same message passing works across network boundaries. You're still limited to the BEAM VM though. This is the 'full' microservice version of that.

stolsvik 4y ago
Thanks, I'll take that Erlang-comparison as praise! (I actually mention the actor model as inspiration here: https://mats3.io/background/what-is-mats/#attempt-at-a-condensed-explanation https://mats3.io/background/what-is-mats/#attempt-at-a-conde...)
fy20 4y ago
I'm pretty sure there should be a meme here, that any new async messaging system ends up reinventing Erlang/OTP.
- stolsvik 4y ago
  There is: https://news.ycombinator.com/item?id=34767858 https://news.ycombinator.com/item?id=34767858

ay 4y ago

Makes me think of https://grugbrain.dev/ https://grugbrain.dev/: Microservices grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too seem very confusing to grug

weatherlight 4y ago

“Virding's First Rule of Programming: Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” ― Robert Virding

stolsvik 4y ago
:-) I actually mention the actor model as inspiration for Mats here: https://mats3.io/background/what-is-mats/#attempt-at-a-condensed-explanation https://mats3.io/background/what-is-mats/#attempt-at-a-conde...

awill88 4y ago

Clickbait imho because there are less charged ways to express the point of what is essentially a sales pitch comparing a generic synchronous web service. Why risk the cognitive dissonance by contriving the term “synchronous REST-based systems” What? Lol.

stolsvik 4y ago
The original title was "Why messaging is much better than REST for inter-microservice communications" - @dang / HN changed it. I feel the change is wrong in at least two ways: The linked article literally argues that messaging is much better ("superior"), and it is specifically arguing for the use in inter-service communications: "The arguments here are for service-mesh internal IPC/RPC communications. For e.g. REST endpoints facing the user/client, there is a “synchronous-2-Mats-bridge” called the MatsFuturizer" You can read about that bridge here: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/

deleted 4y ago

[deleted]

SNosTrAnDbLe 4y ago

How is this different from a standard enterprise service bus style architecture using ActiveMQ or Kafka ?

stolsvik 4y ago
I believe this is detailed a few places on that website, e.g. here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/ “Messaging with a call stack”, “invokable, message-based async endpoints”

captainbeef 4y ago

How does Mats compare with dapr?

stolsvik 4y ago
I am not entirely sure. I had bunched dapr together with Istio, as a "service-mesh" layer utilizing side-car pods to mesh the different pieces of your service. Just browsed it again, and there seems to be a bit more tooling in place. But I fail to see something like Mats in its feature list, though. Mats is about endpoints which can invoke other service's endpoints. It is "messaging with a call stack", simplifying how you can code messaging-based inter-service communications. Pub/sub is not the same, and (synchronous) REST/gRPC is what Mats explicitly is an alternative to.

deleted 4y ago

[deleted]

Animats 4y ago

OK, so they are using an "async" system to simulate a thread-oriented system with blocking. You can do that, but why? The primitive they've created is roughly equivalent to QNX messaging. That's a synchronous send message and wait for reply system, like REST. It just has a lot less overhead, and can be used both on the same machine and remotely. QNX lets you have multiple clients talking to the same server, and multiple servers serving the same connection port. People keep re-inventing this, but not improving it much. For this to really work well, the message passing has to be integrated with the CPU dispatcher, so that every interprocess call doesn't put you back to the end of the line for the CPU. That's what nobody seems to get right.

vore 4y ago
What happens when a high priority process RPCs to a low priority process to avoid priority inversion? Does the low priority process inherit the caller's priority for the duration of the call?
- Animats 4y ago
  In QNX, yes, In this new thing, I don't know. That's a key feature of QNX, because it's a hard real time system. Priorities are taken very seriously. Process priority is also RPC priority, so higher priority processes get priority on services, such as file I/O.
- stolsvik 4y ago
  Mats has two levels of priority: "ordinary" and "interactive". JavaDoc for interactive: <https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiator.MatsInitiate.html#interactive() https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiat...> Notice that this flag follows the entire Mats Flow, so that the initiated flow gets a "cut the line"-card for all queues it goes through.
stolsvik 4y ago
> they are using an "async" system to simulate a thread-oriented system with blocking. You can do that, but why? The "simulation" is purely visual, or rather "cognitive load reduction"-wise. It is explained multiple time throughout the pages om https://mats3.io https://mats3.io, for example here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/, and here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ (with multiple examples, both pure Java and Spring-based definitions, also comparing to an ordinary REST controller). The clue is to be able to use the simple mental model of sequential steps, with "RPC calls" intertwined. However, this is only on a superficial level, to make the developer's reasoning simple and familiar - what really happens is that your Mats Endpoint is really multiple stages, where each stage is an independent little message processor. To achieve this, Mats implements "Messaging with a Call Stack", where you get a state object which "magically" follows you through the stages, simulating the stack variables you'd have if it actually was a proper method. It works surprisingly well. > That's a synchronous send message and wait for reply system, like REST. This you get if you employ the MatsFuturizer: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ This is a "tack on"-tool to the otherwise fully async nature of Mats/messaging. > For this to really work well, the message passing has to be integrated with the CPU dispatcher It sounds like you are 100% set on speed. This is not really what Mats is after - it is meant as a inter-service communcation system, and IO will be your limiting factor at any rate. Mats sacrifices a bit of speed for developer ergonomics - the idea is that by easily enabling fully async development of ISC in a complex microservice system, you gain back that potential loss from a) actually being able to use fully async processing (!), and b) the inherent speed of messaging (it is at least as fast as HTTP, and you avoid the overhead of HTTP headers etc. It is mentioned here, "What Mats is not": https://github.com/centiservice/mats3#what-mats-is-not https://github.com/centiservice/mats3#what-mats-is-not

rajeshp1986 4y ago

The author lost me at the first few lines : > In a microservice architecture one needs to communicate between the different services. The golden hammer for such communication is REST-ish services employing JSON over HTTP. Are people really relying on REST-ish inter-services communications? I have seen very large systems and most of the systems are async/promised based or have an underlying message-queue implementation or pub-sub like Kafka.

tsimionescu 4y ago
Yes, REST or something like gRPC are probably the most common ways to do this. For example, the Kubernetes API server that sits at the center of Kubernetes' communication model is a REST server.
stolsvik 4y ago
Yes, as far as I have come to understand, REST and similar protocols like gRPC are definitely the most common inter-service communications solutions. Witness e.g. Netflix's stack with Hystrix, and the popular Kubernetes add-on Istio service mesh with its Envoy service proxy. And as the sibling comment points out, Kubernetes' own control layer is REST. Messaging is an alternative, which I believe is underused due to its completely different mental model which is harder to reason about. Mats tries to make that simple - by "emulating" that you are working with something like REST, by introducing "Messaging with a call stack" and the state object. More here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/, here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ and much more here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ edit: Btw: "async/promised based" - this is not the point. It is true that you can code REST with "async/await" or Promise-semantics. However, you have still then bound the process to a piece of memory on a server. If the server goes down, all the processes it held, be it blocking in a thread, or in some Promise object, goes down with it. With Mats, the process "lives in the message". As long as you got it onto the message broker (i.e., the initiation went OK), it will never disappear: It will either go to completion, or it will pop out into a DLQ (Dead Letter Queue), where it can be inspected, you can jump to your logging system to find its processing, and you can - importantly - reissue the message and thus restart the flow from where it stopped, if the underlying problem is now cleared (e.g. a database that was down, or some data that (due to a bug) was in some erroneous state which now is fixed) I write about that here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/, search for "A comment about the meaning of asynchronous". You can read about the MatsBrokerMonitor, a Mats-specific tool for inspecting the message broker: https://mats3.io/docs/matsbrokermonitor/ https://mats3.io/docs/matsbrokermonitor/. It gives more details about Mats flows than e.g. ActiveMQ's standard inspection GUI gives. I should however really work on that text, and throw in some images. You can clone and run the development-server if you want to see it in action: https://github.com/centiservice/matsbrokermonitor/blob/main/matsbrokermonitor-htmlgui/src/test/java/io/mats3/matsbrokermonitor/htmlgui/MatsBrokerMonitor_TestJettyServer.java https://github.com/centiservice/matsbrokermonitor/blob/main/...

lowbloodsugar 4y ago

> Service Location Discovery is avoided, as messages only targets the logical queue name, without needing information about which nodes are currently consuming from that queue. Something is talking to something over a network. They need to get sent somewhere so the “where” must be known. If the “where” is “a load balancer” then that works for REST too. If the “where” is a mesh discovery protocol then that works for REST too. Ultimately you service is going to send a packet. The network card requires a destination IP.

stolsvik 4y ago
In a message broker architecture, it is typically the clients connecting to the broker. Thus, as you correctly point out, they need to know where to connect. But that is the same value for every single client. Using the failover transport of ActiveMQ, we have defined amq1, amq2 and amq3 - so it'll cycle through these until one of them answers. Once you're in, you only refer to the queue names. Which in Mats effectively are direct references to the endpoint name, typically in a Java-style class.method way, e.g. "OrderService.placeOrder".

mangatmodi 4y ago

Not sure about the claim that async has better Transactionality then REST. >Transactionality: Each endpoint has either processed a message, done its work (possibly including changing something in a database), and sent a message, or none of it.

stolsvik 4y ago
The big point with messaging is that you have rollback, and retries. Mats leverages this. If Stage N in the total process has picked up a message, starts to process it, and then something (temporarily) fails (or the node crashes), then it will roll back whatever DB operations you have done up till then, and roll back that it has picked up the message. The MQ will now reissue the same message, and it will be picked up again. This time, things work out, a new message is produced, and the entire processing of this stage is committed. So, either you have received a message, done your DB-stuff, and sent an outgoing message, or you have done none of that. I do not see you easily do that with REST. Or at least you will have to code quite a bit to get such nice semantics. With messaging and Mats, you get it entirely for free. To be fair, it is not entirely true. There are two separate transactions going here, and it can fail in an annoying way. I write about this here: https://mats3.io/using-mats/transactions-and-redelivery/ https://mats3.io/using-mats/transactions-and-redelivery/
- lnenad 4y ago
  But rollback and retries are specific to the systems executing the commands, not the interface that is invoking them. If you have an RPC based transaction (command1, command2, command3) that fills a list with rollback commands and in case of a failure run through the list of rollback commands that does the same thing as putting them in a queue. If you don't have rollbacks in the system (ie stuff was written in the db during the first command) the queue isn't gonna help with that.
  - stolsvik 4y ago
    I might not have gotten across clearly wrt. how this works. It is each stage that is transactional. If the stage processing fails (or the node crashes), both the DB transactiona, and the messaging transaction, is rolled back. It is then retried. ActiveMQ has a default of 1 delivery, and 6 redelivery attempts. If those 7 fails, the message is assumed "poison", and is put on the DLQ. But this means that a Mats Flow is a series of transactions, each is individually handled. As you probably allude to, you cannot roll back the entire flow - it is just the particular step that is rolled back. Thus, if the message ends up on the DLQ, you have a mid-way process, where the steps in front are already done and committed, while this stage, and any downstream, are not yet done. The message is however on the DLQ. If the problem was e.g. an temporary database failure, which now is resolved, you can now just reissue the DLQ (move it from the DLQ back to its queue), and the process will continue as if nothing had happened.

volf_ 4y ago

I made a very similar project in Rust that seems to mimic this idea: https://github.com/volfco/boxcar https://github.com/volfco/boxcar The core idea I had was to decouple the connection from the execution of the RPC. Mats3 looks to be doing a lot more than what I've done so far, but it's nice to see similar ideas out there to take inspiration from.