4 ms·

> The ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all

> The ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. Async frameworks don't eliminate the possibility of long running processes, that continue to process long after responding a request - this is still possible with specific libraries/frameworks, they'll only take away the synchronous interface and provide as asychonous one instead. It is also important to note that error handling will be different between these 2 paradigms and it's important, whatever the most suitable one is, to acknowledge this since it forces us (developers) to handle the potential errors differently depending on the approach we choose.

zmmmmm 4y ago
I think you're stating exactly my point? The pitch of MATS is that it let's: > developers code message-based endpoints that themselves may “invoke” other such endpoints, in a manner that closely resembles the familiar synchronous “straight down” linear code style In other words, they want to encourage you to feel like you are coding a synchronous workflow while actually coding an asynchronous one. You are pointing out that error handling needs to be different b/w these paradigms and you are correct, but that is only the start of it. A framework that papers over the differences is at very high risk of just creating a massive number of leaky abstractions that don't show up in the happy scenario but come back and bite you heavily when things go wrong. (I'm saying this as a long time user of Camel which models this exact concept heavily and also experiences many of these issues)
- stolsvik 4y ago
  Hmm. I want to distance this library pretty far from Camel! Wrt. "papering over": Not really. I make it "feel like" you're coding straight down, sequential, linear, as if you're coding synchronously. But if you look at the examples, e.g. at the very start of the Walkthrough: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/, you'll understand that you are actually coding completely message-driven: Each stage is a completely separate little "server", picking up messages from one queue, and most often putting a new message onto another queue. It is true that the error handling is very different. Don't code errors! You cannot throw an exception back to the caller. You can however make for "error return" style DTOs, but otherwise, if you have an actual error, it'll "pop out" of the Mats Fabric and end up on a DLQ. This is nice! It is not just a WARN or ERROR log-line in some log that no-one will see until way later, if ever: It immediately demands your attention. I wrote quite a long answer to something similar on a Reddit-thread a month ago: https://www.reddit.com/r/programming/comments/1059jpv/messaging_with_a_call_stack_messageoriented_async/j3cgefl/ https://www.reddit.com/r/programming/comments/1059jpv/messag...
  - zmmmmm 4y ago
    > Hmm. I want to distance this library pretty far from Camel! I'm curious what makes you distinguish it heavily from Camel? From everything you say it sounds to me like you are building Camel - or at least the routing part of it :-)
    - stolsvik 4y ago
      While these ideas was brewing, I ventured down into many libraries, Camel was one of them. It did not solve any of my problems. Mats is messaging with a call stack. One messaging endpoint can invoke another messaging endpoint, and get the reply supplied to its next stage, with the state from the previous stage magically present: https://mats3.io/docs/message-oriented-rpc/#isc-using-mats3 https://mats3.io/docs/message-oriented-rpc/#isc-using-mats3 and other pages. Camel has, AFAIK, nothing of the kind.
      - zmmmmm 4y ago
        It's interesting how we can have such different takes on it. Still everything you say maps naturally onto Camel for me. Your example using Camel would look something like: from('rest:/api/myprocess') // using REST API endpoint as an example starting point .inOut() .to('activemq:main') .to('activemq:mid') .to('activemq:leaf') .end() Then you would define the handlers: from('activemq:main') .transform { e -> return new State(...) } from('activemq:mid') .transform { e -> e.body.number1 = } All the processing stages are happening asynchronously and results are passed back and forth by Camel. If things error out half way through you get a printout from Camel of the whole workflow including each step, where exactly it failed and what the content of the state was at that point (and it'll throw the lost message onto DLQ if you want to re-do the process).
        stolsvik 4y ago
        Mats does not explicitly define a set "route" through a bunch of stages. I view Camel more like a Workflow system, which is pointed out elsewhere in the discussion threads here - the workflows are defined external to the actual processing steps. With Mats, you define Endpoints, which can be generic like a REST endpoint can be generic: "AccountService.getAccountListForCustomer" - which can be used (by invocation) by sevaral other endpoints. Also, IIRC, the process you show with Camel is now effectively thread bound - or at least JVM bound. If the node running that process goes down, it takes all unfinished, mid-flow, processes with it. The steps are not transactional. With Mats, the "life" of the process lives in the message. You can literally bring down your entire system from GCP, and rebuild it on Azure, and then when you start it up again, the mid-flow processes will just continue and finish as if nothing happened - as long as you brought along the state from the message broker (in addition to your databases). Viewed "from within the process", the only thing visible is that there was a bit more latency between one step and the next than usual. AFAIR, you cannot get anything like this with Camel. The idea is of course not the ability to move clouds, but that the result is exceptionally robust and stable.

zmmmmm 4y ago

I think you're stating exactly my point? The pitch of MATS is that it let's: > developers code message-based endpoints that themselves may “invoke” other such endpoints, in a manner that closely resembles the familiar synchronous “straight down” linear code style In other words, they want to encourage you to feel like you are coding a synchronous workflow while actually coding an asynchronous one. You are pointing out that error handling needs to be different b/w these paradigms and you are correct, but that is only the start of it. A framework that papers over the differences is at very high risk of just creating a massive number of leaky abstractions that don't show up in the happy scenario but come back and bite you heavily when things go wrong. (I'm saying this as a long time user of Camel which models this exact concept heavily and also experiences many of these issues)

stolsvik 4y ago
Hmm. I want to distance this library pretty far from Camel! Wrt. "papering over": Not really. I make it "feel like" you're coding straight down, sequential, linear, as if you're coding synchronously. But if you look at the examples, e.g. at the very start of the Walkthrough: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/, you'll understand that you are actually coding completely message-driven: Each stage is a completely separate little "server", picking up messages from one queue, and most often putting a new message onto another queue. It is true that the error handling is very different. Don't code errors! You cannot throw an exception back to the caller. You can however make for "error return" style DTOs, but otherwise, if you have an actual error, it'll "pop out" of the Mats Fabric and end up on a DLQ. This is nice! It is not just a WARN or ERROR log-line in some log that no-one will see until way later, if ever: It immediately demands your attention. I wrote quite a long answer to something similar on a Reddit-thread a month ago: https://www.reddit.com/r/programming/comments/1059jpv/messaging_with_a_call_stack_messageoriented_async/j3cgefl/ https://www.reddit.com/r/programming/comments/1059jpv/messag...
- zmmmmm 4y ago
  > Hmm. I want to distance this library pretty far from Camel! I'm curious what makes you distinguish it heavily from Camel? From everything you say it sounds to me like you are building Camel - or at least the routing part of it :-)
  - stolsvik 4y ago
    While these ideas was brewing, I ventured down into many libraries, Camel was one of them. It did not solve any of my problems. Mats is messaging with a call stack. One messaging endpoint can invoke another messaging endpoint, and get the reply supplied to its next stage, with the state from the previous stage magically present: https://mats3.io/docs/message-oriented-rpc/#isc-using-mats3 https://mats3.io/docs/message-oriented-rpc/#isc-using-mats3 and other pages. Camel has, AFAIK, nothing of the kind.
    - zmmmmm 4y ago
      It's interesting how we can have such different takes on it. Still everything you say maps naturally onto Camel for me. Your example using Camel would look something like: from('rest:/api/myprocess') // using REST API endpoint as an example starting point .inOut() .to('activemq:main') .to('activemq:mid') .to('activemq:leaf') .end() Then you would define the handlers: from('activemq:main') .transform { e -> return new State(...) } from('activemq:mid') .transform { e -> e.body.number1 = } All the processing stages are happening asynchronously and results are passed back and forth by Camel. If things error out half way through you get a printout from Camel of the whole workflow including each step, where exactly it failed and what the content of the state was at that point (and it'll throw the lost message onto DLQ if you want to re-do the process).
      - stolsvik 4y ago
        Mats does not explicitly define a set "route" through a bunch of stages. I view Camel more like a Workflow system, which is pointed out elsewhere in the discussion threads here - the workflows are defined external to the actual processing steps. With Mats, you define Endpoints, which can be generic like a REST endpoint can be generic: "AccountService.getAccountListForCustomer" - which can be used (by invocation) by sevaral other endpoints. Also, IIRC, the process you show with Camel is now effectively thread bound - or at least JVM bound. If the node running that process goes down, it takes all unfinished, mid-flow, processes with it. The steps are not transactional. With Mats, the "life" of the process lives in the message. You can literally bring down your entire system from GCP, and rebuild it on Azure, and then when you start it up again, the mid-flow processes will just continue and finish as if nothing happened - as long as you brought along the state from the message broker (in addition to your databases). Viewed "from within the process", the only thing visible is that there was a bit more latency between one step and the next than usual. AFAIR, you cannot get anything like this with Camel. The idea is of course not the ability to move clouds, but that the result is exceptionally robust and stable.