5 ms·

> Back-pressure (e.g. slowing down the entry-points) can easily be introduced if queues becomes too large. ...which presumably includes load-shedding to stop m

by derefr 4y ago

> Back-pressure (e.g. slowing down the entry-points) can easily be introduced if queues becomes too large. ...which presumably includes load-shedding to stop misbehaving components from overloading the queues; at which point, unless you want clients to just lose track of the things they wanted done when they get a "we're too busy to handle this right now" response, you've essentially circled back around to clients having to use a client with REST-like "synchronous/blocking requests with retry/backpressure" semantics — just where the requests that are being synchronously-blocked on are "register this as a work-item and give me an ID to check on its status" rather than "do this entire job and tell me the result." And if you're doing that, why force the client to think in terms of async messaging at all? Just let them do REST, and hide the queue under the API layer of the receiver.

tass 4y ago
This is where I always end up. You can have queues which give you certain benefits, but there’a a lot of stuff to be built on top to make it as operationally simple as http.
- stolsvik 4y ago
  I will argue that this simplicity is exactly what Mats provides. At least that is the intention.
  - revskill 4y ago
    I don't see the code on webpage to explain things. Simplicity means you can explain complex things with simple code. Because English is ambigous and subjective. Just use code ?
    - stolsvik 4y ago
      There is code here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/ .. and here: https://mats3.io/docs/mats-flow-initiation/ https://mats3.io/docs/mats-flow-initiation/ .. and here: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ .. and here: https://mats3.io/docs/springconfig/ https://mats3.io/docs/springconfig/ .. and here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ .. and here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ .. and on the github page here: https://github.com/centiservice/mats3/blob/main/README.md https://github.com/centiservice/mats3/blob/main/README.md .. and you are advised to explore the code here: https://mats3.io/docs/explore/ https://mats3.io/docs/explore/
      - cerved 4y ago
        yes but there's sadly no code in what you posted
stolsvik 4y ago
Well, yes - there is nothing with Mats that you cannot do with any other communication form, if you code it up. When you say "register this as a work-item and give me an ID to check on its status", you've implemented a queue, right? The intention is that Mats gives you an easy way to perform async message-oriented communications. Somewhat of a bonus, you can also use it for synchronous tasks, using the MatsFuturizer, or MatsSocket. A queue can handle transient peaks of load much better than direct synchronous code. It is also quite simple to scale out. But if you do get into problems of getting too much traffic for the system to process, you will have to handle that - and Mats does not currently have any magic for performing e.g. load shedding, so you're on your own. (I have several thoughts on this. E.g. monitor the queue sizes, and deny any further initiations if the queues are too large). Wrt. synchronous comms, Mats do provide a nice feature, where you can mark a Mats Flow as "interactive", meaning that some human is waiting for the result. This results in the flow getting priority on every stage it passes through - so that if it competes with internal, more batchy processes, it will cut the lines.
- derefr 4y ago
  > A queue can handle transient peaks of load much better than direct synchronous code. Whether a workload is being managed upon creation using a work queue within the backend, has nothing to do with the semantics of the communications protocol used to talk about the state of said workload. You can arbitrarily combine these — for example, DBMSes have the unusual combination of having a stateful connection-oriented protocol for scheduling blocking workloads, but also having the ability to introspect the state of those ongoing workloads with queries on other connections. My point is that clients in a distributed system can literally never do "fire and forget" messaging anyway — which is the supposed advantage of an "asynchronous message-oriented communications" protocol over a REST-like one. Any client built to do "fire and forget" messaging, when used at scale, always, always ends up needing some sort of outbox-queue abstraction, where the outbox controller is internally doing synchronous blocking retries of RPC calls to get an acknowledgement that a message got safely pushed into the queue and can be locally forgotten. And that "outbox" is a leaky abstraction, because in trying to expose "fire and forget" semantics to its caller, it has no way of imposing backpressure on its caller. So the client's outbox overflows. Every time. This is why Google famously switched every internal protocol they use away from using message queues/busses with asynchronous "fire and forget" messaging, toward synchronous blocking RPC calls between services. With an explicitly-synchronous workload-submission protocol (which may as well just be over a request-oriented protocol like HTTP, as gRPC is), all operational errors and backpressure get bubbled back up from the workload-submission client library to its caller, where the caller can then have logic to decide the business-logic-level response that is most appropriate, for each particular fault, in each particular calling context. Message queues are the quintessential "smart pipe", trying to make the network handle all problems itself, so that the nodes (clients and backends) connected via such a network can be naive to some operational concerns. But this will never truly solve the problems it sets out to solve, as the policy knowledge to properly drive the decision-making for the mechanism that handles operational exigencies in message-handling, isn't available "within the network"; it lives only at the edges, in the client and backend application code of each service. Those exigencies — those failures and edge-case states — must be pushed out to the client or backend, so that policy can be applied. And if you're doing that, you may as well move the mechanism to enforce the policy there, too. At which point you're back to a dumb pipe, with smart nodes.
charrondev 4y ago
The system I’m currently on is currently moving a lot of work into queues. Some operations, like “change the criteria of this rank” could be anywhere between 5 seconds (if the number of users of the criteria to evaluate are small) or 10+ hours if we need re-evaluate the rules against 10m+ users. In this case we write our jobs as generators that can be paused, serialized and picked up again later. We give the job 5 seconds synchronously, then if it passes that time, queue and job and let the client know a job has been registered. The users account holds the IDs of the jobs as well as some basic information about the the tasks they have queued. There is a rest endpoint to return the current status of the jobs and information about them (what are they doing, what’s their progress, how much work remains). The client will negotiate a web socket connection with a different service to be notified whenever progress is made on the job and the client can then check the endpoint for the latest status.
- latchkey 4y ago
  That 5 seconds is going to bite you. There is going to be some sort of stall in the future that causes all of your jobs to hit that 5 seconds and everything is going to start to back up and cause other problems up the line that are really hard to test for in advance. You're better off designing a system that doesn't rely on some arbitrary number of seconds (why not 4 or 6 seconds?) to begin with.
  - naasking 4y ago
    Yes, non-determinism is the bane of distributed systems. It should be minimized whenever possible.
    - convolvatron 4y ago
      in general I've been a lot more successful making fewer assumptions than trying to enforce them
  - charrondev 4y ago
    The system doesn’t rely on the 5s FWIW worth and is capable of running entirely through the distributed queue. The 5s check is just there to improve UX on small user actions.
Supermancho 4y ago
> you've essentially circled back around to clients having to use a client with REST-like "synchronous/blocking requests with retry/backpressure" semantics Yes, they both do the same thing. That's not even the starting point of the discussion. The implementation from HTTP to a message queue (mailbox system) is the discussion point. Having the caller (who needs work done) wait to be informed when the work is done (or not done) is less deterministic than telling the callee how long before the work doesn't matter anymore. The callee gives back a transaction ID/is provided a callerID or is unavailable, and the caller knows (very quickly) it's not going to get done or knows where to look for the work (or abandon it). Either way, it allows for optimization on both sides.
- stolsvik 4y ago
  > .. telling the callee how long before the work doesn't matter anymore. That is interesting - this is actually a feature of JMS, which is employed in Mats: You can say that a Mats Flow is nonPersistent (i.e. "not all that important, really"), and in the same call you can also say how many seconds until timeout. <https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiator.MatsInitiate.html#nonPersistent(long) https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiat...> This is employed by the MatsFuturizer if you use the "nonEssential" invocation: <https://mats3.io/javadoc/mats3/0.19/modern/io/mats3/util/MatsFuturizer.html#futurizeNonessential(java.lang.String,java.lang.String,java.lang.String,java.lang.Class,java.lang.Object) https://mats3.io/javadoc/mats3/0.19/modern/io/mats3/util/Mat...> The point is that if there are too big queues, bunching up even more work on them which will not be processed in time does not make any sense: If the Future on the caller side have already timed out, it makes no sense at all that the callee should process the work and return the answer.
peoplefromibiza 4y ago
> And if you're doing that, why force the client to think in terms of async messaging at all? Just let them do REST, and hide the queue under the API layer of the receiver. because REST is stupid REST is request -> response, single connection, one direction only, which is a very limited way to model messaging . There is more than one communication mode and bidirectional messaging is a thing. REST also offers no control whatsoever over the communication channel, so you are stuck with the configuration set on the server side which might or might not be correct for your use case See RSocket for an example of a message driven protocol which solves most of the shortcomings of REST on the bright side REST is also stupid simple which is why is so widely deployed, it doesn't require thinking > response, you've essentially circled back around to clients having to use a client with REST-like no, because you did not block there waiting for the timeout which defaults to 30 seconds for HTTP and even if you abandon on the client side, the server will still process the request, there's no way to abort it once it's been started.
- naasking 4y ago
  > REST is request -> response, single connection, one direction only, which is a very limited way to model messaging You seem to be implying that being limited is a bad thing. Constraints are important to keep problems tractable.
  - klabb3 4y ago
    In my experience the amount of serialized (network blocking) calls needed under the request reply paradigm always grows over time as the application gets larger. At least this limitation can cause massive complexity once perf optimizations are needed. I think that’s important to factor in when we’re talking about the issues with large and resilient systems in either paradigm. Personally I like message passing because it’s more true to the underlying protocol (TCP or UDP) and actually interops quite well (all things considered) with request-reply systems - it just requires two separate messages and a request id which is standard practice in request-response anyway. The inverse is not true though: we have like 10 different hacky solutions in the last decade for sending server initiated messages to clients.
naasking 4y ago
> And if you're doing that, why force the client to think in terms of async messaging at all? Just let them do REST, and hide the queue under the API layer of the receiver. Yes, exactly. And on top of that, async messaging implicitly introduces DoS vulnerabilities exactly because of the buffering required. At least with sync messaging exposing a queue in the API layer, you opt-into this vulnerability.
- stolsvik 4y ago
  As mentioned here: https://mats3.io/background/system-of-services/ https://mats3.io/background/system-of-services/ .. Mats is meant to be an inter-service communication solution. It is explicitly not meant to be your front-facing endpoints. If you are DoS'ed, it would be from your own services. Of course, that might still happen, but then things would not have been much better if you used sync comms. It is true that you can bridge from sync to the async world of Mats using the MatsFuturizer (https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/), but then you still have your e.g. Servlet Container as the front-facing entity. (Also check out https://matssocket.io/ https://matssocket.io/, though)

by derefr 4y ago

tass 4y ago

This is where I always end up. You can have queues which give you certain benefits, but there’a a lot of stuff to be built on top to make it as operationally simple as http.

stolsvik 4y ago
I will argue that this simplicity is exactly what Mats provides. At least that is the intention.
- revskill 4y ago
  I don't see the code on webpage to explain things. Simplicity means you can explain complex things with simple code. Because English is ambigous and subjective. Just use code ?
  - stolsvik 4y ago
    There is code here: https://mats3.io/docs/message-oriented-rpc/ https://mats3.io/docs/message-oriented-rpc/ .. and here: https://mats3.io/docs/mats-flow-initiation/ https://mats3.io/docs/mats-flow-initiation/ .. and here: https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/ .. and here: https://mats3.io/docs/springconfig/ https://mats3.io/docs/springconfig/ .. and here: https://mats3.io/background/what-is-mats/ https://mats3.io/background/what-is-mats/ .. and here: https://mats3.io/using-mats/endpoints-and-initiations/ https://mats3.io/using-mats/endpoints-and-initiations/ .. and on the github page here: https://github.com/centiservice/mats3/blob/main/README.md https://github.com/centiservice/mats3/blob/main/README.md .. and you are advised to explore the code here: https://mats3.io/docs/explore/ https://mats3.io/docs/explore/
    - cerved 4y ago
      yes but there's sadly no code in what you posted

stolsvik 4y ago

Well, yes - there is nothing with Mats that you cannot do with any other communication form, if you code it up. When you say "register this as a work-item and give me an ID to check on its status", you've implemented a queue, right? The intention is that Mats gives you an easy way to perform async message-oriented communications. Somewhat of a bonus, you can also use it for synchronous tasks, using the MatsFuturizer, or MatsSocket. A queue can handle transient peaks of load much better than direct synchronous code. It is also quite simple to scale out. But if you do get into problems of getting too much traffic for the system to process, you will have to handle that - and Mats does not currently have any magic for performing e.g. load shedding, so you're on your own. (I have several thoughts on this. E.g. monitor the queue sizes, and deny any further initiations if the queues are too large). Wrt. synchronous comms, Mats do provide a nice feature, where you can mark a Mats Flow as "interactive", meaning that some human is waiting for the result. This results in the flow getting priority on every stage it passes through - so that if it competes with internal, more batchy processes, it will cut the lines.

derefr 4y ago
> A queue can handle transient peaks of load much better than direct synchronous code. Whether a workload is being managed upon creation using a work queue within the backend, has nothing to do with the semantics of the communications protocol used to talk about the state of said workload. You can arbitrarily combine these — for example, DBMSes have the unusual combination of having a stateful connection-oriented protocol for scheduling blocking workloads, but also having the ability to introspect the state of those ongoing workloads with queries on other connections. My point is that clients in a distributed system can literally never do "fire and forget" messaging anyway — which is the supposed advantage of an "asynchronous message-oriented communications" protocol over a REST-like one. Any client built to do "fire and forget" messaging, when used at scale, always, always ends up needing some sort of outbox-queue abstraction, where the outbox controller is internally doing synchronous blocking retries of RPC calls to get an acknowledgement that a message got safely pushed into the queue and can be locally forgotten. And that "outbox" is a leaky abstraction, because in trying to expose "fire and forget" semantics to its caller, it has no way of imposing backpressure on its caller. So the client's outbox overflows. Every time. This is why Google famously switched every internal protocol they use away from using message queues/busses with asynchronous "fire and forget" messaging, toward synchronous blocking RPC calls between services. With an explicitly-synchronous workload-submission protocol (which may as well just be over a request-oriented protocol like HTTP, as gRPC is), all operational errors and backpressure get bubbled back up from the workload-submission client library to its caller, where the caller can then have logic to decide the business-logic-level response that is most appropriate, for each particular fault, in each particular calling context. Message queues are the quintessential "smart pipe", trying to make the network handle all problems itself, so that the nodes (clients and backends) connected via such a network can be naive to some operational concerns. But this will never truly solve the problems it sets out to solve, as the policy knowledge to properly drive the decision-making for the mechanism that handles operational exigencies in message-handling, isn't available "within the network"; it lives only at the edges, in the client and backend application code of each service. Those exigencies — those failures and edge-case states — must be pushed out to the client or backend, so that policy can be applied. And if you're doing that, you may as well move the mechanism to enforce the policy there, too. At which point you're back to a dumb pipe, with smart nodes.

charrondev 4y ago

The system I’m currently on is currently moving a lot of work into queues. Some operations, like “change the criteria of this rank” could be anywhere between 5 seconds (if the number of users of the criteria to evaluate are small) or 10+ hours if we need re-evaluate the rules against 10m+ users. In this case we write our jobs as generators that can be paused, serialized and picked up again later. We give the job 5 seconds synchronously, then if it passes that time, queue and job and let the client know a job has been registered. The users account holds the IDs of the jobs as well as some basic information about the the tasks they have queued. There is a rest endpoint to return the current status of the jobs and information about them (what are they doing, what’s their progress, how much work remains). The client will negotiate a web socket connection with a different service to be notified whenever progress is made on the job and the client can then check the endpoint for the latest status.

latchkey 4y ago
That 5 seconds is going to bite you. There is going to be some sort of stall in the future that causes all of your jobs to hit that 5 seconds and everything is going to start to back up and cause other problems up the line that are really hard to test for in advance. You're better off designing a system that doesn't rely on some arbitrary number of seconds (why not 4 or 6 seconds?) to begin with.
- naasking 4y ago
  Yes, non-determinism is the bane of distributed systems. It should be minimized whenever possible.
  - convolvatron 4y ago
    in general I've been a lot more successful making fewer assumptions than trying to enforce them
- charrondev 4y ago
  The system doesn’t rely on the 5s FWIW worth and is capable of running entirely through the distributed queue. The 5s check is just there to improve UX on small user actions.

Supermancho 4y ago

> you've essentially circled back around to clients having to use a client with REST-like "synchronous/blocking requests with retry/backpressure" semantics Yes, they both do the same thing. That's not even the starting point of the discussion. The implementation from HTTP to a message queue (mailbox system) is the discussion point. Having the caller (who needs work done) wait to be informed when the work is done (or not done) is less deterministic than telling the callee how long before the work doesn't matter anymore. The callee gives back a transaction ID/is provided a callerID or is unavailable, and the caller knows (very quickly) it's not going to get done or knows where to look for the work (or abandon it). Either way, it allows for optimization on both sides.

stolsvik 4y ago
> .. telling the callee how long before the work doesn't matter anymore. That is interesting - this is actually a feature of JMS, which is employed in Mats: You can say that a Mats Flow is nonPersistent (i.e. "not all that important, really"), and in the same call you can also say how many seconds until timeout. <https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiator.MatsInitiate.html#nonPersistent(long) https://mats3.io/javadoc/mats3/0.19/api/io/mats3/MatsInitiat...> This is employed by the MatsFuturizer if you use the "nonEssential" invocation: <https://mats3.io/javadoc/mats3/0.19/modern/io/mats3/util/MatsFuturizer.html#futurizeNonessential(java.lang.String,java.lang.String,java.lang.String,java.lang.Class,java.lang.Object) https://mats3.io/javadoc/mats3/0.19/modern/io/mats3/util/Mat...> The point is that if there are too big queues, bunching up even more work on them which will not be processed in time does not make any sense: If the Future on the caller side have already timed out, it makes no sense at all that the callee should process the work and return the answer.

peoplefromibiza 4y ago

> And if you're doing that, why force the client to think in terms of async messaging at all? Just let them do REST, and hide the queue under the API layer of the receiver. because REST is stupid REST is request -> response, single connection, one direction only, which is a very limited way to model messaging . There is more than one communication mode and bidirectional messaging is a thing. REST also offers no control whatsoever over the communication channel, so you are stuck with the configuration set on the server side which might or might not be correct for your use case See RSocket for an example of a message driven protocol which solves most of the shortcomings of REST on the bright side REST is also stupid simple which is why is so widely deployed, it doesn't require thinking > response, you've essentially circled back around to clients having to use a client with REST-like no, because you did not block there waiting for the timeout which defaults to 30 seconds for HTTP and even if you abandon on the client side, the server will still process the request, there's no way to abort it once it's been started.

naasking 4y ago
> REST is request -> response, single connection, one direction only, which is a very limited way to model messaging You seem to be implying that being limited is a bad thing. Constraints are important to keep problems tractable.
- klabb3 4y ago
  In my experience the amount of serialized (network blocking) calls needed under the request reply paradigm always grows over time as the application gets larger. At least this limitation can cause massive complexity once perf optimizations are needed. I think that’s important to factor in when we’re talking about the issues with large and resilient systems in either paradigm. Personally I like message passing because it’s more true to the underlying protocol (TCP or UDP) and actually interops quite well (all things considered) with request-reply systems - it just requires two separate messages and a request id which is standard practice in request-response anyway. The inverse is not true though: we have like 10 different hacky solutions in the last decade for sending server initiated messages to clients.

naasking 4y ago

> And if you're doing that, why force the client to think in terms of async messaging at all? Just let them do REST, and hide the queue under the API layer of the receiver. Yes, exactly. And on top of that, async messaging implicitly introduces DoS vulnerabilities exactly because of the buffering required. At least with sync messaging exposing a queue in the API layer, you opt-into this vulnerability.

stolsvik 4y ago
As mentioned here: https://mats3.io/background/system-of-services/ https://mats3.io/background/system-of-services/ .. Mats is meant to be an inter-service communication solution. It is explicitly not meant to be your front-facing endpoints. If you are DoS'ed, it would be from your own services. Of course, that might still happen, but then things would not have been much better if you used sync comms. It is true that you can bridge from sync to the async world of Mats using the MatsFuturizer (https://mats3.io/docs/sync-async-bridge/ https://mats3.io/docs/sync-async-bridge/), but then you still have your e.g. Servlet Container as the front-facing entity. (Also check out https://matssocket.io/ https://matssocket.io/, though)