6 ms·

Well, yes - there is nothing with Mats that you cannot do with any other communication form, if you code it up. When you say "register this as a work-item and g

Well, yes - there is nothing with Mats that you cannot do with any other communication form, if you code it up. When you say "register this as a work-item and give me an ID to check on its status", you've implemented a queue, right? The intention is that Mats gives you an easy way to perform async message-oriented communications. Somewhat of a bonus, you can also use it for synchronous tasks, using the MatsFuturizer, or MatsSocket. A queue can handle transient peaks of load much better than direct synchronous code. It is also quite simple to scale out. But if you do get into problems of getting too much traffic for the system to process, you will have to handle that - and Mats does not currently have any magic for performing e.g. load shedding, so you're on your own. (I have several thoughts on this. E.g. monitor the queue sizes, and deny any further initiations if the queues are too large). Wrt. synchronous comms, Mats do provide a nice feature, where you can mark a Mats Flow as "interactive", meaning that some human is waiting for the result. This results in the flow getting priority on every stage it passes through - so that if it competes with internal, more batchy processes, it will cut the lines.

derefr 4y ago
> A queue can handle transient peaks of load much better than direct synchronous code. Whether a workload is being managed upon creation using a work queue within the backend, has nothing to do with the semantics of the communications protocol used to talk about the state of said workload. You can arbitrarily combine these — for example, DBMSes have the unusual combination of having a stateful connection-oriented protocol for scheduling blocking workloads, but also having the ability to introspect the state of those ongoing workloads with queries on other connections. My point is that clients in a distributed system can literally never do "fire and forget" messaging anyway — which is the supposed advantage of an "asynchronous message-oriented communications" protocol over a REST-like one. Any client built to do "fire and forget" messaging, when used at scale, always, always ends up needing some sort of outbox-queue abstraction, where the outbox controller is internally doing synchronous blocking retries of RPC calls to get an acknowledgement that a message got safely pushed into the queue and can be locally forgotten. And that "outbox" is a leaky abstraction, because in trying to expose "fire and forget" semantics to its caller, it has no way of imposing backpressure on its caller. So the client's outbox overflows. Every time. This is why Google famously switched every internal protocol they use away from using message queues/busses with asynchronous "fire and forget" messaging, toward synchronous blocking RPC calls between services. With an explicitly-synchronous workload-submission protocol (which may as well just be over a request-oriented protocol like HTTP, as gRPC is), all operational errors and backpressure get bubbled back up from the workload-submission client library to its caller, where the caller can then have logic to decide the business-logic-level response that is most appropriate, for each particular fault, in each particular calling context. Message queues are the quintessential "smart pipe", trying to make the network handle all problems itself, so that the nodes (clients and backends) connected via such a network can be naive to some operational concerns. But this will never truly solve the problems it sets out to solve, as the policy knowledge to properly drive the decision-making for the mechanism that handles operational exigencies in message-handling, isn't available "within the network"; it lives only at the edges, in the client and backend application code of each service. Those exigencies — those failures and edge-case states — must be pushed out to the client or backend, so that policy can be applied. And if you're doing that, you may as well move the mechanism to enforce the policy there, too. At which point you're back to a dumb pipe, with smart nodes.
- jgilias 4y ago
  Is there something I can read about Google switching to sync RPC? Like a blog post or something like that? Thanks!
- stolsvik 4y ago
  "Not everybody is Google" These concepts has worked surprisingly well for us for nearly a decade. We're not Google-sized, but this architecture should work well for a few more orders of magnitude traffic. Also, you can mix and match. If you have some parts of your system with absolutely massive traffic, then don't use this there, then. Note that we very seldom use "fire and forget" (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging.
  - derefr 4y ago
    > Note that very we use "fire and forget" very seldom (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging. That doesn't help one bit. You're still firing-and-forgetting the request itself. The reply (presumably with a timeout) ensures that the client doesn't sit around forever waiting for a lost message; but it does nothing prevent badly-written request logic from overloading your backend (or overloading the queue, or "bunging up" the queue such that it'll be ~forever before your backend finishes handling the request spike and gets back to processing normal workloads.) > If you have some parts of your system with absolutely massive traffic, then don't use this there, then. I'm not talking about massive intended traffic. These problems come from failures in the architecture of the system to inherently bound requests to the current scale of the system (where autoscaling changes the "current scale of the system" before such limits kick in.) So, for example, there might be an endpoint in your system that allows the caller to trigger logic that does O(MN) work (the controller for that endpoint calls service X O(M) times, and then for each response from X, calls service Y O(N) times); where it's fully expected that this endpoint takes 60+ seconds to return a response. The endpoint was designed to serve the need of some existing internal team, who calls it for reporting once per day, with a batch-size N=2. But, unexpectedly, a new team, building a new component, with a new use-case for the same endpoint, writes logic that begins calling the endpoint once every 20 seconds, with a batch-size of 20. Now the queues for the services X and Y called by this endpoint are filling faster than they're emptying. No DDoS is happening; the requests are quite small, and in networking terms, quite sparse. Everything is working as intended — and yet it'll all fall over, because you've chosen yourself into a protocol where there's no inherent, by-default mechanism for "the backend is overloaded" to apply backpressure to make new requests from the frontend stop coming (as it would in a synchronous RPC protocol, where 1. you can't submit a request on an open socket when it's in the "waiting for reply" state; and 2. you can't get a new open socket if the backend isn't calling accept(2)); and you didn't think that this endpoint would be one that gets called much, so you didn't bother to think about explicitly implementing such a mechanism.
tacticus 4y ago
> you've implemented a queue, right? Yeah you have just without having to run it on top of a shared DB. message queues are just shared DBs with some extra ordering
- stolsvik 4y ago
  > message queues are just shared DBs with some extra ordering Completely agree. And I have stated that elsewhere in these threads. I mention it here: https://mats3.io/using-mats/matsfactory/#connecting-to-the-correct-jms-broker https://mats3.io/using-mats/matsfactory/#connecting-to-the-c... And I have a feature-issue that explores implementing Mats using DBs: https://github.com/centiservice/mats3/issues/15 https://github.com/centiservice/mats3/issues/15

derefr 4y ago

> A queue can handle transient peaks of load much better than direct synchronous code. Whether a workload is being managed upon creation using a work queue within the backend, has nothing to do with the semantics of the communications protocol used to talk about the state of said workload. You can arbitrarily combine these — for example, DBMSes have the unusual combination of having a stateful connection-oriented protocol for scheduling blocking workloads, but also having the ability to introspect the state of those ongoing workloads with queries on other connections. My point is that clients in a distributed system can literally never do "fire and forget" messaging anyway — which is the supposed advantage of an "asynchronous message-oriented communications" protocol over a REST-like one. Any client built to do "fire and forget" messaging, when used at scale, always, always ends up needing some sort of outbox-queue abstraction, where the outbox controller is internally doing synchronous blocking retries of RPC calls to get an acknowledgement that a message got safely pushed into the queue and can be locally forgotten. And that "outbox" is a leaky abstraction, because in trying to expose "fire and forget" semantics to its caller, it has no way of imposing backpressure on its caller. So the client's outbox overflows. Every time. This is why Google famously switched every internal protocol they use away from using message queues/busses with asynchronous "fire and forget" messaging, toward synchronous blocking RPC calls between services. With an explicitly-synchronous workload-submission protocol (which may as well just be over a request-oriented protocol like HTTP, as gRPC is), all operational errors and backpressure get bubbled back up from the workload-submission client library to its caller, where the caller can then have logic to decide the business-logic-level response that is most appropriate, for each particular fault, in each particular calling context. Message queues are the quintessential "smart pipe", trying to make the network handle all problems itself, so that the nodes (clients and backends) connected via such a network can be naive to some operational concerns. But this will never truly solve the problems it sets out to solve, as the policy knowledge to properly drive the decision-making for the mechanism that handles operational exigencies in message-handling, isn't available "within the network"; it lives only at the edges, in the client and backend application code of each service. Those exigencies — those failures and edge-case states — must be pushed out to the client or backend, so that policy can be applied. And if you're doing that, you may as well move the mechanism to enforce the policy there, too. At which point you're back to a dumb pipe, with smart nodes.

jgilias 4y ago
Is there something I can read about Google switching to sync RPC? Like a blog post or something like that? Thanks!
stolsvik 4y ago
"Not everybody is Google" These concepts has worked surprisingly well for us for nearly a decade. We're not Google-sized, but this architecture should work well for a few more orders of magnitude traffic. Also, you can mix and match. If you have some parts of your system with absolutely massive traffic, then don't use this there, then. Note that we very seldom use "fire and forget" (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging.
- derefr 4y ago
  > Note that very we use "fire and forget" very seldom (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging. That doesn't help one bit. You're still firing-and-forgetting the request itself. The reply (presumably with a timeout) ensures that the client doesn't sit around forever waiting for a lost message; but it does nothing prevent badly-written request logic from overloading your backend (or overloading the queue, or "bunging up" the queue such that it'll be ~forever before your backend finishes handling the request spike and gets back to processing normal workloads.) > If you have some parts of your system with absolutely massive traffic, then don't use this there, then. I'm not talking about massive intended traffic. These problems come from failures in the architecture of the system to inherently bound requests to the current scale of the system (where autoscaling changes the "current scale of the system" before such limits kick in.) So, for example, there might be an endpoint in your system that allows the caller to trigger logic that does O(MN) work (the controller for that endpoint calls service X O(M) times, and then for each response from X, calls service Y O(N) times); where it's fully expected that this endpoint takes 60+ seconds to return a response. The endpoint was designed to serve the need of some existing internal team, who calls it for reporting once per day, with a batch-size N=2. But, unexpectedly, a new team, building a new component, with a new use-case for the same endpoint, writes logic that begins calling the endpoint once every 20 seconds, with a batch-size of 20. Now the queues for the services X and Y called by this endpoint are filling faster than they're emptying. No DDoS is happening; the requests are quite small, and in networking terms, quite sparse. Everything is working as intended — and yet it'll all fall over, because you've chosen yourself into a protocol where there's no inherent, by-default mechanism for "the backend is overloaded" to apply backpressure to make new requests from the frontend stop coming (as it would in a synchronous RPC protocol, where 1. you can't submit a request on an open socket when it's in the "waiting for reply" state; and 2. you can't get a new open socket if the backend isn't calling accept(2)); and you didn't think that this endpoint would be one that gets called much, so you didn't bother to think about explicitly implementing such a mechanism.

tacticus 4y ago

> you've implemented a queue, right? Yeah you have just without having to run it on top of a shared DB. message queues are just shared DBs with some extra ordering

stolsvik 4y ago
> message queues are just shared DBs with some extra ordering Completely agree. And I have stated that elsewhere in these threads. I mention it here: https://mats3.io/using-mats/matsfactory/#connecting-to-the-correct-jms-broker https://mats3.io/using-mats/matsfactory/#connecting-to-the-c... And I have a feature-issue that explores implementing Mats using DBs: https://github.com/centiservice/mats3/issues/15 https://github.com/centiservice/mats3/issues/15