6 ms·

"Not everybody is Google" These concepts has worked surprisingly well for us for nearly a decade. We're not Google-sized, but this architecture should work wel

"Not everybody is Google" These concepts has worked surprisingly well for us for nearly a decade. We're not Google-sized, but this architecture should work well for a few more orders of magnitude traffic. Also, you can mix and match. If you have some parts of your system with absolutely massive traffic, then don't use this there, then. Note that we very seldom use "fire and forget" (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging.

derefr 4y ago
> Note that very we use "fire and forget" very seldom (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging. That doesn't help one bit. You're still firing-and-forgetting the request itself. The reply (presumably with a timeout) ensures that the client doesn't sit around forever waiting for a lost message; but it does nothing prevent badly-written request logic from overloading your backend (or overloading the queue, or "bunging up" the queue such that it'll be ~forever before your backend finishes handling the request spike and gets back to processing normal workloads.) > If you have some parts of your system with absolutely massive traffic, then don't use this there, then. I'm not talking about massive intended traffic. These problems come from failures in the architecture of the system to inherently bound requests to the current scale of the system (where autoscaling changes the "current scale of the system" before such limits kick in.) So, for example, there might be an endpoint in your system that allows the caller to trigger logic that does O(MN) work (the controller for that endpoint calls service X O(M) times, and then for each response from X, calls service Y O(N) times); where it's fully expected that this endpoint takes 60+ seconds to return a response. The endpoint was designed to serve the need of some existing internal team, who calls it for reporting once per day, with a batch-size N=2. But, unexpectedly, a new team, building a new component, with a new use-case for the same endpoint, writes logic that begins calling the endpoint once every 20 seconds, with a batch-size of 20. Now the queues for the services X and Y called by this endpoint are filling faster than they're emptying. No DDoS is happening; the requests are quite small, and in networking terms, quite sparse. Everything is working as intended — and yet it'll all fall over, because you've chosen yourself into a protocol where there's no inherent, by-default mechanism for "the backend is overloaded" to apply backpressure to make new requests from the frontend stop coming (as it would in a synchronous RPC protocol, where 1. you can't submit a request on an open socket when it's in the "waiting for reply" state; and 2. you can't get a new open socket if the backend isn't calling accept(2)); and you didn't think that this endpoint would be one that gets called much, so you didn't bother to think about explicitly implementing such a mechanism.
- stolsvik 4y ago
  Relying on the e.g. Servlet Container not being able to handle requests seems rather bad to me. That is a very rough error handling. We seem to have come to the exact opposite conclusions wrt. this. Your explanations are entirely in line with mine, but I found this "messy" error handling to be exactly what I wanted to avoid. There is one particular point where we might not be in line: I made Mats first and formost not for the synchronus situation, where there is a user waiting. This is the "bonus" part, where you can actually do that with the MatsFuturizer, or the MatsSocket. I first and foremost made it for internal, batch-like processes like "we got a new price (NAV) for this fund, we now need to settle these 5000 waiting orders". In that case, the work is bounded, and an error situation with not-enough-threads would be extremely messy. Queues solves this 100%. I've written some about my thinking on the About page: https://mats3.io/about/ https://mats3.io/about/
  - derefr 4y ago
    > Relying on the e.g. Servlet Container not being able to handle requests seems rather bad to me. That is a very rough error handling. It's one of those situations where the simplest "what you get by accident with a single-threaded non-evented server" solution, and the most fancy-and-complex solution, actually look alike from a client's perspective. What you actually want is that each of your backends monitors its own resource usage, and flags itself as unhealthy in its readiness-check endpoint when it's approaching its known per-backend maximum resource capacity along any particular dimension — threads, memory usage, DB pool checked-out connections, etc. (Which can be measured quite predictably, because you're very likely running these backends in containers or VMs that enforce bounds on these resources, and then scaling the resulting predictable-consumption workload-runners horizontally.) This readiness-check failure then causes the backend to be removed from consideration as an upstream for your load-balancer / routing target for your k8s Service / etc; but existing connected flows continue to flow, gradually draining the resource consumption on that backend, until it's low enough that the backend begins reporting itself as healthy again. Meanwhile, if the load-balancer gets a request and finds that it currently has no ready upstreams it can route to (because they're all unhealthy, because they're all at capacity) — then it responds with a 503. Just as if all those upstreams had crashed. > Your explanations are entirely in line with mine, but I found this "messy" error handling to be exactly what I wanted to avoid. Well, yes, but that's my point made above: this error handling is "messy" precisely because it's an encoding of user intent. It's irreducible complexity, because it's something where you want to make the decision of what to do differently in each case — e.g. a call from A to X might consider the X response critical (and so failures should be backoff-retried, and if retries exceeded, the whole job failed and rescheduled for later); while a call from B to X might consider the X response only a nice-to-have optimization over calculating the same data itself, and so it can try once, give up, and keep going. > I made Mats first and formost not for the synchronus situation, where there is a user waiting. I said nothing about users-as-in-humans. We're presumably both talking about a Service-Oriented Architecture here; perhaps even a microservice-oriented architecture. The "users" of Service X, above, are Service A and Service B. There's a Service X client library, that both Service A and Service B import, and make calls to Service X through. But these are still, necessarily, synchronous requests, since the further computations of Services A and B are dependent on the response from Service X. Sure, you can queue the requests to Services A and B as long as you like; but once they're running, they're going to sit around waiting on the response from Service X (because they have nothing better to be doing while the Service X response-promise resolves.) Whether or not the Service X request is synchronous or asynchronous doesn't matter to them; they have a synchronous (though not timely) need for the data, within their own asynchronous execution. Is this not the common pattern you see for inter-service requests within your own architecture? If not, then what is? If what you're really talking about here is forward-only propagation of values — i.e. never needing a response (timely or not) from most of the messages you send in the first place — then you're not really talking about a messaging protocol. You're talking about a dataflow programming model, and/or a distributed CQRS/ES event store — both of which can and often are implemented on top of message queues to great effect, and neither of which purport to be sensible to use to build RPC request-response code on top of.
bcrosby95 4y ago
RPC works for just non Google and Google scale. This is one of the times where, IMHO, you can skip the middle section. Novices resort to RPC, Google resorts to RPC, and in the mid tier you have something where messaging can step in. Why not skip it? Use RPC like a novice. If it becomes problematic, start putting in compensating measures.
- TedDoesntTalk 4y ago
  What about systems where the clients push more messages per second than consumers can process?
  - lenkite 4y ago
    Internal Mailbox/Queue on the receiving side. Caller gets back a transaction id that they can query for progress or caller provides a webhook for confirmation/failure. If mailbox is full, callee immediately responds with 429.
  - zeendo 4y ago
    To quote the gp hide the queue under the API layer of the receiver

derefr 4y ago

> Note that very we use "fire and forget" very seldom (aka "send(..)"). We use the request-replyTo paradigm much more. Which is basically the basic premise of Mats, as an abstraction over pure "forward-only" messaging. That doesn't help one bit. You're still firing-and-forgetting the request itself. The reply (presumably with a timeout) ensures that the client doesn't sit around forever waiting for a lost message; but it does nothing prevent badly-written request logic from overloading your backend (or overloading the queue, or "bunging up" the queue such that it'll be ~forever before your backend finishes handling the request spike and gets back to processing normal workloads.) > If you have some parts of your system with absolutely massive traffic, then don't use this there, then. I'm not talking about massive intended traffic. These problems come from failures in the architecture of the system to inherently bound requests to the current scale of the system (where autoscaling changes the "current scale of the system" before such limits kick in.) So, for example, there might be an endpoint in your system that allows the caller to trigger logic that does O(MN) work (the controller for that endpoint calls service X O(M) times, and then for each response from X, calls service Y O(N) times); where it's fully expected that this endpoint takes 60+ seconds to return a response. The endpoint was designed to serve the need of some existing internal team, who calls it for reporting once per day, with a batch-size N=2. But, unexpectedly, a new team, building a new component, with a new use-case for the same endpoint, writes logic that begins calling the endpoint once every 20 seconds, with a batch-size of 20. Now the queues for the services X and Y called by this endpoint are filling faster than they're emptying. No DDoS is happening; the requests are quite small, and in networking terms, quite sparse. Everything is working as intended — and yet it'll all fall over, because you've chosen yourself into a protocol where there's no inherent, by-default mechanism for "the backend is overloaded" to apply backpressure to make new requests from the frontend stop coming (as it would in a synchronous RPC protocol, where 1. you can't submit a request on an open socket when it's in the "waiting for reply" state; and 2. you can't get a new open socket if the backend isn't calling accept(2)); and you didn't think that this endpoint would be one that gets called much, so you didn't bother to think about explicitly implementing such a mechanism.

stolsvik 4y ago
Relying on the e.g. Servlet Container not being able to handle requests seems rather bad to me. That is a very rough error handling. We seem to have come to the exact opposite conclusions wrt. this. Your explanations are entirely in line with mine, but I found this "messy" error handling to be exactly what I wanted to avoid. There is one particular point where we might not be in line: I made Mats first and formost not for the synchronus situation, where there is a user waiting. This is the "bonus" part, where you can actually do that with the MatsFuturizer, or the MatsSocket. I first and foremost made it for internal, batch-like processes like "we got a new price (NAV) for this fund, we now need to settle these 5000 waiting orders". In that case, the work is bounded, and an error situation with not-enough-threads would be extremely messy. Queues solves this 100%. I've written some about my thinking on the About page: https://mats3.io/about/ https://mats3.io/about/
- derefr 4y ago
  > Relying on the e.g. Servlet Container not being able to handle requests seems rather bad to me. That is a very rough error handling. It's one of those situations where the simplest "what you get by accident with a single-threaded non-evented server" solution, and the most fancy-and-complex solution, actually look alike from a client's perspective. What you actually want is that each of your backends monitors its own resource usage, and flags itself as unhealthy in its readiness-check endpoint when it's approaching its known per-backend maximum resource capacity along any particular dimension — threads, memory usage, DB pool checked-out connections, etc. (Which can be measured quite predictably, because you're very likely running these backends in containers or VMs that enforce bounds on these resources, and then scaling the resulting predictable-consumption workload-runners horizontally.) This readiness-check failure then causes the backend to be removed from consideration as an upstream for your load-balancer / routing target for your k8s Service / etc; but existing connected flows continue to flow, gradually draining the resource consumption on that backend, until it's low enough that the backend begins reporting itself as healthy again. Meanwhile, if the load-balancer gets a request and finds that it currently has no ready upstreams it can route to (because they're all unhealthy, because they're all at capacity) — then it responds with a 503. Just as if all those upstreams had crashed. > Your explanations are entirely in line with mine, but I found this "messy" error handling to be exactly what I wanted to avoid. Well, yes, but that's my point made above: this error handling is "messy" precisely because it's an encoding of user intent. It's irreducible complexity, because it's something where you want to make the decision of what to do differently in each case — e.g. a call from A to X might consider the X response critical (and so failures should be backoff-retried, and if retries exceeded, the whole job failed and rescheduled for later); while a call from B to X might consider the X response only a nice-to-have optimization over calculating the same data itself, and so it can try once, give up, and keep going. > I made Mats first and formost not for the synchronus situation, where there is a user waiting. I said nothing about users-as-in-humans. We're presumably both talking about a Service-Oriented Architecture here; perhaps even a microservice-oriented architecture. The "users" of Service X, above, are Service A and Service B. There's a Service X client library, that both Service A and Service B import, and make calls to Service X through. But these are still, necessarily, synchronous requests, since the further computations of Services A and B are dependent on the response from Service X. Sure, you can queue the requests to Services A and B as long as you like; but once they're running, they're going to sit around waiting on the response from Service X (because they have nothing better to be doing while the Service X response-promise resolves.) Whether or not the Service X request is synchronous or asynchronous doesn't matter to them; they have a synchronous (though not timely) need for the data, within their own asynchronous execution. Is this not the common pattern you see for inter-service requests within your own architecture? If not, then what is? If what you're really talking about here is forward-only propagation of values — i.e. never needing a response (timely or not) from most of the messages you send in the first place — then you're not really talking about a messaging protocol. You're talking about a dataflow programming model, and/or a distributed CQRS/ES event store — both of which can and often are implemented on top of message queues to great effect, and neither of which purport to be sensible to use to build RPC request-response code on top of.

bcrosby95 4y ago

RPC works for just non Google and Google scale. This is one of the times where, IMHO, you can skip the middle section. Novices resort to RPC, Google resorts to RPC, and in the mid tier you have something where messaging can step in. Why not skip it? Use RPC like a novice. If it becomes problematic, start putting in compensating measures.

TedDoesntTalk 4y ago
What about systems where the clients push more messages per second than consumers can process?
- lenkite 4y ago
  Internal Mailbox/Queue on the receiving side. Caller gets back a transaction id that they can query for progress or caller provides a webhook for confirmation/failure. If mailbox is full, callee immediately responds with 429.
- zeendo 4y ago
  To quote the gp hide the queue under the API layer of the receiver