7 ms·

Building a fair multi-tenant queuing system

176 points by tonyhb 3y ago

quantum_bits 3y ago
This is cool but pretty light on the details. Judging by how it reads, they create queues for each function, then create queues of queues for workers. It sounds similar to the QuiCK paper from Apple: https://www.foundationdb.org/files/QuiCK.pdf https://www.foundationdb.org/files/QuiCK.pdf
jawns 3y ago
This problem can be extended to systems like project management, where your workers are not computers but people. For instance: Suppose you have a team of six people, and you're being asked to tackle two projects for two clients, each of equal value to the business. One project will take two weeks to complete; another will take four months to complete. How do you determine a "fair" way to divide up those projects? Do you do one before the other? Do you divide up your team's capacity equally and do them in parallel? This is the same sort of queuing problem described in the post. So a solution to the computer queuing problem might have interesting implications for the broader problem.
fabian2k 3y ago
There are plenty of articles about writing simple job queues in Postgres using SKIP LOCKED, and I personally like that version quite a bit for cases where you don't want to add another separate component just for the queue. But I haven't seen anyone addressing fairness in such a queue, e.g. for the multi-tenant case. Is there a simple way to get a very rough kind of fairness between tenants in this case? Doesn't have to be actually fair, just fair enough that no tenant gets fully stuck behind a large block of jobs from another tenant. The idea behind SKIP LOCKED is that you just get the first free job (potentially ordered by e.g. a priority column). Doing that in a way that switches between tenants doesn't seem straightforward. You could execute the query once per tenant, but that would be rather inefficient for many tenants. Is there a reasonably straightforward way to achieve this?
- dimitropoulos 3y ago
  Skip locked seems to be the way that new queueing packages are going, though they don’t solve fairness or multi-tenancy. It’s nice to keep things simple, but how does that work at scale? If I have 1,000 users I don’t think that these standard queueing systems can prevent one tenant from impacting another. It doesn’t look like that’s possible in SQS, either. On the other hand, is that even necessary? At what scale is this important?
  - fabian2k 3y ago
    I'm thinking about relatively simple systems and not "real" fairness. The case where I've seen this being a bit problematic is with such a queue in an application where jobs are sometimes inserted in large bursts. It doesn't actually matter much if they are truly fair, but in the cases where a large number of jobs is inserted at once the result is that for some tenants the queue doesn't seem to move at all. It's not a huge problem in that case because the regular priority system still puts the urgent jobs at the front of the queue. But it does sometimes cause confusion for someone looking at this from a single-tenant perspective because the queue looks stuck. There are other ways to fix the confusion, but it would be nice to have a very rudimentary fairness in this simple queue that would ensure that the queue iterates a bit over the tenants.
    - deleted 3y ago
      [deleted]
  - vosper 3y ago
    > On the other hand, is that even necessary? At what scale is this important? I can see it being helpful in front of a shared resource that doesn't have any way to control priority or fairness. My use case would be Elasticsearch. I don't want to block any searches from running if there's capacity, but also I don't want misbehaving clients to monopolise resources, and I want to make sure that all customers/users get to run their searches eventually, though I may prioritise some customers/users over others. And I may need my own internal jobs to run at a lower priority than a user search, but still within a guaranteed window (to avoid timeouts on the client side). Honestly for my use case this could be useful with just a handful of users - less than 10, even.
ctvo 3y ago
Is "step function" a term of art I'm just ignorant of in computer science? AWS Step Functions, sure. How it's used here? No idea. > It enables developers to write declarative step functions... > With thousands of customers running thousands of step functions containing parallel steps.... > If you're familiar with step functions, you probably already see what's happening here...
- hot_gril 3y ago
  "Step function" is sort of a standard term, yeah. Each step is some piece of code that runs at some point in the near future and persists its result for the next step to ingest. It could randomly fail (crashes, preemption, etc) before persisting, in which case the step is retried soon, so it better not have consequential side effects. Most likely you're interacting with external systems (like credit card processors) if you're using step functions in the first place, so idempotency is key. I've done basically this sorta manually at a smaller scale. Each step was a Postgres table, like "user order initiated," "order confirmed," "order fulfilled," with automated steps (sometimes cronjobs) in between. Sometimes I wanted to expose that status to users. I could see using Inngest at a larger scale.
- deleted 3y ago
  [deleted]
- rmbyrro 3y ago
  I don't think that's exactly the meaning on this article in particular, but I've seen people using "step function" as synonym to a step in a Finite-State Machine. SWE technical terms are used very loosely, especially on blog posts like this.
  - hot_gril 3y ago
    It also means something unrelated in mathematics
ykhli 3y ago
Great job abstracting away so much complexity! > each step is a code-level transaction backed by its own job in the queue. If the step fails, it retries automatically. Any data returned from the step is automatically captured into the function run's state and injected on each step call. This is one thing I've seen so many companies spending tons of time implementing themselves, and happens _everywhere_ -- no code apps, finance software, hospitals, anything that deals with ordering system...the list goes on. Glad I no longer need to write this from scratch!
hot_gril 3y ago
A "nuclear option" I've seen in the wild is Queue-It. Puts your entire flow behind a queue that's user-visible, with a waiting list page. Then presumably your own site's logic doesn't do any queuing.
- danfarrelly 3y ago
  How do they handle multi-tenancy for their users? Any post on that?
  - hot_gril 3y ago
    I don't know, but the requirements are different. They manage a single huge queue for your website (or part of it), and your own backend is expected to be able to handle some trivially small number of concurrent users. The most common use case is highly contentious online ticket sales, where users might even wait an hour in line. Latency isn't much of a concern.
andrewstuart 3y ago
>> with Postgres and SKIP LOCKED always cropping up Note that SKIP LOCKED is available in Microsoft SQL Server and MySQL and I believe possibly also in Oracle and IBM DB/2. To the article, I suspect there will be alot of detail challenged here. There's talk in here about "creating queues" to handle the fairness problem. I might be misunderstanding what the hard bit is, but with a database backed queue you don't have to create a queue - queues are virtual depending on what messages are in there and what other information is available such as username or tag. if you want to be "fair" around username then round robin on that, or fair around some other bit of data then tag the message when it goes into the queue and when you are processing, round robin on that. What is the hard bit here, what am I missing?
kakayako 3y ago
[dead]
maerF0x0 3y ago
btw "fair" is an entirely subjective and unproductive way to describe a system. There's simply properties and priorities. Many of us have learned fair to mean a great variance of things. From per capita equal load, to you get what you pay for, to anything goes just don't hurt anyone. In families we might see this as each child gets the same number of cookies. In capitalism fair is what you negotiate and agree to. The point isn't what is or isn't fair, but that there is no universal "fair". Should a user be able to jump the queue? What if they're your number one driver of revenue? Or they have a time-sensitive function w/ a deadline? Or if they've produced no load on the system in the past $TIME_UNIT. All of these are not fair, just properties to be designed into a product.
- 8n4vidtmkvmk 3y ago
  Yeah I didn't understand what the author even meant in the article. How is first come first serve not fair?
  - tonyhb 3y ago
    Fair means that we should be giving users roughly the same capacity, or at least roughly the same chances to be worked on. In the case of this queue, where each letter represents a user: [A, Bx10000, C, D, E] We're being unfair to C, D, and E. Realistically, while working on B's jobs we should have some mechanism to know that latency for C, D, and E are increasing and that we can start assigning them to workers. Without that, latency for any step function you run is highly variable and impacted by other users. With multi-tenant fairness, latency is purely driven by how well we auto-scale and your own concurrency capacity. The post here is about multi-tenant fairness in particular, so the intent is that fairness is viewed from a multi-tenant / PaaS lense.
    - maerF0x0 3y ago
      > or at least roughly the same chances to be worked on. Says who? What if B pays 10000x what ACDE do combined? What if B pays for that capacity? What if B was silent for the 10000 seconds prior? And is now bursting their load just like ACDE do at other times? There is no objective fair. Perhaps the only objectively unfair I could think of is if you paid for a service and it's not rendered, at all, ever. That's probably unfair. But everything else is a grey spectrum in between and depends on the agreed upon properties.
Nilithus 3y ago
I found Amazon's Builders Library to have a very insightful list of how to handle building multi tenant queue systems https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/ https://aws.amazon.com/builders-library/avoiding-insurmounta... I think one of the big tools is Shuffle Sharding. The article talks about standard sharding by itself as not being enough to provide robustness in multitenant queues. But Shuffle Sharding I.E. assigning users to virtual groups of underlying queues and enqueueing to the queue with the smallest size gets you pretty far. It can limit throughput for individual users but implementing some simple work stealing logic on the consumer helps make sure you keep your throughput up.
- extractionmech 3y ago
  That extra layer of indirection, virtual queues, is key.
- donavanm 3y ago
  David mentioned it but SQS `MessageGroupId` can get you really far on fair-ish work from the queue. Its effectively a virtual partition key for the queue. Set your customer id, resource id, etc as the MessageGroupId and use that to do the work allocation. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
jll29 3y ago
Fair queuing in multi-tenant scenarios is also what describes what is going on in some financial trading systems: these queuing mechanisms are often opened up to regulators to demonstrate the fairness.
brightball 3y ago
This honestly sounds like a job for the BEAM (Erlang/Elixir). The preemptive scheduler would cover almost every need mentioned in the article.
- darwin67 3y ago
  Which part of BEAM are you talking about? I know some cases can be solved by it already, but not almost every need. The fairness of the BEAM scheduler is not the same as multi-tenant fairness. I'm aware of lcnt in Erlang that helps with contention, but that will have a hit in throughput like any other locks. Unless I'm missing something?
  - brightball 3y ago
    The way the BEAM scheduler works you get max runtime per process before it switches over to another, which lets you run potentially millions of concurrent processes without having to worry about 1 taking it over because it was heavier. Your big stuff takes longer, but your response times across the board will remain very consistent. If you wanted to make it truly fair you could spin up a GenServer per tenant and you would have a max concurrency per tenant of 1, but still all executing equally in parallel without 1 tenant stealing CPU time from the others. Contention is a non-issue. Fairness is built in. You get the fault tolerance, isolation and observability as a by product too. That's just my first reaction on reading it.
- clarkdave 3y ago
  Not sure if this is what you're getting at, but RabbitMQ is indeed pretty great at having thousands of queues, even on modest hardware, and can be used for per-tenant queues in this way. It does have some drawbacks though: if you have thousands of mirrored or quorum queues, managing the cluster becomes more cumbersome - it can take a while to replace nodes (rebalancing thousands of queues around). Also, the management tools (the web UI especially) don't perform that well if you have 10,000s of queues and 100,000s of consumers
wilgertvelinga 3y ago
Sounds a lot like Azure Durable functions. Which we started using recently. Curious to know if inngest could also fit our use case.
siliconc0w 3y ago
I think you want controls around: 0) Load - I'm too loaded right now, I'm not processing your message. Maybe another worker will. This might be enqueued back into the low latency queue. 1) Role Limit - You've sent too many messages in too short of a time, I'm dequeueing this into the high latency queue. 2) Cost Limit - You've sent too many expensive messages, I'm dequeing this into the high latency queue. 3). Deadletter- Your messages are failing to be processed. We're going to dequeue into deadletter and maybe we're going stop processing them for for now. So you have the normal low latency queue, the high latency backlog queue, and the deadletter queue. Ideally you have SLOs around end to end queue times and monitoring/alerting to let you and your senders know their expectations may be not being met (i.e 99% of messages are processed within n MS).
- just_some_guy 3y ago
  As we've moved up in scaling the number of tenants we found tiered queues to be an anti-pattern. Inevitably some worker or upstream will get broken and you end up the high latency or dead letter queues with much more traffic than they're really designed to handle. Clearing it out then becomes a scramble since just scaling the high latency workers will often cause cascading issues. It makes sense if you frame it as "we're having trouble managing our 1 million queues" and the solution is to "create up to 1 million more queues". Instead we use a priority queue and adaptive concurrency. The priority queue lets you reschedule failed messages with a capped exponential backoff. That way it doesn't significantly impair the good items in the queue while still giving plenty of time to potentially fix the failing items. Adaptive concurrency increases & decreases the concurrency for that queue based on success & failure of workers on the queues items up to some upper & lower bounds. That way a single queue can't waste time scheduling work that's just going to most likely fail. We still have a dead letter queue, but it only ends up containing bugs since expiration is considered a normal message lifecycle event. So dead letter items just contain messages that were too broken to either send or expire correctly. The key take away is that as the number of queues increase it's better to just operate on the concurrency & ordering of the queue than to just keep creating more queues for the same work stream.
1oooqooq 3y ago
i thought RH had solved that. just write a systemd unit file with the cpu quotas. bam. done. /s?
mbreese 3y ago
Another place to look for inspiration is the HPC world. Fairshare job queues have been active on multi tenant HPC clusters for decades. Each job typically has a priority value assigned to it based on the submission time, expected resources required, and how much of the cluster the account/user has used recently. Priorities can be continuously updated until the job is released for processing.
deleted 3y ago
[deleted]
osigurdson 3y ago
NATS has multi tenancy built in.
mariya12 3y ago
[flagged]
ceritium 3y ago
For a small side project, instead of queuing all the tasks or writing a custom fair multi-tenant queuing system, I added all the "tasks" to the db, then with a periodic job I query with SKIP LOCKED for pending tasks and enqueue them on my queue system (sidekiq, Goodjob, etc). I only enqueue them if there are less than some amount of jobs pending, it is hardcoded, but I could do it dynamically. The query logic take into consideration the tenants so, a big tenant (or misbehaving) can't block small ones. The periodic job is executed every few second, but it's also enqueued when added new tasks to the db (unless it's already enqueued) or when the query logic detect that there is more jobs available, so the lag is minimal. At the moment I am doing this with only one type of jobs, but I want to expand it to most of the jobs, and let the "query logic" balance the queues.

176 points by tonyhb 3y ago

This is cool but pretty light on the details. Judging by how it reads, they create queues for each function, then create queues of queues for workers. It sounds similar to the QuiCK paper from Apple: https://www.foundationdb.org/files/QuiCK.pdf https://www.foundationdb.org/files/QuiCK.pdf

jawns 3y ago

This problem can be extended to systems like project management, where your workers are not computers but people. For instance: Suppose you have a team of six people, and you're being asked to tackle two projects for two clients, each of equal value to the business. One project will take two weeks to complete; another will take four months to complete. How do you determine a "fair" way to divide up those projects? Do you do one before the other? Do you divide up your team's capacity equally and do them in parallel? This is the same sort of queuing problem described in the post. So a solution to the computer queuing problem might have interesting implications for the broader problem.

fabian2k 3y ago

There are plenty of articles about writing simple job queues in Postgres using SKIP LOCKED, and I personally like that version quite a bit for cases where you don't want to add another separate component just for the queue. But I haven't seen anyone addressing fairness in such a queue, e.g. for the multi-tenant case. Is there a simple way to get a very rough kind of fairness between tenants in this case? Doesn't have to be actually fair, just fair enough that no tenant gets fully stuck behind a large block of jobs from another tenant. The idea behind SKIP LOCKED is that you just get the first free job (potentially ordered by e.g. a priority column). Doing that in a way that switches between tenants doesn't seem straightforward. You could execute the query once per tenant, but that would be rather inefficient for many tenants. Is there a reasonably straightforward way to achieve this?

dimitropoulos 3y ago
Skip locked seems to be the way that new queueing packages are going, though they don’t solve fairness or multi-tenancy. It’s nice to keep things simple, but how does that work at scale? If I have 1,000 users I don’t think that these standard queueing systems can prevent one tenant from impacting another. It doesn’t look like that’s possible in SQS, either. On the other hand, is that even necessary? At what scale is this important?
- fabian2k 3y ago
  I'm thinking about relatively simple systems and not "real" fairness. The case where I've seen this being a bit problematic is with such a queue in an application where jobs are sometimes inserted in large bursts. It doesn't actually matter much if they are truly fair, but in the cases where a large number of jobs is inserted at once the result is that for some tenants the queue doesn't seem to move at all. It's not a huge problem in that case because the regular priority system still puts the urgent jobs at the front of the queue. But it does sometimes cause confusion for someone looking at this from a single-tenant perspective because the queue looks stuck. There are other ways to fix the confusion, but it would be nice to have a very rudimentary fairness in this simple queue that would ensure that the queue iterates a bit over the tenants.
  - deleted 3y ago
    [deleted]
- vosper 3y ago
  > On the other hand, is that even necessary? At what scale is this important? I can see it being helpful in front of a shared resource that doesn't have any way to control priority or fairness. My use case would be Elasticsearch. I don't want to block any searches from running if there's capacity, but also I don't want misbehaving clients to monopolise resources, and I want to make sure that all customers/users get to run their searches eventually, though I may prioritise some customers/users over others. And I may need my own internal jobs to run at a lower priority than a user search, but still within a guaranteed window (to avoid timeouts on the client side). Honestly for my use case this could be useful with just a handful of users - less than 10, even.

ctvo 3y ago

Is "step function" a term of art I'm just ignorant of in computer science? AWS Step Functions, sure. How it's used here? No idea. > It enables developers to write declarative step functions... > With thousands of customers running thousands of step functions containing parallel steps.... > If you're familiar with step functions, you probably already see what's happening here...

hot_gril 3y ago
"Step function" is sort of a standard term, yeah. Each step is some piece of code that runs at some point in the near future and persists its result for the next step to ingest. It could randomly fail (crashes, preemption, etc) before persisting, in which case the step is retried soon, so it better not have consequential side effects. Most likely you're interacting with external systems (like credit card processors) if you're using step functions in the first place, so idempotency is key. I've done basically this sorta manually at a smaller scale. Each step was a Postgres table, like "user order initiated," "order confirmed," "order fulfilled," with automated steps (sometimes cronjobs) in between. Sometimes I wanted to expose that status to users. I could see using Inngest at a larger scale.
deleted 3y ago
[deleted]
rmbyrro 3y ago
I don't think that's exactly the meaning on this article in particular, but I've seen people using "step function" as synonym to a step in a Finite-State Machine. SWE technical terms are used very loosely, especially on blog posts like this.
- hot_gril 3y ago
  It also means something unrelated in mathematics

ykhli 3y ago

Great job abstracting away so much complexity! > each step is a code-level transaction backed by its own job in the queue. If the step fails, it retries automatically. Any data returned from the step is automatically captured into the function run's state and injected on each step call. This is one thing I've seen so many companies spending tons of time implementing themselves, and happens _everywhere_ -- no code apps, finance software, hospitals, anything that deals with ordering system...the list goes on. Glad I no longer need to write this from scratch!

hot_gril 3y ago

A "nuclear option" I've seen in the wild is Queue-It. Puts your entire flow behind a queue that's user-visible, with a waiting list page. Then presumably your own site's logic doesn't do any queuing.

danfarrelly 3y ago
How do they handle multi-tenancy for their users? Any post on that?
- hot_gril 3y ago
  I don't know, but the requirements are different. They manage a single huge queue for your website (or part of it), and your own backend is expected to be able to handle some trivially small number of concurrent users. The most common use case is highly contentious online ticket sales, where users might even wait an hour in line. Latency isn't much of a concern.

andrewstuart 3y ago

>> with Postgres and SKIP LOCKED always cropping up Note that SKIP LOCKED is available in Microsoft SQL Server and MySQL and I believe possibly also in Oracle and IBM DB/2. To the article, I suspect there will be alot of detail challenged here. There's talk in here about "creating queues" to handle the fairness problem. I might be misunderstanding what the hard bit is, but with a database backed queue you don't have to create a queue - queues are virtual depending on what messages are in there and what other information is available such as username or tag. if you want to be "fair" around username then round robin on that, or fair around some other bit of data then tag the message when it goes into the queue and when you are processing, round robin on that. What is the hard bit here, what am I missing?

kakayako 3y ago

[dead]

maerF0x0 3y ago

btw "fair" is an entirely subjective and unproductive way to describe a system. There's simply properties and priorities. Many of us have learned fair to mean a great variance of things. From per capita equal load, to you get what you pay for, to anything goes just don't hurt anyone. In families we might see this as each child gets the same number of cookies. In capitalism fair is what you negotiate and agree to. The point isn't what is or isn't fair, but that there is no universal "fair". Should a user be able to jump the queue? What if they're your number one driver of revenue? Or they have a time-sensitive function w/ a deadline? Or if they've produced no load on the system in the past $TIME_UNIT. All of these are not fair, just properties to be designed into a product.

8n4vidtmkvmk 3y ago
Yeah I didn't understand what the author even meant in the article. How is first come first serve not fair?
- tonyhb 3y ago
  Fair means that we should be giving users roughly the same capacity, or at least roughly the same chances to be worked on. In the case of this queue, where each letter represents a user: [A, Bx10000, C, D, E] We're being unfair to C, D, and E. Realistically, while working on B's jobs we should have some mechanism to know that latency for C, D, and E are increasing and that we can start assigning them to workers. Without that, latency for any step function you run is highly variable and impacted by other users. With multi-tenant fairness, latency is purely driven by how well we auto-scale and your own concurrency capacity. The post here is about multi-tenant fairness in particular, so the intent is that fairness is viewed from a multi-tenant / PaaS lense.
  - maerF0x0 3y ago
    > or at least roughly the same chances to be worked on. Says who? What if B pays 10000x what ACDE do combined? What if B pays for that capacity? What if B was silent for the 10000 seconds prior? And is now bursting their load just like ACDE do at other times? There is no objective fair. Perhaps the only objectively unfair I could think of is if you paid for a service and it's not rendered, at all, ever. That's probably unfair. But everything else is a grey spectrum in between and depends on the agreed upon properties.

Nilithus 3y ago

I found Amazon's Builders Library to have a very insightful list of how to handle building multi tenant queue systems https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/ https://aws.amazon.com/builders-library/avoiding-insurmounta... I think one of the big tools is Shuffle Sharding. The article talks about standard sharding by itself as not being enough to provide robustness in multitenant queues. But Shuffle Sharding I.E. assigning users to virtual groups of underlying queues and enqueueing to the queue with the smallest size gets you pretty far. It can limit throughput for individual users but implementing some simple work stealing logic on the consumer helps make sure you keep your throughput up.

extractionmech 3y ago
That extra layer of indirection, virtual queues, is key.
donavanm 3y ago
David mentioned it but SQS `MessageGroupId` can get you really far on fair-ish work from the queue. Its effectively a virtual partition key for the queue. Set your customer id, resource id, etc as the MessageGroupId and use that to do the work allocation. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

jll29 3y ago

Fair queuing in multi-tenant scenarios is also what describes what is going on in some financial trading systems: these queuing mechanisms are often opened up to regulators to demonstrate the fairness.

brightball 3y ago

This honestly sounds like a job for the BEAM (Erlang/Elixir). The preemptive scheduler would cover almost every need mentioned in the article.

darwin67 3y ago
Which part of BEAM are you talking about? I know some cases can be solved by it already, but not almost every need. The fairness of the BEAM scheduler is not the same as multi-tenant fairness. I'm aware of lcnt in Erlang that helps with contention, but that will have a hit in throughput like any other locks. Unless I'm missing something?
- brightball 3y ago
  The way the BEAM scheduler works you get max runtime per process before it switches over to another, which lets you run potentially millions of concurrent processes without having to worry about 1 taking it over because it was heavier. Your big stuff takes longer, but your response times across the board will remain very consistent. If you wanted to make it truly fair you could spin up a GenServer per tenant and you would have a max concurrency per tenant of 1, but still all executing equally in parallel without 1 tenant stealing CPU time from the others. Contention is a non-issue. Fairness is built in. You get the fault tolerance, isolation and observability as a by product too. That's just my first reaction on reading it.
clarkdave 3y ago
Not sure if this is what you're getting at, but RabbitMQ is indeed pretty great at having thousands of queues, even on modest hardware, and can be used for per-tenant queues in this way. It does have some drawbacks though: if you have thousands of mirrored or quorum queues, managing the cluster becomes more cumbersome - it can take a while to replace nodes (rebalancing thousands of queues around). Also, the management tools (the web UI especially) don't perform that well if you have 10,000s of queues and 100,000s of consumers

wilgertvelinga 3y ago

Sounds a lot like Azure Durable functions. Which we started using recently. Curious to know if inngest could also fit our use case.

siliconc0w 3y ago

I think you want controls around: 0) Load - I'm too loaded right now, I'm not processing your message. Maybe another worker will. This might be enqueued back into the low latency queue. 1) Role Limit - You've sent too many messages in too short of a time, I'm dequeueing this into the high latency queue. 2) Cost Limit - You've sent too many expensive messages, I'm dequeing this into the high latency queue. 3). Deadletter- Your messages are failing to be processed. We're going to dequeue into deadletter and maybe we're going stop processing them for for now. So you have the normal low latency queue, the high latency backlog queue, and the deadletter queue. Ideally you have SLOs around end to end queue times and monitoring/alerting to let you and your senders know their expectations may be not being met (i.e 99% of messages are processed within n MS).

just_some_guy 3y ago
As we've moved up in scaling the number of tenants we found tiered queues to be an anti-pattern. Inevitably some worker or upstream will get broken and you end up the high latency or dead letter queues with much more traffic than they're really designed to handle. Clearing it out then becomes a scramble since just scaling the high latency workers will often cause cascading issues. It makes sense if you frame it as "we're having trouble managing our 1 million queues" and the solution is to "create up to 1 million more queues". Instead we use a priority queue and adaptive concurrency. The priority queue lets you reschedule failed messages with a capped exponential backoff. That way it doesn't significantly impair the good items in the queue while still giving plenty of time to potentially fix the failing items. Adaptive concurrency increases & decreases the concurrency for that queue based on success & failure of workers on the queues items up to some upper & lower bounds. That way a single queue can't waste time scheduling work that's just going to most likely fail. We still have a dead letter queue, but it only ends up containing bugs since expiration is considered a normal message lifecycle event. So dead letter items just contain messages that were too broken to either send or expire correctly. The key take away is that as the number of queues increase it's better to just operate on the concurrency & ordering of the queue than to just keep creating more queues for the same work stream.

1oooqooq 3y ago

i thought RH had solved that. just write a systemd unit file with the cpu quotas. bam. done. /s?

mbreese 3y ago

Another place to look for inspiration is the HPC world. Fairshare job queues have been active on multi tenant HPC clusters for decades. Each job typically has a priority value assigned to it based on the submission time, expected resources required, and how much of the cluster the account/user has used recently. Priorities can be continuously updated until the job is released for processing.

deleted 3y ago

[deleted]

osigurdson 3y ago

NATS has multi tenancy built in.

mariya12 3y ago

[flagged]

ceritium 3y ago

For a small side project, instead of queuing all the tasks or writing a custom fair multi-tenant queuing system, I added all the "tasks" to the db, then with a periodic job I query with SKIP LOCKED for pending tasks and enqueue them on my queue system (sidekiq, Goodjob, etc). I only enqueue them if there are less than some amount of jobs pending, it is hardcoded, but I could do it dynamically. The query logic take into consideration the tenants so, a big tenant (or misbehaving) can't block small ones. The periodic job is executed every few second, but it's also enqueued when added new tasks to the db (unless it's already enqueued) or when the query logic detect that there is more jobs available, so the lag is minimal. At the moment I am doing this with only one type of jobs, but I want to expand it to most of the jobs, and let the "query logic" balance the queues.