This is unworkable if you are actively scaling your system. Am i supposed to calculate ideal queue size with each scale out of my data platform? Instead, the r

This is unworkable if you are actively scaling your system. Am i supposed to calculate ideal queue size with each scale out of my data platform? Instead, the right way to think about limiting queue size is load shedding when you feel back pressure. Here’s an example at Cronitor: if our main sqs ingestion queue backs up, our pipeline will automatically move from streaming to micro-batching, drastically reducing the number of messages on the queue at the expense of slightly increased latency per message. At the same time, a less critical piece of our infrastructure pauses itself until the queue is healthy again, shedding one of the largest sources of db load and giving that capacity to ingestion. To me the goal is to feel and respond to back pressure before blowing up and rejecting messages.