3 ms·

Queues should be measured in wallclock time, not in terms of items. That removes the problem you mention.

brightball 10y ago
The only other consideration is stress of jobs on peripheral resources like the database. If you have multiple jobs that may be intense enough to insist on sequential execution it could inflate a wall clock number even if scaling wouldn't matter. You essentially need a metric for wallclock time exclusive to jobs that benefit from scaling.
jcrites 10y ago
That's a good point. One of the properties I typically instrument in queue processing applications is the time delay from when messages were inserted into a queue, until when they have completed being fully processed. The application emits message processing latency into the instrumentation system for monitoring and examination. I haven't ever tried using this for purposes of auto-scaling, though. I'll have to give it a try sometime. I've had the most success auto-scaling systems by measuring properties of system processing capacity, like percentage of threads in use, or percentage of maximum connections, or CPU; things like this that are properties of the system, rather than the work it's processing. These all have the desirable property that (i) they're relatively work-type agnostic (ii) you can begin scaling up when a utilization threshold is crossed, which is before there's ever an observable impact (like messages have been delayed an undesirable amount). Scaling on percent of thread capacity means that you can discriminate when to scale up even while your queue size is zero and your message latency is approximately zero (which means there has not yet been an impact on the work you're handling, such as a delay). That being said, there's also a flip side: once utilization is at 100%, you can't tell how far behind you are, or how much to scale up. To combat this one establishes scaling rules that are aimed to prevent utilization reaching 100%, which is feasible if your mean time to bring new capacity online is low enough that you can follow the demand curve. This is also where you could bring in wall clock time as a measurement of how far behind the system is, and how much it needs to scale up to return to meeting its SLA. (I should clarify that most of my automatic scaling experience is in the context of soft real-time systems with reasonably tight SLAs, not large batch systems that can afford to run large backlogs and that prefer to be at 100% utilization for efficiency.)

The only other consideration is stress of jobs on peripheral resources like the database. If you have multiple jobs that may be intense enough to insist on sequential execution it could inflate a wall clock number even if scaling wouldn't matter. You essentially need a metric for wallclock time exclusive to jobs that benefit from scaling.

jcrites 10y ago

That's a good point. One of the properties I typically instrument in queue processing applications is the time delay from when messages were inserted into a queue, until when they have completed being fully processed. The application emits message processing latency into the instrumentation system for monitoring and examination. I haven't ever tried using this for purposes of auto-scaling, though. I'll have to give it a try sometime. I've had the most success auto-scaling systems by measuring properties of system processing capacity, like percentage of threads in use, or percentage of maximum connections, or CPU; things like this that are properties of the system, rather than the work it's processing. These all have the desirable property that (i) they're relatively work-type agnostic (ii) you can begin scaling up when a utilization threshold is crossed, which is before there's ever an observable impact (like messages have been delayed an undesirable amount). Scaling on percent of thread capacity means that you can discriminate when to scale up even while your queue size is zero and your message latency is approximately zero (which means there has not yet been an impact on the work you're handling, such as a delay). That being said, there's also a flip side: once utilization is at 100%, you can't tell how far behind you are, or how much to scale up. To combat this one establishes scaling rules that are aimed to prevent utilization reaching 100%, which is feasible if your mean time to bring new capacity online is low enough that you can follow the demand curve. This is also where you could bring in wall clock time as a measurement of how far behind the system is, and how much it needs to scale up to return to meeting its SLA. (I should clarify that most of my automatic scaling experience is in the context of soft real-time systems with reasonably tight SLAs, not large batch systems that can afford to run large backlogs and that prefer to be at 100% utilization for efficiency.)