3 ms·

This is all very interesting, but one thing that occurs to me is: why are there so many idle pods? Is there any way to fold the work that is currently being don

This is all very interesting, but one thing that occurs to me is: why are there so many idle pods? Is there any way to fold the work that is currently being done in multiple different pods into one pod? Perhaps via multiple threads, or even just plain single-threaded concurrency? Unless there is some special tenancy requirement, that might be the most efficient way to deal with this situation.

tannhaeuser 7y ago
The idle pots are so that your cloud provider maxes out its profit.
zbentley 7y ago
The author alluded to this by referencing task queues. It's not uncommon to have task queue workers listening on "bursty" queues which are usually quiet, with the constraint that when work arrives in the queue it should be picked up really quickly (I.e. no waiting for program, pod/container, or hardware startup). If you have multiple task queues and sets of workers for different kinds of work (I don't know if the author does, but it is definitely a common pattern), then you can easily end up with a decent number of idle pods sitting around. This isn't unique to k8s either; all sorts of queue-worker-oriented deployments have this issue. The wasted idle capacity can be mitigated by having separate "burst" capacity (idle workers sitting around waiting for work) and "post-burst" capacity (a bunch of new workers that get created in response to a detected backlog on a queue), but orchestrating that is complicated: how much of a backlog merits the need for post-burst workers to begin starting? Instead, can the normal burst workers pay it down fast enough that no new instances/pods need to be scheduled? Do your post-burst workers always start up at the same rate (hint: as your application/dependency stack grows, they start slower and slower)? How do you define SLOs/SLAs for a service with a two-tiered scale-out behavior like this (some folks are content with just a max time-to-processed SLA based on how long it takes a post-burst worker to come online and consume the oldest message on the queue at a given point in time, other workloads have more demanding requirements for the worst/average cases)? In many cases, just keeping the peak-scale amount of idle workers sitting around waiting for queue bursts is cheaper (from an operational/engineering time perspective) than building something that satisfactorily answers those questions.

tannhaeuser 7y ago

The idle pots are so that your cloud provider maxes out its profit.

zbentley 7y ago

The author alluded to this by referencing task queues. It's not uncommon to have task queue workers listening on "bursty" queues which are usually quiet, with the constraint that when work arrives in the queue it should be picked up really quickly (I.e. no waiting for program, pod/container, or hardware startup). If you have multiple task queues and sets of workers for different kinds of work (I don't know if the author does, but it is definitely a common pattern), then you can easily end up with a decent number of idle pods sitting around. This isn't unique to k8s either; all sorts of queue-worker-oriented deployments have this issue. The wasted idle capacity can be mitigated by having separate "burst" capacity (idle workers sitting around waiting for work) and "post-burst" capacity (a bunch of new workers that get created in response to a detected backlog on a queue), but orchestrating that is complicated: how much of a backlog merits the need for post-burst workers to begin starting? Instead, can the normal burst workers pay it down fast enough that no new instances/pods need to be scheduled? Do your post-burst workers always start up at the same rate (hint: as your application/dependency stack grows, they start slower and slower)? How do you define SLOs/SLAs for a service with a two-tiered scale-out behavior like this (some folks are content with just a max time-to-processed SLA based on how long it takes a post-burst worker to come online and consume the oldest message on the queue at a given point in time, other workloads have more demanding requirements for the worst/average cases)? In many cases, just keeping the peak-scale amount of idle workers sitting around waiting for queue bursts is cheaper (from an operational/engineering time perspective) than building something that satisfactorily answers those questions.