Honestly it sounds like the main problem here is the scheduler. I'm not saying you should run many massive threadpools, but at the end of the day if you have a

Honestly it sounds like the main problem here is the scheduler. I'm not saying you should run many massive threadpools, but at the end of the day if you have a latency-sensitive service that isn't being given CPU for seconds at a time your scheduler isn't suited for a latency-sensitive service. Bursting is good. You are using resources that would otherwise be idle. It sounds here like the scheduler is punishing the task for the scheduler's mistake. CFS is ensuring that the job gets N cores on average what you actually want is the scheduler to ensure that the job gets N cores minimum. So while having too many threads laying around is slight unnecessary pressure on the scheduler and wasted memory I don't think it should be causing these huge latency spikes.