If cache eviction and I/O usage are limiting, perhaps a better way to design this system is as a cluster management system that can very quickly change the numb

If cache eviction and I/O usage are limiting, perhaps a better way to design this system is as a cluster management system that can very quickly change the number of cores allocated to foreground vs background tasks (within less than your latency SLO). That means each core is only doing one kind of work for many milliseconds in a row so CPU caches remain hot, and I/O can be sorted into separate low and high priority queues. IO blocking is much less of a concern with NVMe drvies anyway, since they're intrinsically so parallel. Memory usage is still a concern, since you do need to have the code and in-memory data for both workloads loaded on some number of your servers in order to rapidly change what tasks they're doing. Physical machines have so much memory today (compared to the code size or working set of most programs) that I suspect this isn't actually a blocker in practice, and it's not necessarily required for every machine in the cluster to have both workloads resident as long as some of them do. This definitely feels like diving into an alternate universe of cluster management, and I'm not sure how easily Linux gives you tools to do things like this.