I used to work on system like this and even built the logic to use preemptible pool effectively just like OP. If I had to design it from scratch today I would u

I used to work on system like this and even built the logic to use preemptible pool effectively just like OP. If I had to design it from scratch today I would use Temporal for job scheduling - their durable compute concept is perfect fit for this and we had a lot of trouble maturing the equivalent scheduling system trying to keep up with rapidly growing scale