When I needed to solve this problem, I spun up 4x as many workers as I needed and managed them through supervisor. This handles two of the requirements: 1. Auto

When I needed to solve this problem, I spun up 4x as many workers as I needed and managed them through supervisor. This handles two of the requirements: 1. Autoscaling - Unless your workers are very expensive to run, or your load is exceptionally spiky (e.g. 100x jumps), just pre-scale instead of autoscaling. 2. If a process dies, supervisor (or runit or whatever) will restart it. My supervisor config has each worker log to its own file (e.g. /var/log/acmeinc/worker_01.log). For death detection, I'd probably use a watchdog process that checks the mtime of each log file. If it's more than X minutes old, then no work has been done in that time, so kill the process and allow supervisor to restart it. Combine this with an in-process watchdog log message ("hey, I'm still alive!") on the main consumer thread every X minutes, and you have a fairly robust solution. Tangentially, deferring engineering work by buying more hardware is almost always a good idea. :)