From what I've seen, there are sort of two paths if you're running on your own hardware or VMs. I'll provide a well known example from each. 1. lang specific

From what I've seen, there are sort of two paths if you're running on your own hardware or VMs. I'll provide a well known example from each. 1. lang specific distributed task library For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config. * https://github.com/celery/celery https://github.com/celery/celery Or lower level: * https://github.com/dask/dask https://github.com/dask/dask 2. DAG Workflow systems There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow: * https://github.com/apache/airflow https://github.com/apache/airflow