3 ms·
I work on a workflow engine (aka DAG processor) that dates to before 2008. We use it for many things in an astronomy experiment. It has executed over 900M batch
by batbomb 10y ago
I work on a workflow engine (aka DAG processor) that dates to before 2008. We use it for many things in an astronomy experiment. It has executed over 900M batch jobs and 200M "scriptlets", and most of those workflows can be re-executed at any stage, even today, five+ years after they were originally ran, to recreate data products. The workflow engine also serves effectively as a provenance database for each data product to trace its origins back to raw data.