3 ms·
It's helpful in other domains, too. For example, in ETL pipelines, I would greatly prefer to have an entire DAG go down quickly and noisily than to risk having
by mumblemumble 7y ago
It's helpful in other domains, too.
For example, in ETL pipelines, I would greatly prefer to have an entire DAG go down quickly and noisily than to risk having it generate incorrect data. It's the difference between a crappy morning, and a crappy day or even a crappy week.
- alexchantavy 7y agoJust curious since a lot of my work is on an ETL - what are your favorite DAG libraries/approaches? Some of my ETL workflows can run in parallel with each other because there aren't data dependencies between them, and others should definitely crash loudly
- rozap 7y agoOr ETL approach is all based on elixir (erlang) and it has some drawbacks, but ease of debugging is not one of them. No libraries, just code.
- mumblemumble 7y agoWe use a home-grown solution, and try to keep it pretty minimal. For example, we can run all our pipeline steps serially and still get all the work done plenty fast, so, for now, we're keeping that parallelism can of worms firmly shut.
- tonyarkles 7y ago> a crappy day or even a crappy week Or a crappy discovery at the end of the quarter...
- dllthomas 7y agoOr crappy decisions.