4 ms·
Slight tangent: is anyone aware of a library (in Python or JS… or C) for specifying DAG pipielines that supports caching results (based on input)? Make is clos
by schemescape 4y ago
Slight tangent: is anyone aware of a library (in Python or JS… or C) for specifying DAG pipielines that supports caching results (based on input)?
Make is closest I’ve found, but there’s probably something better out there.
Edit: I should look at other build tools, but ideally I want a library.
Edit again: I will be running this locally. Forgot to mention that.
- tjfl 4y agoMaybe Dagster with Assets? https://docs.dagster.io/concepts/assets/software-defined-assets https://docs.dagster.io/concepts/assets/software-defined-ass...
- deleted 4y ago[deleted]
- mstokholm 4y agoalkymi (https://github.com/MathiasStokholm/alkymi https://github.com/MathiasStokholm/alkymi) is such a library (disclaimer: I'm the author). I built alkymi to help me cache complex data pipelines, while still having all the niceties of Python.
- schemescape 4y agoI haven’t dug too deep yet, but this sounds exactly like what I was looking for, thanks!
- brahbrah 4y agofn_graph: https://github.com/BusinessOptics/fn_graph https://github.com/BusinessOptics/fn_graph (My favorite one ^) pyungo: https://github.com/cedricleroy/pyungo https://github.com/cedricleroy/pyungo Loman: https://github.com/janushendersonassetallocation/loman https://github.com/janushendersonassetallocation/loman Tributary: https://github.com/timkpaine/tributary https://github.com/timkpaine/tributary