3 ms·
Have you tried dask? It already has dataflow programming built in for numpy arrays, out of core and custom structures. http://dask.pydata.org/en/latest/ http:
by tanlermin 11y ago
Have you tried dask? It already has dataflow programming built in for numpy arrays, out of core and custom structures.
http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/
- samuell 11y agoInteresting. Do you have a link to description of their "dataflow" implementation or API? From what I can see in the docs I'm getting afraid Dask does the same mistake as so many other recent tools: Allow only task dependencies, while what is needed in general scientific workflows is data dependencies (connect outports and inports of process). I have explained this difference in a blog post earlier: http://bionics.it/posts/workflows-dataflow-not-task-deps http://bionics.it/posts/workflows-dataflow-not-task-deps (UPDATE: in all fairness, they seem to be doing something in-between, a little like Snakemake, in that they allow to specify data inputs based on a naming scheme. What we want is a totally naming scheme independent declarative way of explicitly connecting one output to one input, as that is the most generic and pluggable way you could do it.) If they allow true data dependencies though, that would be very interesting.
- tanlermin 11y agoVery interesting I didn't realize the difference. How about this? https://github.com/shashi/ComputeFramework.jl https://github.com/shashi/ComputeFramework.jl But I suspect it has the same problem. Edit: There is also this https://github.com/JuliaDB/DataStreams.jl https://github.com/JuliaDB/DataStreams.jl
- tanlermin 11y agoOkay, check this out: DataFlow programming for Julia https://github.com/MikeInnes/Flow.jl https://github.com/MikeInnes/Flow.jl