4 ms·

Yep, but this adds complexity and fear of maintenance problems, depending on how "close to the core" the library is.

by samuell 11y ago

samuell 11y ago
For example, I looked into implementing this with Python 3.5's coroutines and async/await syntax, but this seems to add an enourmous amount of complexity. For example you need specialized versions of many of the standard library methods, just to make them usable in the async setting. In either case I couldn't get my head around how to implement this. In Go, the implementation is conceptually extremely simple (although the code might not always be the most readable).
- tanlermin 11y ago
  Have you tried dask? It already has dataflow programming built in for numpy arrays, out of core and custom structures. http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/
  - samuell 11y ago
    Interesting. Do you have a link to description of their "dataflow" implementation or API? From what I can see in the docs I'm getting afraid Dask does the same mistake as so many other recent tools: Allow only task dependencies, while what is needed in general scientific workflows is data dependencies (connect outports and inports of process). I have explained this difference in a blog post earlier: http://bionics.it/posts/workflows-dataflow-not-task-deps http://bionics.it/posts/workflows-dataflow-not-task-deps (UPDATE: in all fairness, they seem to be doing something in-between, a little like Snakemake, in that they allow to specify data inputs based on a naming scheme. What we want is a totally naming scheme independent declarative way of explicitly connecting one output to one input, as that is the most generic and pluggable way you could do it.) If they allow true data dependencies though, that would be very interesting.
    - tanlermin 11y ago
      Very interesting I didn't realize the difference. How about this? https://github.com/shashi/ComputeFramework.jl https://github.com/shashi/ComputeFramework.jl But I suspect it has the same problem. Edit: There is also this https://github.com/JuliaDB/DataStreams.jl https://github.com/JuliaDB/DataStreams.jl
    - tanlermin 11y ago
      Okay, check this out: DataFlow programming for Julia https://github.com/MikeInnes/Flow.jl https://github.com/MikeInnes/Flow.jl

Yep, but this adds complexity and fear of maintenance problems, depending on how "close to the core" the library is.

by samuell 11y ago

Yep, but this adds complexity and fear of maintenance problems, depending on how "close to the core" the library is.

For example, I looked into implementing this with Python 3.5's coroutines and async/await syntax, but this seems to add an enourmous amount of complexity. For example you need specialized versions of many of the standard library methods, just to make them usable in the async setting. In either case I couldn't get my head around how to implement this. In Go, the implementation is conceptually extremely simple (although the code might not always be the most readable).

tanlermin 11y ago
Have you tried dask? It already has dataflow programming built in for numpy arrays, out of core and custom structures. http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/
- samuell 11y ago
  Interesting. Do you have a link to description of their "dataflow" implementation or API? From what I can see in the docs I'm getting afraid Dask does the same mistake as so many other recent tools: Allow only task dependencies, while what is needed in general scientific workflows is data dependencies (connect outports and inports of process). I have explained this difference in a blog post earlier: http://bionics.it/posts/workflows-dataflow-not-task-deps http://bionics.it/posts/workflows-dataflow-not-task-deps (UPDATE: in all fairness, they seem to be doing something in-between, a little like Snakemake, in that they allow to specify data inputs based on a naming scheme. What we want is a totally naming scheme independent declarative way of explicitly connecting one output to one input, as that is the most generic and pluggable way you could do it.) If they allow true data dependencies though, that would be very interesting.
  - tanlermin 11y ago
    Very interesting I didn't realize the difference. How about this? https://github.com/shashi/ComputeFramework.jl https://github.com/shashi/ComputeFramework.jl But I suspect it has the same problem. Edit: There is also this https://github.com/JuliaDB/DataStreams.jl https://github.com/JuliaDB/DataStreams.jl
  - tanlermin 11y ago
    Okay, check this out: DataFlow programming for Julia https://github.com/MikeInnes/Flow.jl https://github.com/MikeInnes/Flow.jl