4 ms·
> Logical operator graphs for graph dataflow-style execution (think TensorFlow or PyTorch, but for data frames) > A multicore schedular for parallel evaluation
by anentropic 9y ago
> Logical operator graphs for graph dataflow-style execution (think TensorFlow or PyTorch, but for data frames)
> A multicore schedular for parallel evaluation of operator graphs
Does anything like this already exist somewhere?
- deleted 9y ago[deleted]
- andrioni 9y agoSpark SQL does exactly that: declarative graph dataflow computations + a multicore (also distributed) scheduler and executor, although it doesn't really scale down that well to a single machine and the execution layer happens mostly on the JVM, even when using PySpark (with an ungodly mix of of serialized Python and JVM objects and code). Ibis (also by Wes McKinney) does the first part, but it offloads scheduling and execution to the underlying database you are using.
- almostkorean 9y agoWe are using Airflow for this
- RHSman2 9y agoWe use a luigi type app for this.