TF is mainly "for loops". TF is also missing a whole schema setup. TF is mainly a linear algebra library. I don't see google moving TF towards "schemas". Spar

TF is mainly "for loops". TF is also missing a whole schema setup. TF is mainly a linear algebra library. I don't see google moving TF towards "schemas". Spark (and flink and other competitors to it) are general purpose execution engines. I think what's missing here is the target execution paradigms are different. If you are curious about these things, I would look at how database internals are implemented. Those are a mix of "simd" and "schemas". Generally you have to make storage format assumptions among other things. One of the biggest advantages of spark and flink are "partitions". The DAG engine for spark has this out of the box. Partitions are a lot more generalizable than say: "model parallelism on devices" which is only for floats anyways. There's also a usability factor here. Spark's graphs are a lot more expressive/extensible. It's like I mentioned earlier with py4j vs pyjnius/pointers. 1 is more expressive/general purpose but has overhead. TF is specialized. So if we kill the "TF cuz hype" discussion here and focus on "linear algebra" extending this discussion to pytorch, our library, caffe, etc, there's just not much incentive to "put lipstick on a pig" so to speak. Allowing writing arbitrary code with a type system is by definition not going to help any of these "computation graph linear algebra" libraries.