Not really. It's just different graph structures and optimizations. Spark in general just isn't well suited for tensorflow graphs. You're hitting on exactly w

Not really. It's just different graph structures and optimizations. Spark in general just isn't well suited for tensorflow graphs. You're hitting on exactly what I'm talking about. It's very different use cases. 1 is tensors, the other is data types with schemas. Tensors are also n dimensional. Schemas are inherently 2d. The ops are also different. What makes spark's op graph good vs tensorflows are very different. Tensorflow in general tends to lean on pandas and the like. They are sorta trying to do that with pipelines, but at the end of the day, I think it would overextend tensorflow's scope to "care" about anything outside of math. "ETL" and dataframes is already pretty well served. If you want something that does this well, look at arrow which is a great communications layer. It even has a tensor type now. Disclaimer: Biased rant below. I compete with databricks, google cloud,tensorflow, mxnet and a lot of the spark ecosystem The other problem is..pyspark itself is just bad for linear algebra. Period. You need specialized communications for tensors because the data often lives elsewhere and is in a blob type format. To get benefits from simd and the like you shouldn't be adding networked communications on top of this. That's why we made jumpy for our library. We interop directly at the pointer level and then use our parameter server and gpu stack. Using arrow I think mapd is attempting to do this as well. I don't see spark ever becoming good at linear algebra anytime soon despite what the folks at berkeley are pushing. As for "reasoning about data frames" spark underneath basically turns your dataframe declarations into a DAG. It does this in a very different way than tensorflow's XLA though. XLA's code gen is focused on math ops. Schemas are a whole other different animal. The great thing about spark is it already has "typed rdds" like dataframes and datasets it uses internally. It's really to rely on the JVM's type system and JIT for this. Whereas with math, if you constrain the domain and can make assumptions you can do codegen assuming that all you're doing are "for loops" which generalizes well for cuda and cpu with simd/AVX etc. You can't make the assumption that everything is a "for loop" as much with spark. We write and maintain datavec: https://github.com/deeplearning4j/DataVec https://github.com/deeplearning4j/DataVec and nd4j: https://github.com/deeplearning4j/nd4j https://github.com/deeplearning4j/nd4j which are integrated together and have a "tensor data type" which uses nd4j directly. This is very similar to what arrow does. We define a "transform process" which you could say: do ffts with. That kind of ETL isn't supported with dataframes directly without a lot of hacks. TLDR: TF will mainly stick to linear algebra