4 ms·

that's interesting - but you have mentioned that TF does a better job than spark at parameter servers (and grpc, etc) So instead of pushing spark towards linea

that's interesting - but you have mentioned that TF does a better job than spark at parameter servers (and grpc, etc) So instead of pushing spark towards linear algebra, isnt it easier to push TF towards schemas ? Or are the architectures so different that they will live separate lives

JoeriHer 9y ago
Depends on how you implement the Parameter Server. You could, for example, fork a Python process and implement your own parameter server. No Spark required.
agibsonccc 9y ago
TF is mainly "for loops". TF is also missing a whole schema setup. TF is mainly a linear algebra library. I don't see google moving TF towards "schemas". Spark (and flink and other competitors to it) are general purpose execution engines. I think what's missing here is the target execution paradigms are different. If you are curious about these things, I would look at how database internals are implemented. Those are a mix of "simd" and "schemas". Generally you have to make storage format assumptions among other things. One of the biggest advantages of spark and flink are "partitions". The DAG engine for spark has this out of the box. Partitions are a lot more generalizable than say: "model parallelism on devices" which is only for floats anyways. There's also a usability factor here. Spark's graphs are a lot more expressive/extensible. It's like I mentioned earlier with py4j vs pyjnius/pointers. 1 is more expressive/general purpose but has overhead. TF is specialized. So if we kill the "TF cuz hype" discussion here and focus on "linear algebra" extending this discussion to pytorch, our library, caffe, etc, there's just not much incentive to "put lipstick on a pig" so to speak. Allowing writing arbitrary code with a type system is by definition not going to help any of these "computation graph linear algebra" libraries.

JoeriHer 9y ago

Depends on how you implement the Parameter Server. You could, for example, fork a Python process and implement your own parameter server. No Spark required.

agibsonccc 9y ago

TF is mainly "for loops". TF is also missing a whole schema setup. TF is mainly a linear algebra library. I don't see google moving TF towards "schemas". Spark (and flink and other competitors to it) are general purpose execution engines. I think what's missing here is the target execution paradigms are different. If you are curious about these things, I would look at how database internals are implemented. Those are a mix of "simd" and "schemas". Generally you have to make storage format assumptions among other things. One of the biggest advantages of spark and flink are "partitions". The DAG engine for spark has this out of the box. Partitions are a lot more generalizable than say: "model parallelism on devices" which is only for floats anyways. There's also a usability factor here. Spark's graphs are a lot more expressive/extensible. It's like I mentioned earlier with py4j vs pyjnius/pointers. 1 is more expressive/general purpose but has overhead. TF is specialized. So if we kill the "TF cuz hype" discussion here and focus on "linear algebra" extending this discussion to pytorch, our library, caffe, etc, there's just not much incentive to "put lipstick on a pig" so to speak. Allowing writing arbitrary code with a type system is by definition not going to help any of these "computation graph linear algebra" libraries.