6 ms·

I feel like I can see the fingerprints of pytorch on the direction TF leadership is choosing to go. "Eager execution" and "easier to learn and apply" are where

I feel like I can see the fingerprints of pytorch on the direction TF leadership is choosing to go. "Eager execution" and "easier to learn and apply" are where pytorch steals market share from TF, resulting in moving towards a more "dynamic" graph model.

minimaxir 8y ago
Eager Execution was added to TensorFlow a couple versions ago. https://www.tensorflow.org/guide/eager https://www.tensorflow.org/guide/eager
- mi_lk 8y ago
  And the parent's point still stands.
cdavid 8y ago
As someone joked to me, Google has too many compiler people, and this may have explained some design choices. While I am sure pytorch competition contributed to this change, having a graph-based API was a terrible idea whose time would have passed anyway I think. It made some many things complicated for users. As an implementation/optimization strategy, using a graph was a reasonable choice, but to let drive the API as it did was indeed terrible. It is reassuring to see the TF team addressing the main flaws of the library: removal of graph-based API, and fixing the lack of consistency/plethora of APIs attacking the same problems.
nonbel 8y ago
Isn't "easier to learn and apply" usually code for "slower and more difficult to optimize"?
- h4b4n3r0 8y ago
  Not in the case of PyTorch. Massively easier to deal with _and_ faster for just about everything.
pacala 8y ago
The pytorch graph model is not as dynamic as you think it is, because batches. All batch elements must be processed by the exact same code trace, limiting per element dynamism. The dynamic aspect of pytorch is an implementation detail for delivering autograd over a trace of tensor operations. What pytorch does very well is to play nice python/numpy semantics, especially via broadcasting[0]. There is very little cognitive overhead between a pytorch program and its equivalent python/numpy representation. What tensorflow does very well is to execute computational graphs on a wide variety of backends. One gpu, many gpus, distributed gpus, servers, mobile, browsers. The recently announced autograph[1] merges the clarity of python/numpy coding style with tensorflow execution engine, offering autograd via compile-time abstract interpretation over all possible traces. [0] https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html https://docs.scipy.org/doc/numpy/user/basics.broadcasting.ht... [1] https://www.tensorflow.org/guide/autograph https://www.tensorflow.org/guide/autograph
- h4b4n3r0 8y ago
  Multi GPU in PyTorch is usually just a few lines of code. Can’t be any easier. Wrap model in DataParallel and enjoy life. Last time I touched TF, things were pretty gnarly for this frequent use case. Don’t know how it is now.

minimaxir 8y ago

Eager Execution was added to TensorFlow a couple versions ago. https://www.tensorflow.org/guide/eager https://www.tensorflow.org/guide/eager

mi_lk 8y ago
And the parent's point still stands.

cdavid 8y ago

As someone joked to me, Google has too many compiler people, and this may have explained some design choices. While I am sure pytorch competition contributed to this change, having a graph-based API was a terrible idea whose time would have passed anyway I think. It made some many things complicated for users. As an implementation/optimization strategy, using a graph was a reasonable choice, but to let drive the API as it did was indeed terrible. It is reassuring to see the TF team addressing the main flaws of the library: removal of graph-based API, and fixing the lack of consistency/plethora of APIs attacking the same problems.

nonbel 8y ago

Isn't "easier to learn and apply" usually code for "slower and more difficult to optimize"?

h4b4n3r0 8y ago
Not in the case of PyTorch. Massively easier to deal with _and_ faster for just about everything.

pacala 8y ago

The pytorch graph model is not as dynamic as you think it is, because batches. All batch elements must be processed by the exact same code trace, limiting per element dynamism. The dynamic aspect of pytorch is an implementation detail for delivering autograd over a trace of tensor operations. What pytorch does very well is to play nice python/numpy semantics, especially via broadcasting[0]. There is very little cognitive overhead between a pytorch program and its equivalent python/numpy representation. What tensorflow does very well is to execute computational graphs on a wide variety of backends. One gpu, many gpus, distributed gpus, servers, mobile, browsers. The recently announced autograph[1] merges the clarity of python/numpy coding style with tensorflow execution engine, offering autograd via compile-time abstract interpretation over all possible traces. [0] https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html https://docs.scipy.org/doc/numpy/user/basics.broadcasting.ht... [1] https://www.tensorflow.org/guide/autograph https://www.tensorflow.org/guide/autograph

h4b4n3r0 8y ago
Multi GPU in PyTorch is usually just a few lines of code. Can’t be any easier. Wrap model in DataParallel and enjoy life. Last time I touched TF, things were pretty gnarly for this frequent use case. Don’t know how it is now.