3 ms·

I first learned TensorFlow and built a lot of very custom stuff with its "Slim" API and have been very happy about it. The only thing I always struggle with is

I first learned TensorFlow and built a lot of very custom stuff with its "Slim" API and have been very happy about it. The only thing I always struggle with is the grunt work of wrangling tensor shapes and indices, e.g. broadcasting, tf.tile, tf.gather, tf.scatter_nd, tf.where etc. These are a real headache to use, while numpy for example has its so called "fancy indexing" and mask-based indexing, which are not available in TF. Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this. Maybe someone who just uses PyTorch doesn't miss this because they never used it. My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function. That's not the best design in terms of DRY. I understand why it's technically needed due to the working principles of PyTorch (especially for weight sharing in different parts of the model, which in TF may need some getting used with its variable scopes and reuse flags), it's just inconvenient.

cs702 7y ago
> Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this. That's one of the things we realized we disliked most about TensorFlow: Whenever we tried to do anything that was out-of-the-box, we would have to write Python code that would then construct the TensorFlow graph that would actually specify and eventually run the computations. It felt like metaprogramming from a nice, general-purpose language (Python) in an inflexible, non-general purpose one (TF ops). > My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function. That is not true. You can create graphs, compute losses, and backpropagate them on the fly, because autograd records all operations on the fly.[a] The nn.Module class is a convenient, nice, Pythonic wrapper with lots of functionality, but you don't have to use it. You can create your own wrapper if you'd like. [a] https://pytorch.org/docs/stable/notes/autograd.html https://pytorch.org/docs/stable/notes/autograd.html
- bonoboTP 7y ago
  > You can create graphs, compute losses, and backpropagate them on the fly Sure, but you'd have to define the weight variables for your conv layer first outside the train loop, then use them within the train loop. With TF I can just create the weights and the ops in one line. Also, constantly doing .cuda() and .cpu() is not much nicer than feed dicts and fetches. The real advantage of pytorch is the debuggability, you can set a breakpoint and see every value.
  - briggers 7y ago
    I think I understand your point about declaring and then later using your layers. Are you aware of the Sequential module? It allows you to chain together layers into a single variable, making this repetition disappear into a single forward/__call__ on the Sequential.
    - bonoboTP 7y ago
      Thanks, that does improve it for simple cases of pure chains of layers without any skip connections etc. Unfortunately it's most an issue with more complicated architectures. When testing different designs I often forget to modify it in both places and sure it's not a huge deal, but having one place or two does feel like a qualitative difference in cognitive burden. For example if I introduce a bool flag which decides whether I do a particular alternative design, I have to chech the flag and do an if/else in two places: once to create the layers, once to use them.

cs702 7y ago

> Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this. That's one of the things we realized we disliked most about TensorFlow: Whenever we tried to do anything that was out-of-the-box, we would have to write Python code that would then construct the TensorFlow graph that would actually specify and eventually run the computations. It felt like metaprogramming from a nice, general-purpose language (Python) in an inflexible, non-general purpose one (TF ops). > My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function. That is not true. You can create graphs, compute losses, and backpropagate them on the fly, because autograd records all operations on the fly.[a] The nn.Module class is a convenient, nice, Pythonic wrapper with lots of functionality, but you don't have to use it. You can create your own wrapper if you'd like. [a] https://pytorch.org/docs/stable/notes/autograd.html https://pytorch.org/docs/stable/notes/autograd.html

bonoboTP 7y ago
> You can create graphs, compute losses, and backpropagate them on the fly Sure, but you'd have to define the weight variables for your conv layer first outside the train loop, then use them within the train loop. With TF I can just create the weights and the ops in one line. Also, constantly doing .cuda() and .cpu() is not much nicer than feed dicts and fetches. The real advantage of pytorch is the debuggability, you can set a breakpoint and see every value.
- briggers 7y ago
  I think I understand your point about declaring and then later using your layers. Are you aware of the Sequential module? It allows you to chain together layers into a single variable, making this repetition disappear into a single forward/__call__ on the Sequential.
  - bonoboTP 7y ago
    Thanks, that does improve it for simple cases of pure chains of layers without any skip connections etc. Unfortunately it's most an issue with more complicated architectures. When testing different designs I often forget to modify it in both places and sure it's not a huge deal, but having one place or two does feel like a qualitative difference in cognitive burden. For example if I introduce a bool flag which decides whether I do a particular alternative design, I have to chech the flag and do an if/else in two places: once to create the layers, once to use them.