4 ms·
> Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (sessi
by cs702 7y ago
> Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this.
That's one of the things we realized we disliked most about TensorFlow: Whenever we tried to do anything that was out-of-the-box, we would have to write Python code that would then construct the TensorFlow graph that would actually specify and eventually run the computations. It felt like metaprogramming from a nice, general-purpose language (Python) in an inflexible, non-general purpose one (TF ops).
> My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function.
That is not true. You can create graphs, compute losses, and backpropagate them on the fly, because autograd records all operations on the fly.[a] The nn.Module class is a convenient, nice, Pythonic wrapper with lots of functionality, but you don't have to use it. You can create your own wrapper if you'd like.
[a] https://pytorch.org/docs/stable/notes/autograd.html https://pytorch.org/docs/stable/notes/autograd.html
- bonoboTP 7y ago> You can create graphs, compute losses, and backpropagate them on the fly Sure, but you'd have to define the weight variables for your conv layer first outside the train loop, then use them within the train loop. With TF I can just create the weights and the ops in one line. Also, constantly doing .cuda() and .cpu() is not much nicer than feed dicts and fetches. The real advantage of pytorch is the debuggability, you can set a breakpoint and see every value.
- briggers 7y agoI think I understand your point about declaring and then later using your layers. Are you aware of the Sequential module? It allows you to chain together layers into a single variable, making this repetition disappear into a single forward/__call__ on the Sequential.
- bonoboTP 7y agoThanks, that does improve it for simple cases of pure chains of layers without any skip connections etc. Unfortunately it's most an issue with more complicated architectures. When testing different designs I often forget to modify it in both places and sure it's not a huge deal, but having one place or two does feel like a qualitative difference in cognitive burden. For example if I introduce a bool flag which decides whether I do a particular alternative design, I have to chech the flag and do an if/else in two places: once to create the layers, once to use them.