6 ms·

> PyTorch (and Chainer) eschew this tape; instead, every intermediate result records only the subset of the computation graph that was relevant to their computa

> PyTorch (and Chainer) eschew this tape; instead, every intermediate result records only the subset of the computation graph that was relevant to their computation. This means PyTorch users can mix and match independent graphs however they like, in whatever threads they like (without explicit synchronization). An added benefit of structuring graphs this way is that when a portion of the graph becomes dead, it is automatically freed; an important consideration when we want to free large memory chunks as quickly as possible. Either I am an absolute fucking moron or this reads that PyTorch keeps track of a graph. Please explain.

mathisfun123 3y ago
homie it is literally the first sentence in the quoted paragraph >PyTorch (and Chainer) *eschew* this tape; *instead*, every intermediate result records only the *subset* of the computation graph that was relevant to their computation. >Either I am an absolute fucking moron you said it not me
- PartiallyTyped 3y ago
  Did I ever claim that it created a tape? If my comments implied that, then I am sorry for miscommunicating, but by calling it "dynamic" I meant that there is no compiling of a tape or tracking it as done with TF and the likes. > I nternally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph (they are mutated when an in-place operation occurs; see Section 3.1). A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. Please explain how it doesn't hold a graph when it literally says it is holding one, dynamically created and kept alive via ref counting. I am losing my mind here.

mathisfun123 3y ago

homie it is literally the first sentence in the quoted paragraph >PyTorch (and Chainer) *eschew* this tape; *instead*, every intermediate result records only the *subset* of the computation graph that was relevant to their computation. >Either I am an absolute fucking moron you said it not me

PartiallyTyped 3y ago
Did I ever claim that it created a tape? If my comments implied that, then I am sorry for miscommunicating, but by calling it "dynamic" I meant that there is no compiling of a tape or tracking it as done with TF and the likes. > I nternally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph (they are mutated when an in-place operation occurs; see Section 3.1). A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. Please explain how it doesn't hold a graph when it literally says it is holding one, dynamically created and kept alive via ref counting. I am losing my mind here.