4 ms·

> we redefine all the words I work for AWS; these are the definitions we use, more or less. https://stackoverflow.com/questions/17384020/what-do-transparent-a

> we redefine all the words I work for AWS; these are the definitions we use, more or less. https://stackoverflow.com/questions/17384020/what-do-transparent-and-opaque-mean-when-applied-to-programming-concepts https://stackoverflow.com/questions/17384020/what-do-transpa... > whether the graph is created at runtime I was specifically referring to the computation graph of the model that is used by autograd. Taken from [1,2]: > Instead, PyTorch uses the operator overloading approach, which builds up a representation of the computed function every time it is executed. > Internally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph. > A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. But what do I know? > libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch. Sure, libtorch is PyTorch; that doesn't mean one can't take the semantics of PyTorch and implement them in just numpy backed python. Doesn't mean it will be fast. [1] PyTorch: An Imperative Style, High-Performance Deep Learning Library [2] Automatic differentiation in PyTorch

mathisfun123 3y ago
>I work for AWS; these are the definitions we use, more or less. you're just being asinine - we're literally talking about binary code that's never seen by anyone that doesn't compile from source and goes digging around in the build dir - how could you possibly call that code "transparent" in any sense of the word? are blob drivers also transparent according to these "AWS" definitions? >I was specifically referring to the computation graph of the model that is used by autograd. it's literally right there in bolded text on the first page of the original paper (the 2017 neurips paper): >Immediate, eager execution. An eager framework runs tensor computations as it encounters them; it avoids ever materializing a “forward graph”, recording only what is necessary to differentiate the computation autograd has absolutely nothing to do with the graph - autograd is literally 10s of thousands of lines of generated, templatized, code that connects edges one op at a time. you can argue with me all you want or you can just go to repo tip and see for yourself https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/templates/TensorBody.h#L519 https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/t...
- PartiallyTyped 3y ago
  > PyTorch (and Chainer) eschew this tape; instead, every intermediate result records only the subset of the computation graph that was relevant to their computation. This means PyTorch users can mix and match independent graphs however they like, in whatever threads they like (without explicit synchronization). An added benefit of structuring graphs this way is that when a portion of the graph becomes dead, it is automatically freed; an important consideration when we want to free large memory chunks as quickly as possible. Either I am an absolute fucking moron or this reads that PyTorch keeps track of a graph. Please explain.
  - mathisfun123 3y ago
    homie it is literally the first sentence in the quoted paragraph >PyTorch (and Chainer) *eschew* this tape; *instead*, every intermediate result records only the *subset* of the computation graph that was relevant to their computation. >Either I am an absolute fucking moron you said it not me
    - PartiallyTyped 3y ago
      Did I ever claim that it created a tape? If my comments implied that, then I am sorry for miscommunicating, but by calling it "dynamic" I meant that there is no compiling of a tape or tracking it as done with TF and the likes. > I nternally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph (they are mutated when an in-place operation occurs; see Section 3.1). A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. Please explain how it doesn't hold a graph when it literally says it is holding one, dynamically created and kept alive via ref counting. I am losing my mind here.

mathisfun123 3y ago

>I work for AWS; these are the definitions we use, more or less. you're just being asinine - we're literally talking about binary code that's never seen by anyone that doesn't compile from source and goes digging around in the build dir - how could you possibly call that code "transparent" in any sense of the word? are blob drivers also transparent according to these "AWS" definitions? >I was specifically referring to the computation graph of the model that is used by autograd. it's literally right there in bolded text on the first page of the original paper (the 2017 neurips paper): >Immediate, eager execution. An eager framework runs tensor computations as it encounters them; it avoids ever materializing a “forward graph”, recording only what is necessary to differentiate the computation autograd has absolutely nothing to do with the graph - autograd is literally 10s of thousands of lines of generated, templatized, code that connects edges one op at a time. you can argue with me all you want or you can just go to repo tip and see for yourself https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/templates/TensorBody.h#L519 https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/t...

PartiallyTyped 3y ago
> PyTorch (and Chainer) eschew this tape; instead, every intermediate result records only the subset of the computation graph that was relevant to their computation. This means PyTorch users can mix and match independent graphs however they like, in whatever threads they like (without explicit synchronization). An added benefit of structuring graphs this way is that when a portion of the graph becomes dead, it is automatically freed; an important consideration when we want to free large memory chunks as quickly as possible. Either I am an absolute fucking moron or this reads that PyTorch keeps track of a graph. Please explain.
- mathisfun123 3y ago
  homie it is literally the first sentence in the quoted paragraph >PyTorch (and Chainer) *eschew* this tape; *instead*, every intermediate result records only the *subset* of the computation graph that was relevant to their computation. >Either I am an absolute fucking moron you said it not me
  - PartiallyTyped 3y ago
    Did I ever claim that it created a tape? If my comments implied that, then I am sorry for miscommunicating, but by calling it "dynamic" I meant that there is no compiling of a tape or tracking it as done with TF and the likes. > I nternally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph (they are mutated when an in-place operation occurs; see Section 3.1). A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. Please explain how it doesn't hold a graph when it literally says it is holding one, dynamically created and kept alive via ref counting. I am losing my mind here.