5 ms·

Transparent in this case means the average user does not need to interact with the underlying system, it is there, it does stuff, but the user does not see it o

Transparent in this case means the average user does not need to interact with the underlying system, it is there, it does stuff, but the user does not see it or interact directly. What is opaque is jax's tracer objects in the sense that the system exists, is there, the user is aware that it is there, and you can't peek inside it without actively going out of your way. With PyTorch you can implement your own functions with their own forward and backward passes; whether the underlying operations use libtorch is irrelevant, that is simply a backend. The computation graph __is__ created at runtime, whether the scaffolding for this is done in libtorch or directly in python is not relevant.

mathisfun123 3y ago
well i mean if we redefine all the words so they mean whatever we want them to mean then sure blob code that isn't visible even in the source tree can be considered transparent and conversely, perfectly accessible script code (jax's interpreters) can be considered opaque what do i know i'm just a contrib on both projects shrug lots of your other claims are just complete misunderstandings e.g. whether the graph is created at runtime - it's not and you can read adam's original paper to find that out if you don't believe me - and the idea that libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch.
- PartiallyTyped 3y ago
  > we redefine all the words I work for AWS; these are the definitions we use, more or less. https://stackoverflow.com/questions/17384020/what-do-transparent-and-opaque-mean-when-applied-to-programming-concepts https://stackoverflow.com/questions/17384020/what-do-transpa... > whether the graph is created at runtime I was specifically referring to the computation graph of the model that is used by autograd. Taken from [1,2]: > Instead, PyTorch uses the operator overloading approach, which builds up a representation of the computed function every time it is executed. > Internally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph. > A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. But what do I know? > libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch. Sure, libtorch is PyTorch; that doesn't mean one can't take the semantics of PyTorch and implement them in just numpy backed python. Doesn't mean it will be fast. [1] PyTorch: An Imperative Style, High-Performance Deep Learning Library [2] Automatic differentiation in PyTorch
  - mathisfun123 3y ago
    >I work for AWS; these are the definitions we use, more or less. you're just being asinine - we're literally talking about binary code that's never seen by anyone that doesn't compile from source and goes digging around in the build dir - how could you possibly call that code "transparent" in any sense of the word? are blob drivers also transparent according to these "AWS" definitions? >I was specifically referring to the computation graph of the model that is used by autograd. it's literally right there in bolded text on the first page of the original paper (the 2017 neurips paper): >Immediate, eager execution. An eager framework runs tensor computations as it encounters them; it avoids ever materializing a “forward graph”, recording only what is necessary to differentiate the computation autograd has absolutely nothing to do with the graph - autograd is literally 10s of thousands of lines of generated, templatized, code that connects edges one op at a time. you can argue with me all you want or you can just go to repo tip and see for yourself https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/templates/TensorBody.h#L519 https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/t...
    - PartiallyTyped 3y ago
      > PyTorch (and Chainer) eschew this tape; instead, every intermediate result records only the subset of the computation graph that was relevant to their computation. This means PyTorch users can mix and match independent graphs however they like, in whatever threads they like (without explicit synchronization). An added benefit of structuring graphs this way is that when a portion of the graph becomes dead, it is automatically freed; an important consideration when we want to free large memory chunks as quickly as possible. Either I am an absolute fucking moron or this reads that PyTorch keeps track of a graph. Please explain.
      - mathisfun123 3y ago
        homie it is literally the first sentence in the quoted paragraph >PyTorch (and Chainer) *eschew* this tape; *instead*, every intermediate result records only the *subset* of the computation graph that was relevant to their computation. >Either I am an absolute fucking moron you said it not me

mathisfun123 3y ago

well i mean if we redefine all the words so they mean whatever we want them to mean then sure blob code that isn't visible even in the source tree can be considered transparent and conversely, perfectly accessible script code (jax's interpreters) can be considered opaque what do i know i'm just a contrib on both projects shrug lots of your other claims are just complete misunderstandings e.g. whether the graph is created at runtime - it's not and you can read adam's original paper to find that out if you don't believe me - and the idea that libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch.

PartiallyTyped 3y ago
> we redefine all the words I work for AWS; these are the definitions we use, more or less. https://stackoverflow.com/questions/17384020/what-do-transparent-and-opaque-mean-when-applied-to-programming-concepts https://stackoverflow.com/questions/17384020/what-do-transpa... > whether the graph is created at runtime I was specifically referring to the computation graph of the model that is used by autograd. Taken from [1,2]: > Instead, PyTorch uses the operator overloading approach, which builds up a representation of the computed function every time it is executed. > Internally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph. > A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. But what do I know? > libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch. Sure, libtorch is PyTorch; that doesn't mean one can't take the semantics of PyTorch and implement them in just numpy backed python. Doesn't mean it will be fast. [1] PyTorch: An Imperative Style, High-Performance Deep Learning Library [2] Automatic differentiation in PyTorch
- mathisfun123 3y ago
  >I work for AWS; these are the definitions we use, more or less. you're just being asinine - we're literally talking about binary code that's never seen by anyone that doesn't compile from source and goes digging around in the build dir - how could you possibly call that code "transparent" in any sense of the word? are blob drivers also transparent according to these "AWS" definitions? >I was specifically referring to the computation graph of the model that is used by autograd. it's literally right there in bolded text on the first page of the original paper (the 2017 neurips paper): >Immediate, eager execution. An eager framework runs tensor computations as it encounters them; it avoids ever materializing a “forward graph”, recording only what is necessary to differentiate the computation autograd has absolutely nothing to do with the graph - autograd is literally 10s of thousands of lines of generated, templatized, code that connects edges one op at a time. you can argue with me all you want or you can just go to repo tip and see for yourself https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/templates/TensorBody.h#L519 https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/t...
  - PartiallyTyped 3y ago
    > PyTorch (and Chainer) eschew this tape; instead, every intermediate result records only the subset of the computation graph that was relevant to their computation. This means PyTorch users can mix and match independent graphs however they like, in whatever threads they like (without explicit synchronization). An added benefit of structuring graphs this way is that when a portion of the graph becomes dead, it is automatically freed; an important consideration when we want to free large memory chunks as quickly as possible. Either I am an absolute fucking moron or this reads that PyTorch keeps track of a graph. Please explain.
    - mathisfun123 3y ago
      homie it is literally the first sentence in the quoted paragraph >PyTorch (and Chainer) *eschew* this tape; *instead*, every intermediate result records only the *subset* of the computation graph that was relevant to their computation. >Either I am an absolute fucking moron you said it not me