4 ms·

I really don't feel that there is magic in PyTorch or jax, but that may be because I have written my own autograd libs. In PyTorch you have a graph that is cre

I really don't feel that there is magic in PyTorch or jax, but that may be because I have written my own autograd libs. In PyTorch you have a graph that is created on runtime by connecting the operations together in a transparent manner. Jax may feel a bit magic, but all that's done is sending / splitting tracers and recording the operations and compiling; by limiting the language, you have controlled branching with the proper semantics. --- The main reason I have had such with Julia is simply because of how early / soon I ended up needing to use them whereas with the other languages, you can get away without getting into the very messed up things.

kristofferc 3y ago
What did you feel you needed to use macros for?
- PartiallyTyped 3y ago
  I can’t remember, it was some statistical tools, running a few simulations, it was 2 years ago.
freilanzer 3y ago
> The main reason I have had such with Julia Such what? Fun? ;)
mathisfun123 3y ago
>In PyTorch you have a graph that is created on runtime by connecting the operations together in a transparent manner. You've jumped the shark here mate because autodiff in PyTorch is implemented using compile-time generated code in libtorch - it's not only the very definition of opaque but also pretty close in spirit to a macro.
- PartiallyTyped 3y ago
  Transparent in this case means the average user does not need to interact with the underlying system, it is there, it does stuff, but the user does not see it or interact directly. What is opaque is jax's tracer objects in the sense that the system exists, is there, the user is aware that it is there, and you can't peek inside it without actively going out of your way. With PyTorch you can implement your own functions with their own forward and backward passes; whether the underlying operations use libtorch is irrelevant, that is simply a backend. The computation graph __is__ created at runtime, whether the scaffolding for this is done in libtorch or directly in python is not relevant.
  - mathisfun123 3y ago
    well i mean if we redefine all the words so they mean whatever we want them to mean then sure blob code that isn't visible even in the source tree can be considered transparent and conversely, perfectly accessible script code (jax's interpreters) can be considered opaque what do i know i'm just a contrib on both projects shrug lots of your other claims are just complete misunderstandings e.g. whether the graph is created at runtime - it's not and you can read adam's original paper to find that out if you don't believe me - and the idea that libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch.
    - PartiallyTyped 3y ago
      > we redefine all the words I work for AWS; these are the definitions we use, more or less. https://stackoverflow.com/questions/17384020/what-do-transparent-and-opaque-mean-when-applied-to-programming-concepts https://stackoverflow.com/questions/17384020/what-do-transpa... > whether the graph is created at runtime I was specifically referring to the computation graph of the model that is used by autograd. Taken from [1,2]: > Instead, PyTorch uses the operator overloading approach, which builds up a representation of the computed function every time it is executed. > Internally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph. > A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. But what do I know? > libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch. Sure, libtorch is PyTorch; that doesn't mean one can't take the semantics of PyTorch and implement them in just numpy backed python. Doesn't mean it will be fast. [1] PyTorch: An Imperative Style, High-Performance Deep Learning Library [2] Automatic differentiation in PyTorch

kristofferc 3y ago

What did you feel you needed to use macros for?

PartiallyTyped 3y ago
I can’t remember, it was some statistical tools, running a few simulations, it was 2 years ago.

freilanzer 3y ago

> The main reason I have had such with Julia Such what? Fun? ;)

mathisfun123 3y ago

>In PyTorch you have a graph that is created on runtime by connecting the operations together in a transparent manner. You've jumped the shark here mate because autodiff in PyTorch is implemented using compile-time generated code in libtorch - it's not only the very definition of opaque but also pretty close in spirit to a macro.

PartiallyTyped 3y ago
Transparent in this case means the average user does not need to interact with the underlying system, it is there, it does stuff, but the user does not see it or interact directly. What is opaque is jax's tracer objects in the sense that the system exists, is there, the user is aware that it is there, and you can't peek inside it without actively going out of your way. With PyTorch you can implement your own functions with their own forward and backward passes; whether the underlying operations use libtorch is irrelevant, that is simply a backend. The computation graph __is__ created at runtime, whether the scaffolding for this is done in libtorch or directly in python is not relevant.
- mathisfun123 3y ago
  well i mean if we redefine all the words so they mean whatever we want them to mean then sure blob code that isn't visible even in the source tree can be considered transparent and conversely, perfectly accessible script code (jax's interpreters) can be considered opaque what do i know i'm just a contrib on both projects shrug lots of your other claims are just complete misunderstandings e.g. whether the graph is created at runtime - it's not and you can read adam's original paper to find that out if you don't believe me - and the idea that libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch.
  - PartiallyTyped 3y ago
    > we redefine all the words I work for AWS; these are the definitions we use, more or less. https://stackoverflow.com/questions/17384020/what-do-transparent-and-opaque-mean-when-applied-to-programming-concepts https://stackoverflow.com/questions/17384020/what-do-transpa... > whether the graph is created at runtime I was specifically referring to the computation graph of the model that is used by autograd. Taken from [1,2]: > Instead, PyTorch uses the operator overloading approach, which builds up a representation of the computed function every time it is executed. > Internally, a Variable is simply a wrapper around a Tensor that also holds a reference to a graph of Function objects. This graph is an immutable, purely functional representation of the derivative of computed function; Variables are simply mutable pointers to this graph. > A Function can be thought of as a closure that has all context necessary to compute vector-Jacobian products. They accept the gradients of the outputs, and return the gradients of the inputs (formally, the left product including the term for their respective operation.) A graph of Functions is a single argument closure that takes in a left product and multiplies it by the derivatives of all operations it contains. The left products passed around are themselves Variables, making the evaluation of the graph differentiable. But what do I know? > libtorch is just a backend - it's not - it is, for all intents and purposes, pytorch. Sure, libtorch is PyTorch; that doesn't mean one can't take the semantics of PyTorch and implement them in just numpy backed python. Doesn't mean it will be fast. [1] PyTorch: An Imperative Style, High-Performance Deep Learning Library [2] Automatic differentiation in PyTorch