11 ms·

The Elements of Differentiable Programming

131 points by leephillips 3y ago

deleted 3y ago
[deleted]
blurbleblurble 3y ago
This is amazing! Seems like a perfect excuse to get back into Julia. I just wish Julia had more compile targets. Ideally I'd like to have the option to target the browser (wasm/webgpu).
- UncleOxidant 3y ago
  https://julialang.org/jsoc/gsoc/wasm/ https://julialang.org/jsoc/gsoc/wasm/
  - blurbleblurble 3y ago
    Awesome, I hope it pans out
    - ChrisRackauckas 3y ago
      We have a lot of things compiling with web assembly, like the ODE solvers. See https://tshort.github.io/WebAssemblyCompiler.jl/stable/examples/lorenz/ https://tshort.github.io/WebAssemblyCompiler.jl/stable/examp...
- UncleOxidant 3y ago
  The Julia community seems like they've been out front when it comes to differentiable programming. For example: https://www.youtube.com/watch?v=rF2QAJLM730 https://www.youtube.com/watch?v=rF2QAJLM730
fpgamlirfanboy 3y ago
i don't know why people write these things. it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". so it's a pretty bad math book to actually learn math from (you can always spot a pedagogically unsound math book by its lack of exercises). the book doesn't even have any code in it which is surprising considering it has "programming" in the title. actually i know why people write these but i still don't know why they publish them: this is a phase everyone goes through in their "math" life where they look back on everything they've learned hastily between undergrad/phd/postdoc (or whatever) and they have the urge to formalize/crystallize. everyone has the urge - i had a late-career QFT prof tell me that he was excited to take his sabbatical so that he could finally do all of the exercises in peskin&schroeder for real real and type it all up neatly. i've done it too, in-the-small (some very nice notes that i'm proud of, on various things). you sit down, make your list of things, pull up all of the books/papers you're going to use as references and you start essentially transcribing - but you tell yourself you're putting your own spin on it (adding ample "motivation"). and it's all fine and healthy and gratifying for you and yourself alone. but i don't think i'd ever imagine to myself "well hmm my organization of these topics is going to be useful for other people i should put it out there for the world to see". but that's just me. ninja edit: before someone jumps down my throat about "what's the harm?". the harm is n00b/undergrads/young people/etc will look at this and think this is the right way to learn this material and some of them will even make an attempt to learn the material from this thing and they'll struggle and fail and be discouraged - i speak from experience! it's not a good thing for the community. sure maybe 1 in a 100 can learn this stuff from just reading a monograph (what these things used to be called...) but that's the exception that proves the rule.
- deleted 3y ago
  [deleted]
- hyperbovine 3y ago
  [flagged]
  - fpgamlirfanboy 3y ago
    > graduate-level introduction to differentiable programming. go check out any real graduate textbook. what you will find is they all have exercises. > It's a book for ML researchers. I'm excited to read it. Calm down. just because the authors claim something doesn't make it true. i'm not wrong - this is not a good pedagogical resource and i would bet a year of my salary (as an ML researcher) that you will in fact not read more than 5% of this book.
    - smallnamespace 3y ago
      It's a good thing you're an ML researcher and not, say, a quant trader then ;)
    - thfuran 3y ago
      Really? I'd gladly read 6% of a book for a year of pay.
    - hyperbovine 3y ago
      You certainly have the demeanor of a certain type of ML researcher that I have encountered from time to time ...
- bbor 3y ago
  You’re getting hit on a bunch of different things here, but I’d like to focus on two: it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". I mean, yes…? Maybe I’m a terrible programmer but I’ve never applied calculus to my work in any real way. A book that’s just “calculus applications for software design” seems quite useful, and quite unrelated to teaching “math” in a direct way. and it's all fine and healthy and gratifying for you and yourself alone. but i don't think i'd ever imagine to myself "well hmm my organization of these topics is going to be useful for other people i should put it out there for the world to see". but that's just me. Doesn’t this apply to all books of any kind? How do you know what if you can write a book before you try? I feel like “transcribe papers with a spin on them” is a perfect description of Russel and Norvig’s AI book, and many find it valuable. Is this just a math-specific criticism? Either way, thanks for the very interested contrarian comment! You’re well spoken and I do love a discussion more than “wow this looks useful”, which is all I was prepared to give.
amelius 3y ago
Would this be useful for general applications, or just numerical ones?
- dkjaudyeqooe 3y ago
  Your program takes various inputs, does processing and gives you results. To quote the book this applies to programs "including those with control flows and data structures" and are not entirely numerical in the most common sense. Broadly speaking, with AD you can modify the results and the program will spit out the required inputs to get those results.
  - amelius 3y ago
    So if a program's result is "no access", then AD can figure out how you can get access. Sounds like an important hacker tool.
    - cscheid 3y ago
      I mean, “differential cryptanalysis” doesn’t go through AD, but I would be surprised if it’s not possible to get pretty close to a discrete analogue to AD using bit flips instead of differential and abstract interpretation.
    - kelseyfrog 3y ago
      Cryptography functions, I'd assume, are purposely non-differentiable.
    - lsb 3y ago
      Yes, these are “adversarial patches” in image classification, like https://arxiv.org/abs/1712.09665 https://arxiv.org/abs/1712.09665 . Similarly you can take these adversaries and add them to your own larger model, in an arms race.
  - JadeNB 3y ago
    > Broadly speaking, with AD you can modify the results and the program will spit out the required inputs to get those results. I assume you're trying to phrase it in a non-technical way for accessibility, but I wonder if that might have lost some precision. What you describe sounds more like (logically) reversible programming (https://en.wikipedia.org/wiki/Reversible_computing https://en.wikipedia.org/wiki/Reversible_computing). Differentiability doesn't imply reversibility; for example, the program that takes in an input and returns 1 is as differentiable as they come, but there's nothing that differentiation, automatic or otherwise, can do to tell you an input that will make it return 2.
dkjaudyeqooe 3y ago
This is a timely book. Maybe more interesting (at least to me) than the recent results of AI research is the application of techniques used in the field applied elsewhere. For my work going forward catering to automatic differentiation in the code is a no-brainer.
whoevercares 3y ago
TBH I would hope this is a Jax deep dive book
geor9e 3y ago
Why did the deep learning model cross the road? Because it was smooth and differentiable
MikeBattaglia 3y ago
One very interesting thing about automatic differentiation is that you can think of it as involving a new algebra, similar to the complex numbers, where we adjoin an extra element to the reals to form a plane. This new algebra is called the ring of "dual numbers." The difference is that instead of adding a new element "i" with i² = -1, we add one called "h" with h² = 0! Every element in the dual numbers is of the form a + bh, and in fact the entire ring can be turned into a totally ordered ring in a very natural way: simply declare h < r for any real r > 0. In essence, we are saying h is an infinitesimal - so small that its square is 0. So we have a non-Archimedean ring with infinitesimals - the smallest such ring extending the real numbers. Why is this so important? Well, if you have some function f which can be extended to the dual number plane - which many can, similar to the complex plane - we have f(x+h) = f(x) + f'(x)h Which is little more than restating the usual definition of the derivative: f'(x) = (f(x+h) - f(x))/h For instance, suppose we have f(x) = 2x² - 3x + 1, then f(x+h) = 2(x+h)² - 3(x+h) + 1 = 2(x² + 2xh + h²) - 3(x+h) + 1 = (2x² - 3x + 1) + (4x - 3)h Where the last step just involves rearranging terms and canceling out the h² = 0 term. Note that the expression for the derivative we get, (4x-3), is correct, and magically computed itself straight from the properties of the algebra. In short, just like creating i² = -1 revolutionized algebra, setting h² = 0 revolutionizes calculus. Most autodiff packages (such as Pytorch) use something not much more advanced than this, although there are optimizations to speed it up (e.g. reverse mode diff).
- compacct27 3y ago
  Where do I go to learn what you just said?
  - smokel 3y ago
    The book referred to in this post has some information about this. The method with "dual numbers" is called "forward-mode automatic differentiation". PyTorch seems to use "reverse-mode automatic differentiation", which does not use dual numbers, but keeps track of the computation graph.
    - fpgamlirfanboy 3y ago
ubj 3y ago
This looks like a great resource! Differentiable programming is such a cool area. I'll never forget writing a differentiable PID controller a few years ago and watching the PID gains get tuned automagically to stabilize the control system. It's powerful stuff if you use it in the right places.
- persnickety 3y ago
  How did you do it? Is there a resource explaining the process online? If there isn't, and you're willing to create one, I offer reviewing.
  - redadehy2 3y ago
    At Collimator.ai we built a differentiable modeling and simulation platform for dynamical systems on JAX (think Simulink but in the cloud and with Python+JAX instead of Matlab) in order to enable more efficient optimization, autotuning, MPC, neural network control, etc. Here is a simple example of PID tuning with JAX and autodiff: https://py.collimator.ai/examples/pid_tuning/ https://py.collimator.ai/examples/pid_tuning/ (feedback is welcome!)
  - ubj 3y ago
    I haven't ever written up a tutorial, but here's a technical report from someone else that gives an overview: https://lucris.lub.lu.se/ws/portalfiles/portal/61129581/autotuning.pdf https://lucris.lub.lu.se/ws/portalfiles/portal/61129581/auto...

131 points by leephillips 3y ago

deleted 3y ago

[deleted]

This is amazing! Seems like a perfect excuse to get back into Julia. I just wish Julia had more compile targets. Ideally I'd like to have the option to target the browser (wasm/webgpu).

UncleOxidant 3y ago
https://julialang.org/jsoc/gsoc/wasm/ https://julialang.org/jsoc/gsoc/wasm/
- blurbleblurble 3y ago
  Awesome, I hope it pans out
  - ChrisRackauckas 3y ago
    We have a lot of things compiling with web assembly, like the ODE solvers. See https://tshort.github.io/WebAssemblyCompiler.jl/stable/examples/lorenz/ https://tshort.github.io/WebAssemblyCompiler.jl/stable/examp...
UncleOxidant 3y ago
The Julia community seems like they've been out front when it comes to differentiable programming. For example: https://www.youtube.com/watch?v=rF2QAJLM730 https://www.youtube.com/watch?v=rF2QAJLM730

fpgamlirfanboy 3y ago

i don't know why people write these things. it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". so it's a pretty bad math book to actually learn math from (you can always spot a pedagogically unsound math book by its lack of exercises). the book doesn't even have any code in it which is surprising considering it has "programming" in the title. actually i know why people write these but i still don't know why they publish them: this is a phase everyone goes through in their "math" life where they look back on everything they've learned hastily between undergrad/phd/postdoc (or whatever) and they have the urge to formalize/crystallize. everyone has the urge - i had a late-career QFT prof tell me that he was excited to take his sabbatical so that he could finally do all of the exercises in peskin&schroeder for real real and type it all up neatly. i've done it too, in-the-small (some very nice notes that i'm proud of, on various things). you sit down, make your list of things, pull up all of the books/papers you're going to use as references and you start essentially transcribing - but you tell yourself you're putting your own spin on it (adding ample "motivation"). and it's all fine and healthy and gratifying for you and yourself alone. but i don't think i'd ever imagine to myself "well hmm my organization of these topics is going to be useful for other people i should put it out there for the world to see". but that's just me. ninja edit: before someone jumps down my throat about "what's the harm?". the harm is n00b/undergrads/young people/etc will look at this and think this is the right way to learn this material and some of them will even make an attempt to learn the material from this thing and they'll struggle and fail and be discouraged - i speak from experience! it's not a good thing for the community. sure maybe 1 in a 100 can learn this stuff from just reading a monograph (what these things used to be called...) but that's the exception that proves the rule.

deleted 3y ago
[deleted]
hyperbovine 3y ago
[flagged]
- fpgamlirfanboy 3y ago
  > graduate-level introduction to differentiable programming. go check out any real graduate textbook. what you will find is they all have exercises. > It's a book for ML researchers. I'm excited to read it. Calm down. just because the authors claim something doesn't make it true. i'm not wrong - this is not a good pedagogical resource and i would bet a year of my salary (as an ML researcher) that you will in fact not read more than 5% of this book.
  - smallnamespace 3y ago
    It's a good thing you're an ML researcher and not, say, a quant trader then ;)
  - thfuran 3y ago
    Really? I'd gladly read 6% of a book for a year of pay.
  - hyperbovine 3y ago
    You certainly have the demeanor of a certain type of ML researcher that I have encountered from time to time ...
bbor 3y ago
You’re getting hit on a bunch of different things here, but I’d like to focus on two: it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". I mean, yes…? Maybe I’m a terrible programmer but I’ve never applied calculus to my work in any real way. A book that’s just “calculus applications for software design” seems quite useful, and quite unrelated to teaching “math” in a direct way. and it's all fine and healthy and gratifying for you and yourself alone. but i don't think i'd ever imagine to myself "well hmm my organization of these topics is going to be useful for other people i should put it out there for the world to see". but that's just me. Doesn’t this apply to all books of any kind? How do you know what if you can write a book before you try? I feel like “transcribe papers with a spin on them” is a perfect description of Russel and Norvig’s AI book, and many find it valuable. Is this just a math-specific criticism? Either way, thanks for the very interested contrarian comment! You’re well spoken and I do love a discussion more than “wow this looks useful”, which is all I was prepared to give.

amelius 3y ago

Would this be useful for general applications, or just numerical ones?

dkjaudyeqooe 3y ago
Your program takes various inputs, does processing and gives you results. To quote the book this applies to programs "including those with control flows and data structures" and are not entirely numerical in the most common sense. Broadly speaking, with AD you can modify the results and the program will spit out the required inputs to get those results.
- amelius 3y ago
  So if a program's result is "no access", then AD can figure out how you can get access. Sounds like an important hacker tool.
  - cscheid 3y ago
    I mean, “differential cryptanalysis” doesn’t go through AD, but I would be surprised if it’s not possible to get pretty close to a discrete analogue to AD using bit flips instead of differential and abstract interpretation.
  - kelseyfrog 3y ago
    Cryptography functions, I'd assume, are purposely non-differentiable.
  - lsb 3y ago
    Yes, these are “adversarial patches” in image classification, like https://arxiv.org/abs/1712.09665 https://arxiv.org/abs/1712.09665 . Similarly you can take these adversaries and add them to your own larger model, in an arms race.
- JadeNB 3y ago
  > Broadly speaking, with AD you can modify the results and the program will spit out the required inputs to get those results. I assume you're trying to phrase it in a non-technical way for accessibility, but I wonder if that might have lost some precision. What you describe sounds more like (logically) reversible programming (https://en.wikipedia.org/wiki/Reversible_computing https://en.wikipedia.org/wiki/Reversible_computing). Differentiability doesn't imply reversibility; for example, the program that takes in an input and returns 1 is as differentiable as they come, but there's nothing that differentiation, automatic or otherwise, can do to tell you an input that will make it return 2.

dkjaudyeqooe 3y ago

This is a timely book. Maybe more interesting (at least to me) than the recent results of AI research is the application of techniques used in the field applied elsewhere. For my work going forward catering to automatic differentiation in the code is a no-brainer.

whoevercares 3y ago

TBH I would hope this is a Jax deep dive book

geor9e 3y ago

Why did the deep learning model cross the road? Because it was smooth and differentiable

MikeBattaglia 3y ago

One very interesting thing about automatic differentiation is that you can think of it as involving a new algebra, similar to the complex numbers, where we adjoin an extra element to the reals to form a plane. This new algebra is called the ring of "dual numbers." The difference is that instead of adding a new element "i" with i² = -1, we add one called "h" with h² = 0! Every element in the dual numbers is of the form a + bh, and in fact the entire ring can be turned into a totally ordered ring in a very natural way: simply declare h < r for any real r > 0. In essence, we are saying h is an infinitesimal - so small that its square is 0. So we have a non-Archimedean ring with infinitesimals - the smallest such ring extending the real numbers. Why is this so important? Well, if you have some function f which can be extended to the dual number plane - which many can, similar to the complex plane - we have f(x+h) = f(x) + f'(x)h Which is little more than restating the usual definition of the derivative: f'(x) = (f(x+h) - f(x))/h For instance, suppose we have f(x) = 2x² - 3x + 1, then f(x+h) = 2(x+h)² - 3(x+h) + 1 = 2(x² + 2xh + h²) - 3(x+h) + 1 = (2x² - 3x + 1) + (4x - 3)h Where the last step just involves rearranging terms and canceling out the h² = 0 term. Note that the expression for the derivative we get, (4x-3), is correct, and magically computed itself straight from the properties of the algebra. In short, just like creating i² = -1 revolutionized algebra, setting h² = 0 revolutionizes calculus. Most autodiff packages (such as Pytorch) use something not much more advanced than this, although there are optimizations to speed it up (e.g. reverse mode diff).

compacct27 3y ago
Where do I go to learn what you just said?
- smokel 3y ago
  The book referred to in this post has some information about this. The method with "dual numbers" is called "forward-mode automatic differentiation". PyTorch seems to use "reverse-mode automatic differentiation", which does not use dual numbers, but keeps track of the computation graph.
  - fpgamlirfanboy 3y ago

ubj 3y ago

This looks like a great resource! Differentiable programming is such a cool area. I'll never forget writing a differentiable PID controller a few years ago and watching the PID gains get tuned automagically to stabilize the control system. It's powerful stuff if you use it in the right places.

persnickety 3y ago
How did you do it? Is there a resource explaining the process online? If there isn't, and you're willing to create one, I offer reviewing.
- redadehy2 3y ago
  At Collimator.ai we built a differentiable modeling and simulation platform for dynamical systems on JAX (think Simulink but in the cloud and with Python+JAX instead of Matlab) in order to enable more efficient optimization, autotuning, MPC, neural network control, etc. Here is a simple example of PID tuning with JAX and autodiff: https://py.collimator.ai/examples/pid_tuning/ https://py.collimator.ai/examples/pid_tuning/ (feedback is welcome!)
- ubj 3y ago
  I haven't ever written up a tutorial, but here's a technical report from someone else that gives an overview: https://lucris.lub.lu.se/ws/portalfiles/portal/61129581/autotuning.pdf https://lucris.lub.lu.se/ws/portalfiles/portal/61129581/auto...