5 ms·

My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use

My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.

SleekEagle 4y ago
That's part of why Julia is so exciting! Building it specifically to be a differentiable programming language opens so many doors ...
- mountainriver 4y ago
  Julia wasn’t really built specifically to be differentiable, it was just built in a way that you have access to the IR, which is what zygote does. Enzyme AD is the most exciting to me because any LLVM language can be differentiable
  - SleekEagle 4y ago
    Ah I see, thank you for clarifying. And thank you for bringing Enzyme to my attention - I've never seen it before!
    - celrod 4y ago
      Enzyme.jl works quite well (but the possibility of using it across languages is appealing).
melony 4y ago
I am just happy that the previously siloed fields of operations research and various control theory sub-disciplines are now incentivized to pool their research together thanks to the funding in ML. Also many expensive and proprietary optimization software in industry are finally getting some competition.
- SleekEagle 4y ago
  Hm I didn't know different areas of control theory were siloed. Learning about control theory in graduate school was awesome and it seems like a field that would benefit from ML a lot. I know they use RL agents for control networks for e.g. cartpole, but I would've thought it would be more widespread! Do you think the development of Differentiable Programming (i.e. the observation of more generality beyond pure ML/DL) was really the missing piece? Also, just curious, what are your studies in?
  - melony 4y ago
    Control theory has a very, very long parallel history alongside ML. ML, specifically probabilistic and reinforcement learning, uses a lot of dynamic programming ideas and Bellman equations in its theoretical modeling. Lookup the term cybernetics, it is an old term in the pre-internet era to mean control theory and optimization. The Soviets even had a grand scheme to build networked factories that could be centrally optimized and resource allocated. Their Slavic communist AWS-meets-Walmart efforts spawned a Nobel laureate; Kantorovich was given the award for inventing linear programming. Unfortunately the CS field is only just rediscovering control theory while it has been a staple of EE for years. However, there haven't been many new innovations in the field until recently when ML became the new hottest thing.
    - SleekEagle 4y ago
      This is some insanely cool history! I had no idea the Soviets had such a technical vision, that's actually pretty amazing. I've heard the term "cybernetics" but honestly just thought it was some movie-tech term, lol. It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I remember a math professor of mine telling us about optimization techniques that control systems practitioners would know more about than applied mathematicians because they were developed specifically for the field, can't remember what the techniques were though ...
      - 4y ago
potbelly83 4y ago
How do you differentiate a string? Enum?
- adgjlsfhk1 4y ago
  generally you consider them to be piecewise constant.
  - tome 4y ago
    Or more precisely, discrete.
- titanomachy 4y ago
  If you were dealing with e.g. English words rather than arbitrary strings, one approach would be to treat each word as a point in n-dimensional space. Then you can use continuous (and differentiable) functions to output into that space.
- 6gvONxR4sf7o 4y ago
  The answer to that is a huge part of the NLP field. The current answer is that you break down the string into constituent parts and map each of them into a high dimensional space. “cat” becomes a large vector whose position is continuous and therefore differentiable. “the cat” probably becomes a pair of vectors.
  - nerdponx 4y ago
    It's weirder than that. You typically are differentiating a loss function of strings and various opaque weights. You are optimizing the loss function over the weight space, so in some informal contrived sense you are actually differentiating with respect to the string.
- geysersam 4y ago
  Not all functions are differentiable. Sometimes there are other better ways to describe "how does changing x affect y". Derivatives are powerful but they are not the only possible description of such relationships. I'm very excited for what other things future "compilers" will be able to do to programs besides differentiation. That's just the beginning.

SleekEagle 4y ago

That's part of why Julia is so exciting! Building it specifically to be a differentiable programming language opens so many doors ...

mountainriver 4y ago
Julia wasn’t really built specifically to be differentiable, it was just built in a way that you have access to the IR, which is what zygote does. Enzyme AD is the most exciting to me because any LLVM language can be differentiable
- SleekEagle 4y ago
  Ah I see, thank you for clarifying. And thank you for bringing Enzyme to my attention - I've never seen it before!
  - celrod 4y ago
    Enzyme.jl works quite well (but the possibility of using it across languages is appealing).

melony 4y ago

I am just happy that the previously siloed fields of operations research and various control theory sub-disciplines are now incentivized to pool their research together thanks to the funding in ML. Also many expensive and proprietary optimization software in industry are finally getting some competition.

SleekEagle 4y ago
Hm I didn't know different areas of control theory were siloed. Learning about control theory in graduate school was awesome and it seems like a field that would benefit from ML a lot. I know they use RL agents for control networks for e.g. cartpole, but I would've thought it would be more widespread! Do you think the development of Differentiable Programming (i.e. the observation of more generality beyond pure ML/DL) was really the missing piece? Also, just curious, what are your studies in?
- melony 4y ago
  Control theory has a very, very long parallel history alongside ML. ML, specifically probabilistic and reinforcement learning, uses a lot of dynamic programming ideas and Bellman equations in its theoretical modeling. Lookup the term cybernetics, it is an old term in the pre-internet era to mean control theory and optimization. The Soviets even had a grand scheme to build networked factories that could be centrally optimized and resource allocated. Their Slavic communist AWS-meets-Walmart efforts spawned a Nobel laureate; Kantorovich was given the award for inventing linear programming. Unfortunately the CS field is only just rediscovering control theory while it has been a staple of EE for years. However, there haven't been many new innovations in the field until recently when ML became the new hottest thing.
  - SleekEagle 4y ago
    This is some insanely cool history! I had no idea the Soviets had such a technical vision, that's actually pretty amazing. I've heard the term "cybernetics" but honestly just thought it was some movie-tech term, lol. It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I remember a math professor of mine telling us about optimization techniques that control systems practitioners would know more about than applied mathematicians because they were developed specifically for the field, can't remember what the techniques were though ...
    - 4y ago

potbelly83 4y ago

How do you differentiate a string? Enum?

adgjlsfhk1 4y ago
generally you consider them to be piecewise constant.
- tome 4y ago
  Or more precisely, discrete.
titanomachy 4y ago
If you were dealing with e.g. English words rather than arbitrary strings, one approach would be to treat each word as a point in n-dimensional space. Then you can use continuous (and differentiable) functions to output into that space.
6gvONxR4sf7o 4y ago
The answer to that is a huge part of the NLP field. The current answer is that you break down the string into constituent parts and map each of them into a high dimensional space. “cat” becomes a large vector whose position is continuous and therefore differentiable. “the cat” probably becomes a pair of vectors.
- nerdponx 4y ago
  It's weirder than that. You typically are differentiating a loss function of strings and various opaque weights. You are optimizing the loss function over the weight space, so in some informal contrived sense you are actually differentiating with respect to the string.
geysersam 4y ago
Not all functions are differentiable. Sometimes there are other better ways to describe "how does changing x affect y". Derivatives are powerful but they are not the only possible description of such relationships. I'm very excited for what other things future "compilers" will be able to do to programs besides differentiation. That's just the beginning.