4 ms·

I haven't read this yet. Can anyone describe how this compares to Darcs' "calculus"?

by cbsmith 10y ago

mike_hearn 10y ago
They're totally unrelated. This paper is rather hard to read and I've put it into my pile of papers to work through when it's not 1 AM, but basically this is related to machine learning and neural networks. More generally it's about how to transform arbitrary imperative programs with branching and control flow into a "differentiable" form. It means, you can train the program by using algorithms like gradient descent. This article may help: https://en.wikipedia.org/wiki/Differentiable_neural_computer https://en.wikipedia.org/wiki/Differentiable_neural_computer The awesome/scary thing about this line of research is that it's exploring the question of how far computer programming can be automated. Simple machine learning techniques can already be seen as "automatic computer programmers" in a sense, but their capacity is very limited. DeepMind recently made a breakthrough with their DNC architecture, described in that wiki page, which introduces a notion of memory to neural networks, and right now NN research is going at an incredible speed. The more stuff that becomes encoded in a differentiable form, the more stuff can be "trained" based on examples using standard machine learning techniques. I plan to do some blogging on these topics soon.
- joe_the_user 10y ago
  Well, I too put it on my "to read" stack but if it is referring to ordinary automatic differentiation, also known as "algorithmic differentiation" that beast is a bit more specific. That is, ad allows the differentiation of functions defined as algorithms but not with arbitrary "branching and control flow" but essentially only when the "algorithm" involve a composition of primitive functions having known derivatives (example: addition, multiplication, exponentiation). That is still a big set - back propagation uses one form of AD for example. However, for example, AD cannot be correctly applied to functions defined with loop or with arbitrary recursion. I'm not sure if there's a relation to the DeepMind DNC but maybe if I read article I'll find out. See: https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation
- narrator 10y ago
  Isn't there a limit to completely automated programming that has to do more or less with the halting problem? Like if the program builds an infinite loop? I guess it will have to timeout, but that's going to slow the learning down. Deep neural networks don't have conditional loops in them so they don't have to deal with the halting problem.
  - andrewflnr 10y ago
    I'm not sure exactly how it applies here, but in general you can limit the "language" so it only includes things that are guaranteed to halt. You lose a lot of expressive power, but hey, no halting problem. The obvious thing to do is guarantee loops halt by forcing them to have a monotonic countdown, or ban them entirely.
deleted 10y ago
[deleted]

I haven't read this yet. Can anyone describe how this compares to Darcs' "calculus"?

by cbsmith 10y ago

I haven't read this yet. Can anyone describe how this compares to Darcs' "calculus"?

They're totally unrelated. This paper is rather hard to read and I've put it into my pile of papers to work through when it's not 1 AM, but basically this is related to machine learning and neural networks. More generally it's about how to transform arbitrary imperative programs with branching and control flow into a "differentiable" form. It means, you can train the program by using algorithms like gradient descent. This article may help: https://en.wikipedia.org/wiki/Differentiable_neural_computer https://en.wikipedia.org/wiki/Differentiable_neural_computer The awesome/scary thing about this line of research is that it's exploring the question of how far computer programming can be automated. Simple machine learning techniques can already be seen as "automatic computer programmers" in a sense, but their capacity is very limited. DeepMind recently made a breakthrough with their DNC architecture, described in that wiki page, which introduces a notion of memory to neural networks, and right now NN research is going at an incredible speed. The more stuff that becomes encoded in a differentiable form, the more stuff can be "trained" based on examples using standard machine learning techniques. I plan to do some blogging on these topics soon.

joe_the_user 10y ago
Well, I too put it on my "to read" stack but if it is referring to ordinary automatic differentiation, also known as "algorithmic differentiation" that beast is a bit more specific. That is, ad allows the differentiation of functions defined as algorithms but not with arbitrary "branching and control flow" but essentially only when the "algorithm" involve a composition of primitive functions having known derivatives (example: addition, multiplication, exponentiation). That is still a big set - back propagation uses one form of AD for example. However, for example, AD cannot be correctly applied to functions defined with loop or with arbitrary recursion. I'm not sure if there's a relation to the DeepMind DNC but maybe if I read article I'll find out. See: https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation
narrator 10y ago
Isn't there a limit to completely automated programming that has to do more or less with the halting problem? Like if the program builds an infinite loop? I guess it will have to timeout, but that's going to slow the learning down. Deep neural networks don't have conditional loops in them so they don't have to deal with the halting problem.
- andrewflnr 10y ago
  I'm not sure exactly how it applies here, but in general you can limit the "language" so it only includes things that are guaranteed to halt. You lose a lot of expressive power, but hey, no halting problem. The obvious thing to do is guarantee loops halt by forcing them to have a monotonic countdown, or ban them entirely.

deleted 10y ago

[deleted]