3 ms·
>> Our work has connections to existing work not only in deep learning, but also to various other fields Perhaps I'm, er, biased, but the most obvious similari
by YeGoblynQueenne 7y ago
>> Our work has connections to existing work not only in deep learning, but also
to various other fields
Perhaps I'm, er, biased, but the most obvious similarity that immediately
jumped out to me when I started reading this article is with Inductive
Programming, the broader field of machine learning algorithms that learn
programs, that includes Inductive Functional Programming and Inductive Logic
Programming (which I study; hence, "biased").
In fact, I'd go as far as to say that this is a new kind of Inductive
Programming. Perhaps it should be called Inductive Continuous Programming or
Inductive Neural Progamming or some such. But what's exceedingly clear to me
is that what the article describes is a search for a program that is composed
of continuous functions, guided by a strong structural bias.
The hallmarks of Inductive Progamming include an explicit encoding of strong
structural biases, including by the use of a library of primitives from which
the learned model is composed; a strong Occamist bias; learned models that are
programs. This work ticks all the boxes. The authors might benefit from
looking up the relevan bibliography.
A couple of starting pointers:
Magic Haskeller, an Inductive Functional Programming system.
http://nautilus.cs.miyazaki-u.ac.jp/%7Eskata/MagicHaskeller.html http://nautilus.cs.miyazaki-u.ac.jp/%7Eskata/MagicHaskeller....
Metagol, an Inductive Logic Programming system:
https://github.com/metagol/metagol https://github.com/metagol/metagol
ILASP, a system for the Inductive Learning of Answer Set Programs:
http://ilasp.com/ http://ilasp.com/
- darkmighty 7y ago> In fact, I'd go as far as to say that this is a new kind of Inductive Programming. I wouldn't. Traditional NNs are widely called funciton approximators, or function induction. That's because they're not Turing-complete, and in this case don't even have memory. This makes them decidedly not programs, just functions. Of course, they could be modified with recurrence (a suggestion of future work), or even more complete memory systems (e.g. equipping a neural controller with a tape), making it Turing-complete. Keep in mind the functional approximation power isn't increased by architecture search alone (there's no restriction from doing weight training). What seems novel here is how significant architecture alone is (I wouldn't guess it being this powerful), and this possibly has implications on the connection with the human brain -- in which my vague impression is that connections themselves are much more tuned than weights; also possibly giving greater emphasis on architecture training.
- YeGoblynQueenne 7y ago>> I wouldn't. Traditional NNs are widely called funciton approximators, or function induction. That's because they're not Turing-complete, and in this case don't even have memory. This makes them decidedly not programs, just functions. But what the article describes is not function approximation anymore. Their system doesn't optimise a set of parameters. The search for an optimal set of parameters has been replaced entirely with a search for an optimal architecture with a single, shared parameter. Note also that you don't need something to be Turing-complete for it to be a program. For instance, a regular automaton is a program, but it's not Turing complete. Turing completeness makes something, well, a Universal Turing Machine, which can compute any program. But a program is just a set of instructions.
- darkmighty 7y ago> Note also that you don't need something to be Turing-complete for it to be a program. For instance, a regular automaton is a program, but it's not Turing complete. Turing completeness makes something, well, a Universal Turing Machine, which can compute any program. But a program is just a set of instructions. A regular automaton is a program (and a function is a program), but all programs cannot be expressed as finite (regular) automatons (FAs). So FAs are a subset of programs. If we equate programs with mathematical algorithms, in general they really need the unlimited memory aspect (or at least the capability of abstracting memory, which in practice is always finite of course). To make things clear, an example: a fixed function (or FA) cannot sort a list of numbers of variable size. You can train your function to sort any K numbers, which are the input variables. But because it has no memory, you cannot input variables sequentially (in the FA case you cannot input arbitrarily many variables). You've essentially created a sorting network[1]. Not only that, but for each input growth you need to re-train your architecture. The nice thing about program inference is that it hopefully captures the fundamental algorithm behind a program, which generalizes to arbitrary sizes. It's not that arbitrary sizes are necessary in practice (again computers are always bounded), it's that this abstraction/generalization is extremely useful, saving data and allowing large (arbitrary) variation in input. [1] https://en.wikipedia.org/wiki/Sorting_network https://en.wikipedia.org/wiki/Sorting_network > a Universal Turing Machine, which can compute any program I feel there's a bit of confusion here. Turing Machines themselves can compute any program (i.e. you can represent any program as a TM) -- that follows essentially from the definition of algorithm (a set of well defined steps given unlimited paper); again languages accepted by FAs are a restricted subset of TMs (the former accepts a regular language while the latter recursively enumerable languages). There are special Turing Machines that simulate arbitrary Turing Machines -- those are called Universal Turing machines. It's a desirable characteristic of any computer, of course (the ability to input arbitrary programs). We refer to program description languages that can describe any Turing Machine as Turing-complete. You could make a Universal Turing machine that accepts this kind of language directly or you could in principle just build a Turing machine that executes the given algorithm (for example using a gate array/FPGA to encode the state automaton, plus some large memory for the tape).