9 ms·

There is a bit of "can't see the forest for the trees" failure in the article. AI is spearheading a paradigm shift in how we write programs. Or rather, we don't

There is a bit of "can't see the forest for the trees" failure in the article. AI is spearheading a paradigm shift in how we write programs. Or rather, we don't write programs. We write much much shorter programs that search the program space for programs that satisfy some desiderata. The programs we get as the output of the search process are extremely flexible, work very well, are very homogeneous in compute (e.g. conv/relu stacks), and never crash or memory leak. These are huge benefits compared to classical programs. So sure, backprop (the credit assignment scheme that gives us a good search direction in program space, one of multiple techniques that could do so) is pervasive, but AI is starting to work primarily as a result of a deeper epiphany - that we are not very good at all at writing code.

rawnlq 9y ago
Sounds like AI behaves similar to TDD where you blindly refactor until green, except AI have like a zillion more test cases(training data) to pass? Is that really a better way of writing code? (for example compared to being able to reason about the code to create something provably correct)
- stuartaxelowen 9y ago
  Are you maintaining the machine code produced by your code when compiled? You don't care if it's ugly if it works in quantifiable and predictable ways.
- allenz 9y ago
  It's a trade-off. We lose explainability, but we are able to solve completely new classes of problems.
mark_l_watson 9y ago
That is a good way to think about it. I am really encourage by new non-sequential architectures, multiple input and output channels, etc., and how these architectures can be expressed in functional Keras. It seems like building with something like functional composition is another paradigm shift. BTW, thanks for your writing, especially about RNNs: really useful when the ordering of sequential data is important. I use what I have learned from reading your blogs and papers literally every day at work.
pinouchon 9y ago
Just a quick question/remark: I have a feeling that it's best to think about DNNs as approximating a function, not a program. (only then you obtain a program as a result of applying this function). But because mathematically, you can formalise your big NN as one big parameterised function, I think it's more correct to view a NN as a function... Optimisation in program space would be trying to find both the structure (connections and activation functions) and weights of the NN, which is not what we do currently. We tend to hand/engineer (or keep what works best empirically) the architecture (1), then train by finding the best weights. I am very interested in approaches to efficiently opmitise in program space, and DL/backprop doesn't feel like it (1) although this is starting to change.
aws_ls 9y ago
>So sure, backprop (the credit assignment scheme that gives us a good search direction in program space, one of multiple techniques that could do so) is pervasive, but AI is starting to work primarily as a result of a deeper epiphany - that we are not very good at all at writing code. Isn't it applicable to a class of programs only? Best example of which is Computer Vision. Or do you imply your argument to hold for a wider set of programs. I can think of a large set of programs in which direct coding of logic, rather than discovery, is more suited. For example take sorting. I guess, sorting could also be taught to the machine, by having a training set. But what about the latency of the discovered program. Also what about the proof of such a sorting program, which is discovered by Machine learning? Must add, that I largely agree to your excellent point regarding discovery of programs. But I am not sure about its wide applicability. In fact, I contend that it applies only to a subset of all programs. Particulary those which have been traditionally difficult to code. So in that sense, now making a tangential point here, it is good that more complex applications are now possible, by combining both kind of programs. And there will be more programming work in the future. Edit: minor
- abecedarius 9y ago
  Daniel Hillis wrote a nice paper on machine-learned sorting networks decades ago. I think they were for fixed-size inputs, but the good news is that you can validate them using a 0-1 principle.
gilbetron 9y ago
I'm trying to get the company I work for used to the idea of ML in general, and one thing I get push back on is that we don't understand what such system are doing. Beyond showing them the math, I also point out to some of our bug reports that end up spanning thousands of lines, with input by our best, most experienced people, and that how often the bug "fix" is of the "well, it works now" type, and that we actually don't really understand what our system is doing. Of course the most common response back is a bunch of "yeah, but"s :) I'm making progress, but it is slow going. I've been steeped in code for nearly 40 years now, I'm ok with the fact that ML lets me step back a bit from tabs, semicolons, and objects!

rawnlq 9y ago

Sounds like AI behaves similar to TDD where you blindly refactor until green, except AI have like a zillion more test cases(training data) to pass? Is that really a better way of writing code? (for example compared to being able to reason about the code to create something provably correct)

stuartaxelowen 9y ago
Are you maintaining the machine code produced by your code when compiled? You don't care if it's ugly if it works in quantifiable and predictable ways.
allenz 9y ago
It's a trade-off. We lose explainability, but we are able to solve completely new classes of problems.

mark_l_watson 9y ago

That is a good way to think about it. I am really encourage by new non-sequential architectures, multiple input and output channels, etc., and how these architectures can be expressed in functional Keras. It seems like building with something like functional composition is another paradigm shift. BTW, thanks for your writing, especially about RNNs: really useful when the ordering of sequential data is important. I use what I have learned from reading your blogs and papers literally every day at work.

pinouchon 9y ago

Just a quick question/remark: I have a feeling that it's best to think about DNNs as approximating a function, not a program. (only then you obtain a program as a result of applying this function). But because mathematically, you can formalise your big NN as one big parameterised function, I think it's more correct to view a NN as a function... Optimisation in program space would be trying to find both the structure (connections and activation functions) and weights of the NN, which is not what we do currently. We tend to hand/engineer (or keep what works best empirically) the architecture (1), then train by finding the best weights. I am very interested in approaches to efficiently opmitise in program space, and DL/backprop doesn't feel like it (1) although this is starting to change.

aws_ls 9y ago

>So sure, backprop (the credit assignment scheme that gives us a good search direction in program space, one of multiple techniques that could do so) is pervasive, but AI is starting to work primarily as a result of a deeper epiphany - that we are not very good at all at writing code. Isn't it applicable to a class of programs only? Best example of which is Computer Vision. Or do you imply your argument to hold for a wider set of programs. I can think of a large set of programs in which direct coding of logic, rather than discovery, is more suited. For example take sorting. I guess, sorting could also be taught to the machine, by having a training set. But what about the latency of the discovered program. Also what about the proof of such a sorting program, which is discovered by Machine learning? Must add, that I largely agree to your excellent point regarding discovery of programs. But I am not sure about its wide applicability. In fact, I contend that it applies only to a subset of all programs. Particulary those which have been traditionally difficult to code. So in that sense, now making a tangential point here, it is good that more complex applications are now possible, by combining both kind of programs. And there will be more programming work in the future. Edit: minor

abecedarius 9y ago
Daniel Hillis wrote a nice paper on machine-learned sorting networks decades ago. I think they were for fixed-size inputs, but the good news is that you can validate them using a 0-1 principle.

gilbetron 9y ago

I'm trying to get the company I work for used to the idea of ML in general, and one thing I get push back on is that we don't understand what such system are doing. Beyond showing them the math, I also point out to some of our bug reports that end up spanning thousands of lines, with input by our best, most experienced people, and that how often the bug "fix" is of the "well, it works now" type, and that we actually don't really understand what our system is doing. Of course the most common response back is a bunch of "yeah, but"s :) I'm making progress, but it is slow going. I've been steeped in code for nearly 40 years now, I'm ok with the fact that ML lets me step back a bit from tabs, semicolons, and objects!