3 ms·

Ironically, exactly these sorts of "statistics can't..." arguments are sort of theoretically bankrupt. Either the thing you want to do is impossible or else a

Ironically, exactly these sorts of "statistics can't..." arguments are sort of theoretically bankrupt. Either the thing you want to do is impossible or else a learned model can do it at least almost as well as... idk what the alternative even is, something not learned? Taking this to the extreme, on an example where I have first-hand experience: I would never recommend replacing your compiler passes with transformers. The latter will be buggier, at best marginally faster, and it will take several orders of magnitude more effort to get them to work well enough for production. ML isn't the right tool. But, I mean, you can do it. You shouldn't. But you can. To be fair to the article, the title is "won't", not "can't". And that's easier to believe, at least for me. It's not that X is unattainable using ML; it's that some other approach will get there first.

tudelo 5y ago
Are there not situations where a learned model could be insufficient due to lack of real-time learning? It is my understanding that a lot of current applications take a ton of training time / computing power and it's not so trivial to do in the moment. I guess this is less of a theoretical problem and more of a practical problem though.
- throwawaygh 5y ago
  > I guess this is less of a theoretical problem and more of a practical problem though. Yeah, exactly. Theoretically, ML can do anything. Practically, not so much.
snek_case 5y ago
The reason you wouldn't want to use ML for program transformations is that compiler analyses like constant folding are essentially fast, specialized theorem provers. When you need to know if an optimization is safe or not (is this expression equivalent to that one?), you want an exact solution, not just a pretty good guess, and there are known techniques that can give you just that. However, you can use ML in areas where compilers use heuristics. For example, how many times should you unroll this loop, or should you inline function B into function A? Your program is going to be valid with or without inlining/unrolling, and the compiler needs an "intuition" of what's going to be best for performance. Right now, in most cases, this uses simple hardcoded rules. The same goes for math. You wouldn't ask a neural network to certify whether a mathematical proof is valid or not, knowing it has been 99% accurate on the test set. You couldn't trust it to verify all the steps. However, you could use a neural network to help you guide a formal theorem prover in searching for a proof, as in which branches of the infinite tree of mathematical expressions should you search.
- throwawaygh 5y ago
  Sure. It was an extreme example of the difference between "can't" and "won't", and a point about theory vs practice.
didibus 5y ago
I think that's a great example, the question is: can ML be as good as we are at compiling? And then, if ML is as good as we are, and clearly we're not good enough that we had to write a systematic proof system to compile things correctly, could ML have invented this same technique and implemented that compiler correctly? In the end, we're just back at the same place, can ML be as intelligent as we are. Is it the holy grail of AI? What are its limits?

tudelo 5y ago

Are there not situations where a learned model could be insufficient due to lack of real-time learning? It is my understanding that a lot of current applications take a ton of training time / computing power and it's not so trivial to do in the moment. I guess this is less of a theoretical problem and more of a practical problem though.

throwawaygh 5y ago
> I guess this is less of a theoretical problem and more of a practical problem though. Yeah, exactly. Theoretically, ML can do anything. Practically, not so much.

snek_case 5y ago

The reason you wouldn't want to use ML for program transformations is that compiler analyses like constant folding are essentially fast, specialized theorem provers. When you need to know if an optimization is safe or not (is this expression equivalent to that one?), you want an exact solution, not just a pretty good guess, and there are known techniques that can give you just that. However, you can use ML in areas where compilers use heuristics. For example, how many times should you unroll this loop, or should you inline function B into function A? Your program is going to be valid with or without inlining/unrolling, and the compiler needs an "intuition" of what's going to be best for performance. Right now, in most cases, this uses simple hardcoded rules. The same goes for math. You wouldn't ask a neural network to certify whether a mathematical proof is valid or not, knowing it has been 99% accurate on the test set. You couldn't trust it to verify all the steps. However, you could use a neural network to help you guide a formal theorem prover in searching for a proof, as in which branches of the infinite tree of mathematical expressions should you search.

throwawaygh 5y ago
Sure. It was an extreme example of the difference between "can't" and "won't", and a point about theory vs practice.

didibus 5y ago

I think that's a great example, the question is: can ML be as good as we are at compiling? And then, if ML is as good as we are, and clearly we're not good enough that we had to write a systematic proof system to compile things correctly, could ML have invented this same technique and implemented that compiler correctly? In the end, we're just back at the same place, can ML be as intelligent as we are. Is it the holy grail of AI? What are its limits?