3 ms·

Just because you can brute force things doesn't mean it's a practical way to solve certain problems. The set of all C++ programs humans will ever write is finit

Just because you can brute force things doesn't mean it's a practical way to solve certain problems. The set of all C++ programs humans will ever write is finite and therefore parseable with a regular language, but no one's out there writing C++ compilers that way for obvious reasons. That's the essence of what I think GP is getting at. I don't know if NNs are sufficiently powerful escape that argument because my formal understanding of "the world" simply isn't good enough, but it's not obvious to me that they are.

tiborsaas 5y ago
I've seen a case when someone implemented an algorithm after an existing AI was created. I can't remember unfortunately, but the hand made version was better so it's certainly a possibility. NN-s automate human problem solving as it was demonstrated recently by DeepMind's AlphaCode. It can be quite practical though as we failed to come up with solutions that were previously out of reach before deep neural networks (and massive increases in hardware capabilities).
- YeGoblynQueenne 5y ago
  AlphaCode works by first generating many millions of programs from a large language model trained on all of github and fine-tuned on a dataset of problems like the ones it's supposed to be solving. Then, those millions of problems are reduced to several thousand, by "filtering" on input/output examples of the target problem. Finally, these many thousands of examples are further reduced by hand-crafted heuristics, clustering and ranking, that finally select k prorgams, 10 in the paper introducing AlphaCode. This is not a practical approach. It's a maddenlingly resource-intensive approach that spams programs at an insane rate and hopes to get lucky and catch a few that solve a programming task. In the evaluations listed in the AlphaCode paper, it performs awfully against the test set of its training dataset and a pre-existing benchmark. DeepMind have drummed up their systems ability to do about as well as the average participant in a code contest, but that's no measure of coding ability. Deep neural nets are certainly very far from automating human problem solving, like you say in your comment. Your comment is exactly the kind of overhyped misunderstanding of the state-of-the-art that the OP is warning against.
  - tiborsaas 5y ago
    Thanks for the deeper explanation. Even though it's not practical, it's an approach that previously wasn't possible so at least some cheering for it is valid. Problem solving is getting automated, but to what extent is debatable, we are indeed in the early stages. I haven't suggested that it's already AGI.
    - YeGoblynQueenne 5y ago
      >> Even though it's not practical, it's an approach that previously wasn't possible so at least some cheering for it is valid. The approach taken by DeepCode -- generate-and-test -- was always possible and is a staple in inductive program synthesis. DeepCode's version is a particularly crude and backwards-looking attempt at implementing that approach. It's no advance of any kind and it only looks like it is because DeepMind's preprint reporting on it carefully ignores most of what has gone before in inductive program synthesis, the better to show off its new system. It's a scandalous tactic that is a staple of DeepMind's publications and I'm very sorry that I can't find anything positive to say about it. Also, I'm not talking about AGI. At this point, AGI is a fantasy. What we can hope for is to create systems that we can control well, for example to make them learn what we want them to learn and not to overfit to their dataset. In this, also, there seems to be a regression, recently, as the tendency is to train systems end-to-end while trying to remove "the human in the loop" (which is, of course, impossible).
      - tiborsaas 5y ago
        > At this point, AGI is a fantasy. I thought you did, because fully automated problem solving is AGI, but I don't claim that we are anywhere near to reach it. I agree it's probably far away, but it's a fantasy I like to imagine and I don't think it's an impossible goal. Don't be sorry, I appreciate a fair criticism. > What we can hope for is to create systems that we can control well, for example to make them learn what we want them to learn and not to overfit to their dataset. I wish you strong nerves, because it will have AGI written all over it :) Again, this is not a claim, just a prediction.

tiborsaas 5y ago

I've seen a case when someone implemented an algorithm after an existing AI was created. I can't remember unfortunately, but the hand made version was better so it's certainly a possibility. NN-s automate human problem solving as it was demonstrated recently by DeepMind's AlphaCode. It can be quite practical though as we failed to come up with solutions that were previously out of reach before deep neural networks (and massive increases in hardware capabilities).

YeGoblynQueenne 5y ago
AlphaCode works by first generating many millions of programs from a large language model trained on all of github and fine-tuned on a dataset of problems like the ones it's supposed to be solving. Then, those millions of problems are reduced to several thousand, by "filtering" on input/output examples of the target problem. Finally, these many thousands of examples are further reduced by hand-crafted heuristics, clustering and ranking, that finally select k prorgams, 10 in the paper introducing AlphaCode. This is not a practical approach. It's a maddenlingly resource-intensive approach that spams programs at an insane rate and hopes to get lucky and catch a few that solve a programming task. In the evaluations listed in the AlphaCode paper, it performs awfully against the test set of its training dataset and a pre-existing benchmark. DeepMind have drummed up their systems ability to do about as well as the average participant in a code contest, but that's no measure of coding ability. Deep neural nets are certainly very far from automating human problem solving, like you say in your comment. Your comment is exactly the kind of overhyped misunderstanding of the state-of-the-art that the OP is warning against.
- tiborsaas 5y ago
  Thanks for the deeper explanation. Even though it's not practical, it's an approach that previously wasn't possible so at least some cheering for it is valid. Problem solving is getting automated, but to what extent is debatable, we are indeed in the early stages. I haven't suggested that it's already AGI.
  - YeGoblynQueenne 5y ago
    >> Even though it's not practical, it's an approach that previously wasn't possible so at least some cheering for it is valid. The approach taken by DeepCode -- generate-and-test -- was always possible and is a staple in inductive program synthesis. DeepCode's version is a particularly crude and backwards-looking attempt at implementing that approach. It's no advance of any kind and it only looks like it is because DeepMind's preprint reporting on it carefully ignores most of what has gone before in inductive program synthesis, the better to show off its new system. It's a scandalous tactic that is a staple of DeepMind's publications and I'm very sorry that I can't find anything positive to say about it. Also, I'm not talking about AGI. At this point, AGI is a fantasy. What we can hope for is to create systems that we can control well, for example to make them learn what we want them to learn and not to overfit to their dataset. In this, also, there seems to be a regression, recently, as the tendency is to train systems end-to-end while trying to remove "the human in the loop" (which is, of course, impossible).
    - tiborsaas 5y ago
      > At this point, AGI is a fantasy. I thought you did, because fully automated problem solving is AGI, but I don't claim that we are anywhere near to reach it. I agree it's probably far away, but it's a fantasy I like to imagine and I don't think it's an impossible goal. Don't be sorry, I appreciate a fair criticism. > What we can hope for is to create systems that we can control well, for example to make them learn what we want them to learn and not to overfit to their dataset. I wish you strong nerves, because it will have AGI written all over it :) Again, this is not a claim, just a prediction.