4 ms·

It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. That's what is happening here. There is n

It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion. Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those solutions, it fails.

Veedrac 5y ago
It got the value of the sixth and seventeenth bits, moved them into the right positions, and inserted them into the original value. Off a one-line description written in English! I really cannot empathize with the idea that this is not a meaningful capability. If intelligence only means to you “equal in all capabilities to an experienced human”, you are never going to be able to see anything coming ever.
- 37ef_ced3 5y ago
  If you ask CoPilot to solve something it hasn't seen, it won't be able to solve it. It's a transformer. Do you understand what that means? It's just matrix multiplication. It generates maximum likelihood token strings, based on its training data. It doesn't "understand" what those token string mean. You are amazed because you're testing the transformer by asking the transformer to generate human-written code THAT IT WAS TRAINED ON. To make CoPilot fail, all you have to do is ask it to generate something unlikely, something it hasn't seen in training. Maximum likelihood token strings. Period.
hackinthebochs 5y ago
>It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. Funny, today I was just thinking of people's tendencies to dismiss AI advances with this very pattern of reasoning: take a reductive description of the system and then dismiss it as obviously insufficient for understanding or whatever the target is. The assumption is that understanding is fundamentally non-reductive, or that there is insufficient complexity contained within the reductive description. But this is a mistake. The fallacy is that the reductive description is glossing over the source of the complexity, and hence where the capabilities of the model reside. "Generating maximum likelihood token strings" doesn't capture the complexity of the process that generates the token strings, and so an argument that is premised on this reductive description cannot prove the model deficient. For example, the best way to generate maximum likelihood human text is just to simulate a human mind. Genuine understanding is within the solution-space of the problem definition in terms of maximum likelihood strings, thus you cannot dismiss the model based on this reductive description.
- 37ef_ced3 5y ago
  The difference between me and you is that I implement neural nets professionally. Here is one of my (non-professional) open source projects: https://NN-512.com https://NN-512.com I'm sure if you understood what the transformer was doing, you would be less impressed.
  - hackinthebochs 5y ago
    This is the wrong context to go with an appeal to authority. I know what the transformer is doing, I've also developed neural networks before (though not professionally). Your experience is working against you in developing your intuition. There's another common fallacy that because we're somehow "inside" the system, that we understand exactly what is going on, or in this case what isn't going on. Language models are composed of variations of matrix multiplications, but that isn't a complete description of their behavior. It's like saying because we've looked inside the brain and there's just electrical and chemical signals, the mind must reside somewhere else. It's just a specious argument.

Veedrac 5y ago

It got the value of the sixth and seventeenth bits, moved them into the right positions, and inserted them into the original value. Off a one-line description written in English! I really cannot empathize with the idea that this is not a meaningful capability. If intelligence only means to you “equal in all capabilities to an experienced human”, you are never going to be able to see anything coming ever.

37ef_ced3 5y ago
If you ask CoPilot to solve something it hasn't seen, it won't be able to solve it. It's a transformer. Do you understand what that means? It's just matrix multiplication. It generates maximum likelihood token strings, based on its training data. It doesn't "understand" what those token string mean. You are amazed because you're testing the transformer by asking the transformer to generate human-written code THAT IT WAS TRAINED ON. To make CoPilot fail, all you have to do is ask it to generate something unlikely, something it hasn't seen in training. Maximum likelihood token strings. Period.

hackinthebochs 5y ago

>It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. Funny, today I was just thinking of people's tendencies to dismiss AI advances with this very pattern of reasoning: take a reductive description of the system and then dismiss it as obviously insufficient for understanding or whatever the target is. The assumption is that understanding is fundamentally non-reductive, or that there is insufficient complexity contained within the reductive description. But this is a mistake. The fallacy is that the reductive description is glossing over the source of the complexity, and hence where the capabilities of the model reside. "Generating maximum likelihood token strings" doesn't capture the complexity of the process that generates the token strings, and so an argument that is premised on this reductive description cannot prove the model deficient. For example, the best way to generate maximum likelihood human text is just to simulate a human mind. Genuine understanding is within the solution-space of the problem definition in terms of maximum likelihood strings, thus you cannot dismiss the model based on this reductive description.

37ef_ced3 5y ago
The difference between me and you is that I implement neural nets professionally. Here is one of my (non-professional) open source projects: https://NN-512.com https://NN-512.com I'm sure if you understood what the transformer was doing, you would be less impressed.
- hackinthebochs 5y ago
  This is the wrong context to go with an appeal to authority. I know what the transformer is doing, I've also developed neural networks before (though not professionally). Your experience is working against you in developing your intuition. There's another common fallacy that because we're somehow "inside" the system, that we understand exactly what is going on, or in this case what isn't going on. Language models are composed of variations of matrix multiplications, but that isn't a complete description of their behavior. It's like saying because we've looked inside the brain and there's just electrical and chemical signals, the mind must reside somewhere else. It's just a specious argument.