3 ms·

> I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made My poi

> I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made My point of view on this is that this type of AI cannot reason, so it seems to me quite audacious to use it for programming. Following the AI's suggestion is like examining a PR from someone you don't know: don't they try to insert a backdoor or a subtle bug? Even if you use it as some kind of "rubber duck", examining its suggestions - doing the necessary reasoning and the double checks - is pretty time consuming. Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)? It feels like we are in the equivalent of programming language's "honey moon" [1] phase with this kind of technology. People will eventually get over the "wow" effect and use it for what it is good at. [1] https://earthly.dev/blog/brown-green-language/ https://earthly.dev/blog/brown-green-language/

eropple 3y ago
For making new stuff, I agree with you that it's pretty nutty. I'd never ask it to try to invent something novel, the way this does, because it's pretty high-noise. I do think it has significant programming advantages insofar as it can analyze existing code and make reasonable permutations of it, though, especially for stuff that's able to effectively analyze your current set of code. "Would it be better to acquire a deeper knowledge"--for the cases described above, actually no? I already have that deeper knowledge. I don't have the attention span (both because this stuff can be boring and because I am frequently randomized) and for a lot of this stuff the gate is actually not solving the problem but the physical act of typing, or even copying and changing. Copilot, at least, absolutely does create bugs, but in my experience they're mostly bugs of the form "this actually isn't typesafe in the way that it's written" and that makes isolating and fixing them clear. (Rarely are those bugs logic bugs, just a burp in the output.) There is a very clear future where this turns into incredible copy-paste garbage because it's not capable of refactoring, and that's a different problem. Avoiding it is why they pay us the medium bucks.
chii 3y ago
> My point of view on this is that this type of AI cannot reason but it can answer the glass door push or pull question (see https://news.ycombinator.com/item?id=35506666 https://news.ycombinator.com/item?id=35506666), albeit a bit off (it didn't understand the mirrored writing). But some other models had gotten the right answer. This points to some evidence of being able to derive logical reasoning. Just because the model is "doing statistical guessing of the next word", doesn't mean this process does not produce logical reasoning as an outcome.
- vidarh 3y ago
  It's also not at all clear that reasoning is anything more than "doing statistical guessing of the next word" or something very similar. We clearly have some other aspects, but it's not clear it's all that much beyond ongoing stimulus and memory in terms of capabilities.
- sfn42 3y ago
  It can only answer questions it has been trained to know the answers to. Ask it an original question that isn't in the training set and it gets it wrong. This is clear evidence that it does not possess logical reasoning skills.
  - vidarh 3y ago
    This is simply not true. I've asked it many questions it has not been trained to know the answers to because they involve e.g. code it won't have seen in the wild, and gotten the right answer back. In one of my other responses I dug into how I'd asked it to analyse a piece of assembly generated with a compiler I've written, from a test program that didn't exist at GPT4's knowledge cutoff, and it did just fine.
    - sfn42 3y ago
      I'm not saying it has to be trained on the specific question directly, it has to be trained on similar data. It has a decent knowledge of programming languages because it has been trained on a lot of data surrounding them. Being capable of logic and reasoning is more of an illusion that arises as a result of the enormous amount of training they put in. You can make simple questions that it can't answer, so you can't claim it has logical reasoning if it fails at simple tasks. It just looks a lot like it has logical reasoning because it has been trained on text displaying logical reasoning.
vidarh 3y ago
I've had plenty of luck using it for programming, and getting it to output working code. It's still best for things that is either boilerplate or where it's easy to test, but I've gotten it to write e.g. parts of parsers, a DNS server, and a working web app and many other things. I wouldn't trust complex pieces of code that I can't quickly verify, but I wouldn't do that with a human developer either. Getting it to symbolically evaluate code it most certainly have never seen before has also worked well for me, as have getting to explain the hypothetical outcome of executing a piece of code written using a made up combination of two languages. Frankly, I've worked with plenty of developers who'd struggle to reason about simpler tasks than some of the stuff I've successfully thrown at GPT4, so I'd argue that if it can't reason then many humans can't either. It also fails at some very basic things. The biggest challenge is that it's reasoning is different to humans, and it's terrible at some things we consider basic even as it can do other things at a far higher level. > Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)? "Prompt engineering" works best in fields you do have a deep knowledge of, where you can tell whether you're actually getting better solutions. I'm willing to put effort into that too is acquiring deeper knowledge of the field by exploring dependencies and getting an understanding of ambiguities of language we use to talk about a problem. While you can get bogged down in specific "incantations", the more important part of "prompt engineering" is getting better at recognising under-specified problems. In fact, one of the things I've toyed with is to feed it explanations of a problem and instead of asking it to solve it, I first ask it to suggest improvements and ask questions, and that will often reveal problems with the prompts with little effort. With respect to assembler, it seems to hit one of the bad spots, in that you get far fewer textual structural hints. E.g. a label can be the start of a function or the start of a basic block, or part of a loop, or part of an if/else block, and you largely need to recognise contextual clues and know the patterns of whatever produced the asm to know which rules applies. E.g. one of the issues I ran into was that the asm I gave it was output from a compiler that used %esi in a very specific way. There was no way for GPT4 to know that from the sample it was given without "thinking it through" (see below), and it made a "guess" that many novice asm programmers also likely would've, but where someone more experienced would've stopped and demanded a confirmation of the calling convention. That said, to the test I mentioned in the previous comment, I tried giving it the code in a fresh chat, and asked it to reason about the code, and to give me a list of information that I could provide that would make it easier to suggest optimisations. When I explicitly asked it about the calling convention used for %esi in the code I gave it, it correctly reasoned its way through it by observing how it was treated within the given code. That's something an smart beginner at asm might spot, and an intermediate asm programmer pretty likely would spot, but the important part is that it was clearly able to explain why with references to the code. It also when asked correctly recognised that the object was tagged (low bit used to indicate it's not a "real" pointer but a value object), and recognised vtable calling pattern that few developers without compiler experience would pick up on. Frankly, after asking it these follow up questions my opinions on its understanding of asm is way up. It won't beat an experienced asm developer, but most developers I can ask questions of are not experienced asm developers, and even fewer knows the patterns it easily explained and reasoned about.

eropple 3y ago

For making new stuff, I agree with you that it's pretty nutty. I'd never ask it to try to invent something novel, the way this does, because it's pretty high-noise. I do think it has significant programming advantages insofar as it can analyze existing code and make reasonable permutations of it, though, especially for stuff that's able to effectively analyze your current set of code. "Would it be better to acquire a deeper knowledge"--for the cases described above, actually no? I already have that deeper knowledge. I don't have the attention span (both because this stuff can be boring and because I am frequently randomized) and for a lot of this stuff the gate is actually not solving the problem but the physical act of typing, or even copying and changing. Copilot, at least, absolutely does create bugs, but in my experience they're mostly bugs of the form "this actually isn't typesafe in the way that it's written" and that makes isolating and fixing them clear. (Rarely are those bugs logic bugs, just a burp in the output.) There is a very clear future where this turns into incredible copy-paste garbage because it's not capable of refactoring, and that's a different problem. Avoiding it is why they pay us the medium bucks.

chii 3y ago

> My point of view on this is that this type of AI cannot reason but it can answer the glass door push or pull question (see https://news.ycombinator.com/item?id=35506666 https://news.ycombinator.com/item?id=35506666), albeit a bit off (it didn't understand the mirrored writing). But some other models had gotten the right answer. This points to some evidence of being able to derive logical reasoning. Just because the model is "doing statistical guessing of the next word", doesn't mean this process does not produce logical reasoning as an outcome.

vidarh 3y ago
It's also not at all clear that reasoning is anything more than "doing statistical guessing of the next word" or something very similar. We clearly have some other aspects, but it's not clear it's all that much beyond ongoing stimulus and memory in terms of capabilities.
sfn42 3y ago
It can only answer questions it has been trained to know the answers to. Ask it an original question that isn't in the training set and it gets it wrong. This is clear evidence that it does not possess logical reasoning skills.
- vidarh 3y ago
  This is simply not true. I've asked it many questions it has not been trained to know the answers to because they involve e.g. code it won't have seen in the wild, and gotten the right answer back. In one of my other responses I dug into how I'd asked it to analyse a piece of assembly generated with a compiler I've written, from a test program that didn't exist at GPT4's knowledge cutoff, and it did just fine.
  - sfn42 3y ago
    I'm not saying it has to be trained on the specific question directly, it has to be trained on similar data. It has a decent knowledge of programming languages because it has been trained on a lot of data surrounding them. Being capable of logic and reasoning is more of an illusion that arises as a result of the enormous amount of training they put in. You can make simple questions that it can't answer, so you can't claim it has logical reasoning if it fails at simple tasks. It just looks a lot like it has logical reasoning because it has been trained on text displaying logical reasoning.

vidarh 3y ago

I've had plenty of luck using it for programming, and getting it to output working code. It's still best for things that is either boilerplate or where it's easy to test, but I've gotten it to write e.g. parts of parsers, a DNS server, and a working web app and many other things. I wouldn't trust complex pieces of code that I can't quickly verify, but I wouldn't do that with a human developer either. Getting it to symbolically evaluate code it most certainly have never seen before has also worked well for me, as have getting to explain the hypothetical outcome of executing a piece of code written using a made up combination of two languages. Frankly, I've worked with plenty of developers who'd struggle to reason about simpler tasks than some of the stuff I've successfully thrown at GPT4, so I'd argue that if it can't reason then many humans can't either. It also fails at some very basic things. The biggest challenge is that it's reasoning is different to humans, and it's terrible at some things we consider basic even as it can do other things at a far higher level. > Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)? "Prompt engineering" works best in fields you do have a deep knowledge of, where you can tell whether you're actually getting better solutions. I'm willing to put effort into that too is acquiring deeper knowledge of the field by exploring dependencies and getting an understanding of ambiguities of language we use to talk about a problem. While you can get bogged down in specific "incantations", the more important part of "prompt engineering" is getting better at recognising under-specified problems. In fact, one of the things I've toyed with is to feed it explanations of a problem and instead of asking it to solve it, I first ask it to suggest improvements and ask questions, and that will often reveal problems with the prompts with little effort. With respect to assembler, it seems to hit one of the bad spots, in that you get far fewer textual structural hints. E.g. a label can be the start of a function or the start of a basic block, or part of a loop, or part of an if/else block, and you largely need to recognise contextual clues and know the patterns of whatever produced the asm to know which rules applies. E.g. one of the issues I ran into was that the asm I gave it was output from a compiler that used %esi in a very specific way. There was no way for GPT4 to know that from the sample it was given without "thinking it through" (see below), and it made a "guess" that many novice asm programmers also likely would've, but where someone more experienced would've stopped and demanded a confirmation of the calling convention. That said, to the test I mentioned in the previous comment, I tried giving it the code in a fresh chat, and asked it to reason about the code, and to give me a list of information that I could provide that would make it easier to suggest optimisations. When I explicitly asked it about the calling convention used for %esi in the code I gave it, it correctly reasoned its way through it by observing how it was treated within the given code. That's something an smart beginner at asm might spot, and an intermediate asm programmer pretty likely would spot, but the important part is that it was clearly able to explain why with references to the code. It also when asked correctly recognised that the object was tagged (low bit used to indicate it's not a "real" pointer but a value object), and recognised vtable calling pattern that few developers without compiler experience would pick up on. Frankly, after asking it these follow up questions my opinions on its understanding of asm is way up. It won't beat an experienced asm developer, but most developers I can ask questions of are not experienced asm developers, and even fewer knows the patterns it easily explained and reasoned about.