4 ms·

> Do you have an example besides logic/math where it doesn’t understand simple concepts? All the time. It often fails to understand simple concepts. It doesn't

> Do you have an example besides logic/math where it doesn’t understand simple concepts? All the time. It often fails to understand simple concepts. It doesn't really seem to understand anything. For example, try to get it to write some code for a program in a moderately obscure programming language. It's terrible: it will confidently produce stuff, but make errors all over the place. It's unable to understand that it doesn't know the language, and it doesn't know how to ask the right questions to improve. It doesn't have a good model of what it's trying to do, or what you're trying to do. If you point out problems it'll happily try again and repeat the same errors over and over again. What it does is intuit an answer based on the data it's already seen. It's amazingly good at identifying, matching, and combining abstractions that it's already been trained on. This is often good enough for simple tasks, because it has been trained on so much of the world's output that it can frequently map a request to learned concepts, but it's basically a glorified Markov model when it comes to genuinely new or obscure stuff. It's a big step forward, but I think the current approach has a ceiling.

dr_dshiv 3y ago
Sure, so what are the specific concepts it doesn’t understand? I don’t think its ability to program in an obscure program is really a great test. That’s a matter of syntax more than semantics, no? Novel conceptual blends are where it excels. Yes, it needs to understand the concepts involved to blend them —but humans need that too.
- chimprich 3y ago
  I think you missed my point. It's understandable that it doesn't know how to program in a moderately obscure language. But the model doesn't understand that it doesn't. The specific concepts it doesn't understand are understanding what it is, its limitations, and what it's being asked to do. It doesn't seem to have any "meta" understanding. It's subconscious thought only. If I asked a human to program in a language they didn't understand, they'd say they couldn't, or they'd ask for further instructions, or some reference to the documentation, or they'd suggest asking someone else to do it, or they'd eventually figure out how to write in the language by experimenting on small programs and gradually writing more complex ones. GPT4 and friends "just" take an input that seems like it could plausibly answer the request. If it gets it wrong then it just has another go using the same generative technique as before with whatever extra direction the human decides to give it. It doesn't think about the problem. ("just" doing a lot of work in the above sentence: what it does is seriously impressive! But it still seems to be well behind humans in capability.)
  - dr_dshiv 3y ago
    I agree it has very minimal metacognition. That’s partially addressed through prompt chaining—ie, having it reflect critically on its own reasoning. But I agree that it lacks self-awareness. I think artifacts can easily reflect the understanding of the designer (Socrates claims an etymology of Technology from Echo-Nous [1]) But for an artifact to understand — this is entirely dependent on how you operationalize and measure it. Same as with people—we don’t expect people to understand things unless we assess them. And, obviously we need to assess the understanding of machines. It is vitally important to have an assessment of how well it performs on different evals of understanding in different domains. But I have a really interesting supposition about AI understanding that involves it’s ability to access the Platonic world of mathematical forms. I recently read a popular 2016 article on the philosophy of scientific progress. They define scientific progress as increased understanding — and call it the “noetic account.” [2] Thats a bit of theoretical support for the idea that human understanding consists of our ability to conceptualize the world in terms of the Platonic forms. Plato ftw! [1] see his dialogue Cratylus [2] Dellsén, F. (2016). Scientific progress: Knowledge versus understanding. Studies in History and Philosophy of Science Part A, 56, 72-83.
pixl97 3y ago
>, try to get it to write some code for a program in a moderately obscure programming language. It's terrible: it will confidently produce stuff, but make errors all over the place. Is that really any different than asking me to attempt to program in a moderately obscure programming language without a runtime to test my code on? I wouldn't be able to figure out what I don't know without a feedback loop incorporating data. >If you point out problems it'll happily try again and repeat the same errors over and over again. And quite often if you incorporate the correct documentation, it will stop repeating the errors and give a correct answer. It's not a continuous learning model either. It has small token windows where it begins forgetting things. So yea, it has limits far below most humans, but far beyond any we've seen in the past.
- AnimalMuppet 3y ago
  How about this? Flip it into training mode, feed it the language manual for an obscure language, then ask it to write a program in that language? That's a test that many of us here have passed...

dr_dshiv 3y ago

Sure, so what are the specific concepts it doesn’t understand? I don’t think its ability to program in an obscure program is really a great test. That’s a matter of syntax more than semantics, no? Novel conceptual blends are where it excels. Yes, it needs to understand the concepts involved to blend them —but humans need that too.

chimprich 3y ago
I think you missed my point. It's understandable that it doesn't know how to program in a moderately obscure language. But the model doesn't understand that it doesn't. The specific concepts it doesn't understand are understanding what it is, its limitations, and what it's being asked to do. It doesn't seem to have any "meta" understanding. It's subconscious thought only. If I asked a human to program in a language they didn't understand, they'd say they couldn't, or they'd ask for further instructions, or some reference to the documentation, or they'd suggest asking someone else to do it, or they'd eventually figure out how to write in the language by experimenting on small programs and gradually writing more complex ones. GPT4 and friends "just" take an input that seems like it could plausibly answer the request. If it gets it wrong then it just has another go using the same generative technique as before with whatever extra direction the human decides to give it. It doesn't think about the problem. ("just" doing a lot of work in the above sentence: what it does is seriously impressive! But it still seems to be well behind humans in capability.)
- dr_dshiv 3y ago
  I agree it has very minimal metacognition. That’s partially addressed through prompt chaining—ie, having it reflect critically on its own reasoning. But I agree that it lacks self-awareness. I think artifacts can easily reflect the understanding of the designer (Socrates claims an etymology of Technology from Echo-Nous [1]) But for an artifact to understand — this is entirely dependent on how you operationalize and measure it. Same as with people—we don’t expect people to understand things unless we assess them. And, obviously we need to assess the understanding of machines. It is vitally important to have an assessment of how well it performs on different evals of understanding in different domains. But I have a really interesting supposition about AI understanding that involves it’s ability to access the Platonic world of mathematical forms. I recently read a popular 2016 article on the philosophy of scientific progress. They define scientific progress as increased understanding — and call it the “noetic account.” [2] Thats a bit of theoretical support for the idea that human understanding consists of our ability to conceptualize the world in terms of the Platonic forms. Plato ftw! [1] see his dialogue Cratylus [2] Dellsén, F. (2016). Scientific progress: Knowledge versus understanding. Studies in History and Philosophy of Science Part A, 56, 72-83.

pixl97 3y ago

>, try to get it to write some code for a program in a moderately obscure programming language. It's terrible: it will confidently produce stuff, but make errors all over the place. Is that really any different than asking me to attempt to program in a moderately obscure programming language without a runtime to test my code on? I wouldn't be able to figure out what I don't know without a feedback loop incorporating data. >If you point out problems it'll happily try again and repeat the same errors over and over again. And quite often if you incorporate the correct documentation, it will stop repeating the errors and give a correct answer. It's not a continuous learning model either. It has small token windows where it begins forgetting things. So yea, it has limits far below most humans, but far beyond any we've seen in the past.

AnimalMuppet 3y ago
How about this? Flip it into training mode, feed it the language manual for an obscure language, then ask it to write a program in that language? That's a test that many of us here have passed...