4 ms·

> If you cannot suggest any way its output would look different to how it looks now and instead have to resort to changing the subject, shouldn't you retract th

> If you cannot suggest any way its output would look different to how it looks now and instead have to resort to changing the subject, shouldn't you retract that claim? For one thing, I expect we’d not see so many cases of them chasing (if you will) the prompt and request into silliness. The code attempts to satisfy prompts in a transparently mechanical fashion, which is part of why they so gleefully (if you will) mislead. There’s no understanding. You can ask them to correct and they might, but they can also be induced to correct the already-correct, so that means nothing. To the extent we fix that, it’s not by adding any factor that might represent understanding, it’s further prompting that amounts to “follow these patterns slightly differently”. The fix isn’t, so far, “teaching” them to understand. Maybe we’ll get there! But we don’t appear to be anywhere near that yet. > Then you must say a human doesn't understand. For what else is there in a human brain except a finite amount of learned behavioural rules for signal inputs and outputs? Learned over a billion years of evolution in the structure, and filled in by a lifetime of nurture. The thing about the Chinese Room is that we comprehend the entire process, and there’s no room for some unknown factor affecting the output—or for a known factor that might be processing something like what we mean by understanding (let alone consciousness, say). Every single part of what an LLM does can be replicated with big books of lookup tables, dice, and a list of rules. There’s nowhere for anything to do the understanding to exist. It’s not that we have to be confused by part of it for that to be there—I’m not saying mystery is a necessary component—just that this process doesn’t have a place for that to be.

jodrellblank 2y ago
> "For one thing, I expect we’d not see so many cases of them chasing (if you will) the prompt and request into silliness." In the one example I gave you saw one output and declared it "not understanding the concept of bridging". I'm asking specifically that output, how would it look different if ChatGPT had some understanding of the concept of bridging? You're back to arguing "it's not human level!" which was not my claim. My claim is that it's above zero level. In another comment I asked it to use the concept of bridging in new ways, and it provided sentences which have no hits on Google but are plausibly the kind of thing I might see in a book from a human author. > "There’s no understanding" Say to your pet "I like it when you do human-like things such as standing on two feet. Come up with more human-style things for more treats" and it won't. You can ask ChatGPT to come up with more uses of the bridging concept, and it does. That is demonstrating understanding at higher than rock level and higher than rat level, and you can't reject that evidence just by repeatedly saying "there's no understanding there's no understanding there's no understanding". > "they can also be induced to correct the already-correct, so that means nothing." So can I; if my boss tells me there is an error and I need to correct it, I might correct a non-error to please them. Knowingly ("I'll change this part from correct to wrong if that pleases them") or unknowingly ("if they tell me there is an error there must be one, I'll take a guess that this bit is wrong and put something else here"). Does that show I have no understanding? > "Every single part of what an LLM does can be replicated with big books of lookup tables, dice, and a list of rules. There’s nowhere for anything to do the understanding to exist." You're doing the God of the Gaps argument with the human brain - an LED screen is RGB pixels, there's nowhere for a picture of a cat to exist separately from bright and dark pixels. A book is printed characters, there's nowhere for a story to exist separate from blobs of ink on paper. A brain is meat grown from a foetus, uses ~20 Watts of energy, if the blood supply is cutoff then it dies, if it gets too hot or cold then it dies, there are many areas which can be damaged and harm something like leg movement but there is no single area which can be damaged which stops 'understanding' but leaves everything else unchanged, there are no examples of people being decapitated, having no brain, having brain death, and still having 'understanding' provided by whatever other thing you are implying exists and does understanding. There's nowhere for anything to do the understanding to exist, unless there is a) new physics which aligns perfectly with every observation we have about the brain but also augments it an adds some magical 'understanding' thing which can't be done or simulated in software. b) something non-physical such as a soul which is tied closely to the meat and powered by the food and blood and can't be tied to silicon because reasons. c) ??? As far as I can see this isn't reasoning from anything more convincing than you not wanting to accept the Occam's Razor simpler explanation that a purely physical information processing system can understand. (Or that humans don't understand and it's all some weird illusion; the picture of the cat is not in the LED screen, it is in the eye of the beholder. The understanding isn't in your behaviour, it's in the beholder's interpretation, I believe you understand because you demonstrate the behaviours of understanding. We are seeing intelligence in others where there isn't any. And that view turned on ourselves is our own perception of our own understanding - we see ourselves identifying patterns, extrapolating patterns, continuing coherent sentences, and conclude that we must have 'understanding' as a thing separate from those behaviours). > "The thing about the Chinese Room is that we comprehend the entire process, and there’s no room for some unknown factor affecting the output" We don't comprehend the entire Chinese Room; the instructions that Searle is following are a massive handwave. Does following the instructions require Searle to make human judgements on where to branch? Then it's offloading understanding onto his human brain. Does it not require that but it still outputs coherent responses? Then the instructions must encode intelligence in them in some way - if intelligent behaviour doesn't demonstrate intelligence we're in non-scientific nonsense land. Peter Cochrane wrote about 'dying by installments' of a human turned into a cyborg replaced bit by bit, Ship of Theseus style. We can do similar and make up a Cochrane's Chinese Brain - instead of a neuron firing and affecting the connected neurons, it raises an alert and Searle walks over and writes down the firing pattern on a scratch pad, walks to all the other relevant neurons, and taps in the firing pattern on an input device, without understanding the information content of the firing pattern. Does the brain keep responding coherent Chinese but no longer understand Chinese?
- vundercind 2y ago
  I’m not doing God of the Gaps—it’s simply that in the case of LLMs there’s both nothing we know of doing understanding and also no gaps where it might be. We both know what it’s doing, and that it’s not doing understanding. Let’s try this: We could apply an LLM to made-up language and corpus that does not actually carry meaning and it would do exactly what it does with real languages. “Well maybe you accidentally encoded meaning in it. We could always, say, cryptoanalyze even an alien language and maybe be able to come up with some good guesses at meaning” Maybe we could. But now imagine also you have no “knowledge” whatsoever except the trained patterns from that language. Like, no understanding of how to do cryptoanalysis, or linguistics, or what a planet is. Or an alien. All you’re doing is guessing at patterns, based on symbols that you aren’t even attempting to understand and have no basis for understanding anyway. That’s an LLM. I think people are assigning way too much power to language sans… all the rest of what you need to derive meaning from it. None of what’s going into or coming out of an LLM needs to carry any meaning for it to do exactly the same thing it does with languages that do. To the extent that an LLM has a perspective (this is purely figurative) all languages are gibberish alien languages, while also being all that it “knows”. > We don't comprehend the entire Chinese Room; the instructions that Searle is following are a massive handwave. Does following the instructions require Searle to make human judgements on where to branch? Then it's offloading understanding onto his human brain. Does it not require that but it still outputs coherent responses? Then the instructions must encode intelligence in them in some way - if intelligent behaviour doesn't demonstrate intelligence we're in non-scientific nonsense land. I remain stubbornly unconvinced that simulating a real process (by hand or otherwise) is the same thing as it actually happening with real matter and energy, even setting aside that the most efficient way to achieve it is to… not simulate it, and use real matter to actually do the things. It’s why I find the xkcd “what if a guy with infinite time and an infinite beach and infinite rocks moved the rocks around in a way that he had decided simulated a universe?” thing interesting as an example but also trivial to solve: all that happens is he moved some rocks around. The meaning was all his, it doesn’t do anything.
  - jodrellblank 2y ago
    > "thing interesting as an example but also trivial to solve: all that happens is he moved some rocks around. The meaning was all his, it doesn’t do anything." You opened by saying you aren't doing God of the Gaps, but here you are doing it. Brains move chemicals and electrical signals around. That doesn't do anything, apparently. Matter doesn't do understanding. Energy doesn't do understanding. Mathematical calculations don't do understanding. Neural networks don't do understanding. See how Understanding is retreating into the gaps? Brains must have something else, somewhere else, which does understanding? But what, and where? It's a position that becomes less tenable every decade as brains get mapped in finer detail leaving smaller gaps, and non-brains get more and better Human-like abilities > "there’s both nothing we know of doing understanding .. it’s not doing understanding." It is. The math and the training and the inference is the thing doing understanding. Identifying patterns and being able to apply them is part of what understanding is, and that's what it's doing. [Not human level understanding]. > "We could apply an LLM to made-up language and corpus that does not actually carry meaning and it would do exactly what it does with real languages." We do that with language too; the bouba/kiki effect[1] is humans finding meaning in words where there isn't any. We look at the Moon and see a face in it: Pareidolia[2] is 'the tendency for perception to impose a meaningful interpretation on a nebulous stimulus so that one detects an object, pattern, or meaning where there is none'. We are only able to see faces in things because we have some understanding of what it means for something to 'look like a human face'. "We see a face where there isn't one" is no evidence that we don't understand faces and so "an LLM would find patterns in gibberish" is no evidence that LLMs don't understand anything. > "All you’re doing is guessing at patterns, based on symbols that you aren’t even attempting to understand and have no basis for understanding anyway. That’s an LLM." Trying to build patterns is "what attempting to understand" is! You're staring right at the thing happening, and declaring that it isn't happening. "AI is search" said Peter Norvig. The Hutter Prize[3] says "Being able to compress well is closely related to intelligence as explained below. While intelligence is a slippery concept, file sizes are hard numbers. Wikipedia is an extensive snapshot of Human Knowledge. If you can compress the first 1GB of Wikipedia better than your predecessors, your (de)compressor likely has to be smart(er). The intention of this prize is to encourage development of intelligent compressors/programs as a path to AGI". Compression is about searching for patterns. Understanding is either magic, or it functions in some way. Why not this way? > "all languages are gibberish alien languages, while also being all that it “knows”." If we took some writing in a Human language that you don't speak, you can do as much "predict the next word" as you want, take as much time as you need, and put together as an output. The input is asking for a reply in formal Swahili which explains yoga in the style of Tolkein with Tourette's, but you don't know that. The chance of you being able to hit a valid reply out of all possible replies by guessing is absolutely zilch. But you couldn't do it by " predicting the next word" either, how would you predict that the reply should be in Turkish if you can't understand the input? How would you do formal Turkish without understanding the way people use Turkish? Conversely if you could hit on a good and appropriate reply, it would be because your studying to "predicting the next word" had given you some understanding of the input language and Swahili and yoga and Tolkein's style and how Tourette's changes things. > "I remain stubbornly unconvinced that simulating a real process (by hand or otherwise) is the same thing as it actually happening with real matter and energy" Computers are real matter and energy. When someone has a cochlear implant, do you think they aren't really hearing because a microphone turning movement into modulated electricity is fake matter and fake energy, and an eardrum and bones doing it is real matter and real energy? Yes it's true that you can't get on a simulation of a plane and fly to New York, but if you see the output of an arithmetic calculation there's no way to tell if it was done with a redstone computer in Minecraft or with Python or with brain matter. (Is it possible for arithmetic to be not-simulated?). [1] https://en.wikipedia.org/wiki/Bouba/kiki_effect https://en.wikipedia.org/wiki/Bouba/kiki_effect [2] https://en.wikipedia.org/wiki/Pareidolia https://en.wikipedia.org/wiki/Pareidolia [3] http://prize.hutter1.net/ http://prize.hutter1.net/

jodrellblank 2y ago

> "For one thing, I expect we’d not see so many cases of them chasing (if you will) the prompt and request into silliness." In the one example I gave you saw one output and declared it "not understanding the concept of bridging". I'm asking specifically that output, how would it look different if ChatGPT had some understanding of the concept of bridging? You're back to arguing "it's not human level!" which was not my claim. My claim is that it's above zero level. In another comment I asked it to use the concept of bridging in new ways, and it provided sentences which have no hits on Google but are plausibly the kind of thing I might see in a book from a human author. > "There’s no understanding" Say to your pet "I like it when you do human-like things such as standing on two feet. Come up with more human-style things for more treats" and it won't. You can ask ChatGPT to come up with more uses of the bridging concept, and it does. That is demonstrating understanding at higher than rock level and higher than rat level, and you can't reject that evidence just by repeatedly saying "there's no understanding there's no understanding there's no understanding". > "they can also be induced to correct the already-correct, so that means nothing." So can I; if my boss tells me there is an error and I need to correct it, I might correct a non-error to please them. Knowingly ("I'll change this part from correct to wrong if that pleases them") or unknowingly ("if they tell me there is an error there must be one, I'll take a guess that this bit is wrong and put something else here"). Does that show I have no understanding? > "Every single part of what an LLM does can be replicated with big books of lookup tables, dice, and a list of rules. There’s nowhere for anything to do the understanding to exist." You're doing the God of the Gaps argument with the human brain - an LED screen is RGB pixels, there's nowhere for a picture of a cat to exist separately from bright and dark pixels. A book is printed characters, there's nowhere for a story to exist separate from blobs of ink on paper. A brain is meat grown from a foetus, uses ~20 Watts of energy, if the blood supply is cutoff then it dies, if it gets too hot or cold then it dies, there are many areas which can be damaged and harm something like leg movement but there is no single area which can be damaged which stops 'understanding' but leaves everything else unchanged, there are no examples of people being decapitated, having no brain, having brain death, and still having 'understanding' provided by whatever other thing you are implying exists and does understanding. There's nowhere for anything to do the understanding to exist, unless there is a) new physics which aligns perfectly with every observation we have about the brain but also augments it an adds some magical 'understanding' thing which can't be done or simulated in software. b) something non-physical such as a soul which is tied closely to the meat and powered by the food and blood and can't be tied to silicon because reasons. c) ??? As far as I can see this isn't reasoning from anything more convincing than you not wanting to accept the Occam's Razor simpler explanation that a purely physical information processing system can understand. (Or that humans don't understand and it's all some weird illusion; the picture of the cat is not in the LED screen, it is in the eye of the beholder. The understanding isn't in your behaviour, it's in the beholder's interpretation, I believe you understand because you demonstrate the behaviours of understanding. We are seeing intelligence in others where there isn't any. And that view turned on ourselves is our own perception of our own understanding - we see ourselves identifying patterns, extrapolating patterns, continuing coherent sentences, and conclude that we must have 'understanding' as a thing separate from those behaviours). > "The thing about the Chinese Room is that we comprehend the entire process, and there’s no room for some unknown factor affecting the output" We don't comprehend the entire Chinese Room; the instructions that Searle is following are a massive handwave. Does following the instructions require Searle to make human judgements on where to branch? Then it's offloading understanding onto his human brain. Does it not require that but it still outputs coherent responses? Then the instructions must encode intelligence in them in some way - if intelligent behaviour doesn't demonstrate intelligence we're in non-scientific nonsense land. Peter Cochrane wrote about 'dying by installments' of a human turned into a cyborg replaced bit by bit, Ship of Theseus style. We can do similar and make up a Cochrane's Chinese Brain - instead of a neuron firing and affecting the connected neurons, it raises an alert and Searle walks over and writes down the firing pattern on a scratch pad, walks to all the other relevant neurons, and taps in the firing pattern on an input device, without understanding the information content of the firing pattern. Does the brain keep responding coherent Chinese but no longer understand Chinese?

vundercind 2y ago
I’m not doing God of the Gaps—it’s simply that in the case of LLMs there’s both nothing we know of doing understanding and also no gaps where it might be. We both know what it’s doing, and that it’s not doing understanding. Let’s try this: We could apply an LLM to made-up language and corpus that does not actually carry meaning and it would do exactly what it does with real languages. “Well maybe you accidentally encoded meaning in it. We could always, say, cryptoanalyze even an alien language and maybe be able to come up with some good guesses at meaning” Maybe we could. But now imagine also you have no “knowledge” whatsoever except the trained patterns from that language. Like, no understanding of how to do cryptoanalysis, or linguistics, or what a planet is. Or an alien. All you’re doing is guessing at patterns, based on symbols that you aren’t even attempting to understand and have no basis for understanding anyway. That’s an LLM. I think people are assigning way too much power to language sans… all the rest of what you need to derive meaning from it. None of what’s going into or coming out of an LLM needs to carry any meaning for it to do exactly the same thing it does with languages that do. To the extent that an LLM has a perspective (this is purely figurative) all languages are gibberish alien languages, while also being all that it “knows”. > We don't comprehend the entire Chinese Room; the instructions that Searle is following are a massive handwave. Does following the instructions require Searle to make human judgements on where to branch? Then it's offloading understanding onto his human brain. Does it not require that but it still outputs coherent responses? Then the instructions must encode intelligence in them in some way - if intelligent behaviour doesn't demonstrate intelligence we're in non-scientific nonsense land. I remain stubbornly unconvinced that simulating a real process (by hand or otherwise) is the same thing as it actually happening with real matter and energy, even setting aside that the most efficient way to achieve it is to… not simulate it, and use real matter to actually do the things. It’s why I find the xkcd “what if a guy with infinite time and an infinite beach and infinite rocks moved the rocks around in a way that he had decided simulated a universe?” thing interesting as an example but also trivial to solve: all that happens is he moved some rocks around. The meaning was all his, it doesn’t do anything.
- jodrellblank 2y ago
  > "thing interesting as an example but also trivial to solve: all that happens is he moved some rocks around. The meaning was all his, it doesn’t do anything." You opened by saying you aren't doing God of the Gaps, but here you are doing it. Brains move chemicals and electrical signals around. That doesn't do anything, apparently. Matter doesn't do understanding. Energy doesn't do understanding. Mathematical calculations don't do understanding. Neural networks don't do understanding. See how Understanding is retreating into the gaps? Brains must have something else, somewhere else, which does understanding? But what, and where? It's a position that becomes less tenable every decade as brains get mapped in finer detail leaving smaller gaps, and non-brains get more and better Human-like abilities > "there’s both nothing we know of doing understanding .. it’s not doing understanding." It is. The math and the training and the inference is the thing doing understanding. Identifying patterns and being able to apply them is part of what understanding is, and that's what it's doing. [Not human level understanding]. > "We could apply an LLM to made-up language and corpus that does not actually carry meaning and it would do exactly what it does with real languages." We do that with language too; the bouba/kiki effect[1] is humans finding meaning in words where there isn't any. We look at the Moon and see a face in it: Pareidolia[2] is 'the tendency for perception to impose a meaningful interpretation on a nebulous stimulus so that one detects an object, pattern, or meaning where there is none'. We are only able to see faces in things because we have some understanding of what it means for something to 'look like a human face'. "We see a face where there isn't one" is no evidence that we don't understand faces and so "an LLM would find patterns in gibberish" is no evidence that LLMs don't understand anything. > "All you’re doing is guessing at patterns, based on symbols that you aren’t even attempting to understand and have no basis for understanding anyway. That’s an LLM." Trying to build patterns is "what attempting to understand" is! You're staring right at the thing happening, and declaring that it isn't happening. "AI is search" said Peter Norvig. The Hutter Prize[3] says "Being able to compress well is closely related to intelligence as explained below. While intelligence is a slippery concept, file sizes are hard numbers. Wikipedia is an extensive snapshot of Human Knowledge. If you can compress the first 1GB of Wikipedia better than your predecessors, your (de)compressor likely has to be smart(er). The intention of this prize is to encourage development of intelligent compressors/programs as a path to AGI". Compression is about searching for patterns. Understanding is either magic, or it functions in some way. Why not this way? > "all languages are gibberish alien languages, while also being all that it “knows”." If we took some writing in a Human language that you don't speak, you can do as much "predict the next word" as you want, take as much time as you need, and put together as an output. The input is asking for a reply in formal Swahili which explains yoga in the style of Tolkein with Tourette's, but you don't know that. The chance of you being able to hit a valid reply out of all possible replies by guessing is absolutely zilch. But you couldn't do it by " predicting the next word" either, how would you predict that the reply should be in Turkish if you can't understand the input? How would you do formal Turkish without understanding the way people use Turkish? Conversely if you could hit on a good and appropriate reply, it would be because your studying to "predicting the next word" had given you some understanding of the input language and Swahili and yoga and Tolkein's style and how Tourette's changes things. > "I remain stubbornly unconvinced that simulating a real process (by hand or otherwise) is the same thing as it actually happening with real matter and energy" Computers are real matter and energy. When someone has a cochlear implant, do you think they aren't really hearing because a microphone turning movement into modulated electricity is fake matter and fake energy, and an eardrum and bones doing it is real matter and real energy? Yes it's true that you can't get on a simulation of a plane and fly to New York, but if you see the output of an arithmetic calculation there's no way to tell if it was done with a redstone computer in Minecraft or with Python or with brain matter. (Is it possible for arithmetic to be not-simulated?). [1] https://en.wikipedia.org/wiki/Bouba/kiki_effect https://en.wikipedia.org/wiki/Bouba/kiki_effect [2] https://en.wikipedia.org/wiki/Pareidolia https://en.wikipedia.org/wiki/Pareidolia [3] http://prize.hutter1.net/ http://prize.hutter1.net/