4 ms·

I’ll be a lot more open to arguments that the technology described in the Transformer Paper becomes capable of things like understanding when you throw enough d

I’ll be a lot more open to arguments that the technology described in the Transformer Paper becomes capable of things like understanding when you throw enough data and computing power at it when the state of the art moves beyond, “yeah, but don’t human brains just randomly guess at the right thing all the time?” I dunno, do they do more than that? Seems like it to me. Does a Chinese Room “understand”? I say no, but hey, maybe it does. If I laboriously do the math by hand, taking care never to actually know the informational content of any of the input or output myself, does my scratch paper understand your questions? If the output’s just as good as ChatGPT? Where’s the part that understands?

DougN7 2y ago
The scratch paper is a fantastic analogy. I’m going to use it.
fragmede 2y ago
If we open up a human skull, what would we find? A clump of grey matter, a wet piece of meat. As we dissect it further and zoomed in, what would we find? Neurons. Zoom in futher and we get a mess of chemicals. As we look at the pile of cells that make up the brain, we have the same fundamental question: where's the part that understands? Yet somehow, from the connections of all of those cells, and neurotransmitters, there's consciousness and something there that does understand (and think and reason and love). If, instead of LLM architecture, on more powerful computers than we have now, we simulated all of those neurons and their connections, would we have a computer that understands? If we then did those computations on scratch paper, where would the piece that understands be on that piece of paper? The sum of a thing's parts can be greater than the individual parts. Whether or not ChatGPT understands is a whole big question, but we'll have no more luck dissecting LLMs to find out if it does than if we dissected a human brain.
- vundercind 2y ago
  You can simulate a brain to some degree, maybe, but you won’t get the whole thing short of just… using an actual brain, that is physically identical to the one you’re simulating, down to at least the quantum level (and who knows what else?). It’s not at all certain that this can be reduced to math. The calculations for operating an LLM definitely can be reduced to math. No reduction needed, in fact—they are math. This isn’t an argument (to my mind, anyway) against even the possibility of machine whatever-you-like (consciousness, understanding, whatever) but against the idea of equivalence because we could simulate either one—in fact, we can’t simulate one. The other, essentially is already simulation, no further steps needed. What you’re getting at (if I may attempt to present your argument) is that we could reduce either to its components and make it look ridiculous that it might be doing anything particularly advanced. However, in fact we definitely can reproduce exactly what one of them does with a bunch of thick books of lookup tables and some formulas that we could mechanically follow by hand, and it might even be possible to do so in practice, not just hypothetically (at significant, but not impossible, expense) while we do not know we can do that for a human brain, short of just using exactly the brain that we want to “simulate”.
  - jodrellblank 2y ago
    > "It’s not at all certain that this can be reduced to math." It isn't certain that it can be, but can you give any plausible reason why the Universe might allow understanding to (meat + electric patterns) and deny it to (silicon + electric patterns) ?
    - vundercind 2y ago
      Maybe! But some part of it would have to be attempting understanding, not just looking at word patterns.
jodrellblank 2y ago
> "I dunno, do they do more than that? Seems like it to me." When I said "I'm not saying this is an AGI" and you reply with "I dunno human brains do more than ChatGPT" it feels like you haven't understood the discussion - that part was never contested. > "some clever anti-nonsense filtering" Eliezer Yudkowsky wrote 'The opposite of intelligence isn't stupidity'. On an A/B test, stupid is guessing randomly and that approach scores 50%. Scoring 0% takes as much intelligence as 100% because it requires knowing the right answer to be able to avoid it every time. Being able to identify nonsense is sense. At the risk of being tautological, "clever" is clever. If the neurons in your brain are the scratch paper in the Chinese room, each one isn't aware of the content of the light waves or the finger muscle signals, and you conclude the Chinese room doesn't understand, shouldn't you conclude that your brain doesn't understand? If your brain does understand shouldn't you conclude the Chinese room would understand? I claim that ChatGPT being able to explain bridging and give further examples is behaviour which demonstrates more understanding than a rock, than a calculator, a wordlist, a spellchecker, a plain Markov chain has. You say there's "no understanding or anything close to it" - how would ChatGPT's response look different if it did understand the concept of bridging? If you cannot suggest any way its output would look different to how it looks now and instead have to resort to changing the subject, shouldn't you retract that claim? > "Does a Chinese Room “understand”? I say no" Then you must say a human doesn't understand. For what else is there in a human brain except a finite amount of learned behavioural rules for signal inputs and outputs? Learned over a billion years of evolution in the structure, and filled in by a lifetime of nurture.
- vundercind 2y ago
  > If you cannot suggest any way its output would look different to how it looks now and instead have to resort to changing the subject, shouldn't you retract that claim? For one thing, I expect we’d not see so many cases of them chasing (if you will) the prompt and request into silliness. The code attempts to satisfy prompts in a transparently mechanical fashion, which is part of why they so gleefully (if you will) mislead. There’s no understanding. You can ask them to correct and they might, but they can also be induced to correct the already-correct, so that means nothing. To the extent we fix that, it’s not by adding any factor that might represent understanding, it’s further prompting that amounts to “follow these patterns slightly differently”. The fix isn’t, so far, “teaching” them to understand. Maybe we’ll get there! But we don’t appear to be anywhere near that yet. > Then you must say a human doesn't understand. For what else is there in a human brain except a finite amount of learned behavioural rules for signal inputs and outputs? Learned over a billion years of evolution in the structure, and filled in by a lifetime of nurture. The thing about the Chinese Room is that we comprehend the entire process, and there’s no room for some unknown factor affecting the output—or for a known factor that might be processing something like what we mean by understanding (let alone consciousness, say). Every single part of what an LLM does can be replicated with big books of lookup tables, dice, and a list of rules. There’s nowhere for anything to do the understanding to exist. It’s not that we have to be confused by part of it for that to be there—I’m not saying mystery is a necessary component—just that this process doesn’t have a place for that to be.

DougN7 2y ago

The scratch paper is a fantastic analogy. I’m going to use it.

fragmede 2y ago

If we open up a human skull, what would we find? A clump of grey matter, a wet piece of meat. As we dissect it further and zoomed in, what would we find? Neurons. Zoom in futher and we get a mess of chemicals. As we look at the pile of cells that make up the brain, we have the same fundamental question: where's the part that understands? Yet somehow, from the connections of all of those cells, and neurotransmitters, there's consciousness and something there that does understand (and think and reason and love). If, instead of LLM architecture, on more powerful computers than we have now, we simulated all of those neurons and their connections, would we have a computer that understands? If we then did those computations on scratch paper, where would the piece that understands be on that piece of paper? The sum of a thing's parts can be greater than the individual parts. Whether or not ChatGPT understands is a whole big question, but we'll have no more luck dissecting LLMs to find out if it does than if we dissected a human brain.

vundercind 2y ago
You can simulate a brain to some degree, maybe, but you won’t get the whole thing short of just… using an actual brain, that is physically identical to the one you’re simulating, down to at least the quantum level (and who knows what else?). It’s not at all certain that this can be reduced to math. The calculations for operating an LLM definitely can be reduced to math. No reduction needed, in fact—they are math. This isn’t an argument (to my mind, anyway) against even the possibility of machine whatever-you-like (consciousness, understanding, whatever) but against the idea of equivalence because we could simulate either one—in fact, we can’t simulate one. The other, essentially is already simulation, no further steps needed. What you’re getting at (if I may attempt to present your argument) is that we could reduce either to its components and make it look ridiculous that it might be doing anything particularly advanced. However, in fact we definitely can reproduce exactly what one of them does with a bunch of thick books of lookup tables and some formulas that we could mechanically follow by hand, and it might even be possible to do so in practice, not just hypothetically (at significant, but not impossible, expense) while we do not know we can do that for a human brain, short of just using exactly the brain that we want to “simulate”.
- jodrellblank 2y ago
  > "It’s not at all certain that this can be reduced to math." It isn't certain that it can be, but can you give any plausible reason why the Universe might allow understanding to (meat + electric patterns) and deny it to (silicon + electric patterns) ?
  - vundercind 2y ago
    Maybe! But some part of it would have to be attempting understanding, not just looking at word patterns.

jodrellblank 2y ago

> "I dunno, do they do more than that? Seems like it to me." When I said "I'm not saying this is an AGI" and you reply with "I dunno human brains do more than ChatGPT" it feels like you haven't understood the discussion - that part was never contested. > "some clever anti-nonsense filtering" Eliezer Yudkowsky wrote 'The opposite of intelligence isn't stupidity'. On an A/B test, stupid is guessing randomly and that approach scores 50%. Scoring 0% takes as much intelligence as 100% because it requires knowing the right answer to be able to avoid it every time. Being able to identify nonsense is sense. At the risk of being tautological, "clever" is clever. If the neurons in your brain are the scratch paper in the Chinese room, each one isn't aware of the content of the light waves or the finger muscle signals, and you conclude the Chinese room doesn't understand, shouldn't you conclude that your brain doesn't understand? If your brain does understand shouldn't you conclude the Chinese room would understand? I claim that ChatGPT being able to explain bridging and give further examples is behaviour which demonstrates more understanding than a rock, than a calculator, a wordlist, a spellchecker, a plain Markov chain has. You say there's "no understanding or anything close to it" - how would ChatGPT's response look different if it did understand the concept of bridging? If you cannot suggest any way its output would look different to how it looks now and instead have to resort to changing the subject, shouldn't you retract that claim? > "Does a Chinese Room “understand”? I say no" Then you must say a human doesn't understand. For what else is there in a human brain except a finite amount of learned behavioural rules for signal inputs and outputs? Learned over a billion years of evolution in the structure, and filled in by a lifetime of nurture.

vundercind 2y ago
> If you cannot suggest any way its output would look different to how it looks now and instead have to resort to changing the subject, shouldn't you retract that claim? For one thing, I expect we’d not see so many cases of them chasing (if you will) the prompt and request into silliness. The code attempts to satisfy prompts in a transparently mechanical fashion, which is part of why they so gleefully (if you will) mislead. There’s no understanding. You can ask them to correct and they might, but they can also be induced to correct the already-correct, so that means nothing. To the extent we fix that, it’s not by adding any factor that might represent understanding, it’s further prompting that amounts to “follow these patterns slightly differently”. The fix isn’t, so far, “teaching” them to understand. Maybe we’ll get there! But we don’t appear to be anywhere near that yet. > Then you must say a human doesn't understand. For what else is there in a human brain except a finite amount of learned behavioural rules for signal inputs and outputs? Learned over a billion years of evolution in the structure, and filled in by a lifetime of nurture. The thing about the Chinese Room is that we comprehend the entire process, and there’s no room for some unknown factor affecting the output—or for a known factor that might be processing something like what we mean by understanding (let alone consciousness, say). Every single part of what an LLM does can be replicated with big books of lookup tables, dice, and a list of rules. There’s nowhere for anything to do the understanding to exist. It’s not that we have to be confused by part of it for that to be there—I’m not saying mystery is a necessary component—just that this process doesn’t have a place for that to be.