8 ms·

Scott mischaracterises the reasons, which concern capacities of the system not properties of its output. (1) It lacks the capacity to interpret sentences. It's

Scott mischaracterises the reasons, which concern capacities of the system not properties of its output. (1) It lacks the capacity to interpret sentences. It's output possess the property of seeming-to-have-interpreted. (2) It lacks the capacity to learn. Learning is, in part, explaining; it is, in part, causal contact with an environment. It's output has the property of seeming-to-be-generated-by-a-learner. (3) It lacks the capacity to speak either truthfully or otherwise. Lacking the capacity to explain, because at least it lacks the capacity to imagine (counterfactuals), it cannot thereby determine whether a claim accords with the way the world is. It's output has the property: seeming-to-be-true. (4) It lacks the capacity to be moral. For all the above, and more: it lacks the capacity to care. It's output has the property: seeming-to-take-moral-stands. The "Jesuit" here, I fear, is the AI fanatic who have turned their telescopes away from reality (intelligence as it actually exists), and have stolen the lenses to read scripture (intelligence as theorised by partisan academics). One has to agree with Chomsky here at the end, "given the amorality, faux science and linguistic incompetence of these systems, we can only laugh or cry at their popularity." The morality point I think should hit home most extremely: how horrifying to treat generated text output as-if it were constructed by an agent who cares. Apparent properties of the output are not properties of the system; ChatGPT does not think, know, care, intend, speak, commnuicate, etc. One can only "laugh or cry" at how absurd this sales pitch: what a horror to be invited to treat ChatGPT as possessing any of these capacities, simply because correlations across a whole internet of text seems to.

krona 4y ago
(1) As it is with humans. It's not possible to attach a debugger to a human brain and determine the nature of understanding; you have to ask the human questions about that understanding and interpret the response yourself. (2) This is a strange one to me. It's a bit like saying an essential quality of a human is to experience the world as a human, and therefore nothing except a human can be human. (3) If only all humans met this arbitrary standard of secular scientific rationalism. (4) I'd say 5% of humanity fail to meet this criteria.
- mjburgess 4y ago
  (1) You realise that we don't investigate human understanding by looking at input-ouptut text correspondances? (This is quite a funny claim!) The reason we think people have the capacity to interpret sentences is their context-appropriate sensory-motor actions, reasoning, emotion, etc. When I say, "quick! a fire!" people: panic, run, get organised, etc. There are a very very large number of highly complex environmental, interpersonal, sensory-motor, emotional, etc. etc. interactions which follow. ChatGPT not only does not display any, but cannot. This isn't up for debate, the system has no capacity to understand text. If I say, "quick! a fire!" it will not do anything which accords with showing an understanding of that. Worse, as a matter of fact, we know its possible to generate output as-if it did understand. So we (1) have known a mechanism whereby it can fake it; and (2) know that this mechanism is the one its using. (2) All thinking possess intentionality which simply means its about something, and the reason we ask, say, "should i call the police?!" is because we are situated in an environment, which causes us to form mental states about that environment. Any system which emits, "should I call the police?" without having a backing representational state of, eg., the police, care, concern, priority, the environment which requires police, etc. *Does Not Mean* this question. It cannot. (3) The capacity to speak truthfully does not mean you speak truthfully. That capacity is simply that you can form representational states which you can introspect as being accurate or inaccurate. ChatGPT lacks that capacity. It does not form representational (intentional) states when it generates text; each sentence is not caused by such a state. (4) The capacity i named was "caring", pretty much all people care, it's necessary for goal-directed action.
  - krona 4y ago
    ChatGPT has state and representation, clearly. But that's besides the point. It's been demonstrated to have the theory of mind of a 9 year old child, and this will no doubt improve. All of this is a bit like saying a car isn't a horse because it doesn't have hooves. Yes, but so what?
    - mjburgess 4y ago
      The question of whether it has a representational model of the terms it's using is open at the moment. I'm strongly inclined to deny it, but the argument on this is at least quite complex and subject to premises which are less certain than the following: It does not form a representational state prior to speaking which is the reason it speaks. In other-words, it does not say, "I like ice-cream" because it likes ice-cream. That's sufficient to invalidate the idea that it has any relevant capacities here. We even know, for example, that it randomly chooses unlikely words to make its output seem more reasonable: this mechanism alone invalidates it having a representational model. --- And on the former point: if we look at pretty much all plausible theories of meaning, the meaning of terms cannot be anything like what ChatGPT has access to. I think what ChatGPT has is something like "a statistically approximate model of word connotation". Whether a statistical approximation of part of meaning counts as "being meaningful" is a matter of debate. (I think it clearly doesnt.)
PaulHoule 4y ago
I find the morality point being interesting because it is a big deal in the Abrahamic religions that the original sin of discovering the difference between right and wrong is what made us human.
- mjburgess 4y ago
  I think morality is one of the strongest wedge issues against this sort of pseudoscience which says: if it appears X, it is X. ie., there are very few people who would make that claim about morality. There are many who might think, "well if it appears to be playing chess, it is!" (i disagree). But almost none who'd say, "well if it appears to defence murder, it must really think its a good idea!". The latter is clearly absurd, and we've lots of reasons to suppose the system even lacks basic capacities to form any view, preference, intention (etc.) about murder. Once you realise this, I think the house-of-cards falls; because it sets up the fatal gap between: what the output appears to be caused by vs. what it really is caused by. As soon as you ask, "but what really caused the machine to generate that text?" the answer exposes the trick.
  - PaulHoule 4y ago
    Two things about morality. It’s connected with truth and logic because a correct moral decision involved getting the facts straight. In our household (which runs a riding academy) we reject the Abrahamic position and say that morality is a function of mammals (and probably birds.). That’s based on observations of animals having strong emotional reactions to violations of social expectations. The first time I got thrown off a horse, for instance, the horse was much more shook up than I was and acted contrite and extra-sweet afterwards. There was one time I teased my cat by taking the side of the neighbors’ cat (with body language) in a territory dispute and my cat seemed highly offended at me which persisted for a few hours.
- cvoss 4y ago
  > is what made us human. It can't be the case that the first sin is what made humans human, at least not for Christianity. Back up to 1:26-27 for what Christians take to be the thing that makes humans humans. They were fully human prior to eating of the tree. But did they know some right from wrong before eating? Yes, and they are held accountable for exactly that when they eat anyway. What exactly "knowledge of good and evil" entails, given that they are already morally accountable before that point, is an interesting topic to unpack. It must be closely related to an idolatrous belief that, not only am I in the likeness of God and so can discern right from wrong, but I am _so_ like him that I get to _decide_ what's right and wrong.
college_physics 4y ago
Something that I have only really discovered in AI related comments here on HN (and it terrifies me) is that a good number of humans have a deeply-felt need to dehumanize humans (pressumably also themselves). Sure, a fraction is purely driven by career / monetary gain prospects. They adopt the mantle of the AI fanatic to prime themselves for the wrestling in the mud that they hope will get them closer to their goals. I understand that and I am not particularly worried about that. Its a devil we know. But another fraction seems to actively enjoy the absurd suggestions advanced by the AI Jesuits.
- simple-thoughts 4y ago
  If I’m understanding you, the problem may be much worse than you realize. “Homo Sapiens” the wise ones have defined ourselves as the intelligent beings. But with AI demonstrating an alien intelligence, humanness and intelligence become decoupled. We can start to see, though dimly, a world where the aspects that make us uniquely human are not related to intelligence but are instead related to biology , animality, sociology, and so forth. A lot of people are going to be questioning what it means to be human.
- maxdoop 4y ago
  How is it dehumanizing to explore the potential similarities between new AI and LLMs and ourselves? We still have the experience of being human. Isn’t that enough to savor? The experience itself? Clinging to this idea of “I’m human, made of meat, and special” doesn’t really make for an argument as to why / how we are different. I don’t think it’s dehumanizing at all. It’s exciting, and does nothing to take away from our experience of being human.
FredPret 4y ago
He writes a great reply to this in his own comment section: "Michael #3: In a certain sense you’re right. The language models now being adopted by millions of programmers don’t write working code; they only seem-to-write-working-code. They’re not, unfortunately, already doing millions of students’ homework for them; they’re only seeming-to-do-the-homework. Even if in a few years they help me and my colleagues do our research, they won’t actually be helping, but only seeming-to-help. They won’t change civilization; they’ll only seem-to-change-it." [0] https://scottaaronson.blog/?p=7094#comment-1947189 https://scottaaronson.blog/?p=7094#comment-1947189
- mjburgess 4y ago
  > they’ll only seem-to-change-it Then he & I agree! :)
  - raisin_churn 4y ago
    He only seems to agree.
- throw_pm23 4y ago
  Yes, Deep Blue didn't "really understand chess", it just seemed to, and also happened to beat Kasparov in the process.
  - ChainOfFools 4y ago
    I think the proper formulation of the argument is whether Deep Blue seemed-to-understand chess better than Kasparov seemed-to-understand chess. "Understanding" will turn out to be simply a compression with optimization toward survival of the compressing algorithm. When "survival" finally resolves into a conflict between different algorithms, the winner will be that compression algorithm which compresses those that cannot compress themselves.
  - FredPret 4y ago
    Kasparov only seemed to play chess. Who knows what REALLY goes on in his head. Maybe he daydreams and randomly moves the pieces and he's really extremely lucky. Maybe he's only regurgitating chess moves he saw in a book. /s
yowzadave 4y ago
> 1) It lacks the capacity to interpret sentences. It's output possess the property of seeming-to-have-interpreted. This is important, and even here on HN people are frequently ascribing to ChatGPT various qualities (“understanding”, “learning”, etc.) that misunderstand what it is doing. ChatGPT can perform a trick: it produces conversations that have the texture of human ones, skipping entirely the step of understanding the underlying ideas. Obviously a lot of people see utility in this ability, but there are limits to what this approach can do.
- HDThoreaun 4y ago
  I don't really agree. I wouldn't call NLP a trick, it's extremely useful. So what if chatGPT is just a chinese room? It's an effective tool and that's all it needs to be.
shrimp_emoji 4y ago
"Its", not "It's". A helpful tip: whenever you need to decide whether to use one or the other, ask yourself whether you want to say "it is"; if so, use "it's"; otherwise, use "its". Interestingly, ChatGPT knows whether to use "it's" or "its" at what seems like a superhuman level, but we don't even really know how it knows whether to use "a" vs. "an": https://clementneo.com/posts/2023/02/11/we-found-an-neuron https://clementneo.com/posts/2023/02/11/we-found-an-neuron
- ahahahahah 4y ago
  > how it knows whether to use "a" vs. "an": It doesn't "know" that and this is a great example of how clearly it doesn't know things even when it can produce text that if a person produced it you'd assume indicates understanding. Ask it to explain when to use "a" vs "an" and you'll get a good explanation. Give it a test, though and it'll often fail. For example: > me: Imagine that there's an animal called a "honbukle" pronounced like ending in "buckle" and starting with "on". Tell a short story about having seen one. > chatgpt: <blah blah blah> "It sounds like you saw a honbukle," they said. <blah blah blah>
  - ahahahahah 4y ago
    In fact, it will reliably fail such a test because it just reproduces the most likely thing from its training corpus with no understanding of the actual rules under which to use "a" vs "an".
kybernetikos 4y ago
seeming-to vs actual is a real distinction that can be made in some cases. However, I don't think you get to make it unless you can define what 'actual' actually is. Saying that the AI can't interpret but only seem-to-have-interpreted is begging the question unless you can then go on and explain what interpret actually means. It is precisely what is most interesting about the large language models that complex seeming-tos arise from a simple process done on a large scale. If you want to say that humans have access to a qualitatively different 'actual', then you should be able to explain what that difference is, because without that deep understanding of the phenomenon, it's perfectly reasonable to ask how we know that humans do the actual rather than the seeming-to. Your number 3 is pretty surprising, given that explaining is one of few areas I'd say it's frequently superhuman in. It can also certainly generate imaginings with more fluidity than I can too. Until we have a really good understanding of how humans do these things, we shouldn't spend any time pronouncing on how there's a huge difference between a capability as ChatGPT has it and as a human has it.
- mjburgess 4y ago
  The actual capacities are listed: caring, imagining, and so on. Animals form mental representations of their environment, which are directly caused by that environment in conjunction with their prior models of it. On top of these states they combine motivational and emotional attitudes to form goals, to intend, to play, and so on. There is far less mystery here than we suppose. Thousands of years of religion has forced this supposition that there's a "mind/body" problem, and characterising the minds of animals is such a difficult thing; it isnt.
  - kybernetikos 4y ago
    > Animals form mental representations of their environment, which are directly caused by that environment in conjunction with their prior models of it. On top of these states they combine motivational and emotional attitudes to form goals, to intend, to play, and so on. And I take it that you're arguing that ChatGPT does not have a mental representation of its environment? Obviously, we could quibble about 'mental', but for all of its limitations and problems, ChatGPT is very little more than a condensed representation of its environment. What about motivation and emotion? I don't think we have a clear enough definition of how these things are instantiated in humans to properly debate here. ChatGPT acts as if it has certain motivations (to be helpful, to avoid controversy), and has been coached to avoid expressions of emotion. If you start from a view point of 'a next-word prediction engine obviously cannot have emotion, plans, or a world model', then of course you'll conclude that ChatGPT does not have these things and cannot. If you start from the view that we don't really know what mechanisms we need to instantiate these things anyway, then you're much more likely to look at the areas that ChatGPT seems to behave similarly to humans and wonder if big chunks of our brain are actually next token prediction engines, just structured by evolution to work in particular ways. This view was not new with ChatGPT of course, predictive coding has been an interesting approach to human cognition for a while now.
elixxir123 4y ago
I think that a LLM similar to chatgpt could be trained to take a moral stance, the system could be trained in a corpus in which the defense of moral values is explored deeply. Then we could test the premise of Chomsky about the amorality of LLM, it could be an interesting challenge and experience.
mikepalmer 4y ago
> ChatGPT does not think, know, care, intend, speak, commnuicate, etc. These are statements of faith. They can't be proven or disproven by any experiment. "I think therefore I am." applies to just me. I'll never know whether you have intentionality or not. But I have faith that you do. Maybe you have faith that ChatGPT doesn't. I'm agnostic! Chomsky's thinking is a form of vitalism: that biological matter possesses a special life force or energy that nonbiological matter doesn't. Well, vitalism eventually went out of style.
rhn_mk1 4y ago
The second halves of your sentences are easy to verify to be true. But how do you go from them to the first halves? Naively, the assertions about capacities presented here contradict the observations about the outputs.
calf 4y ago
I thought this comment about what science is was also insightful because it explains simply Chomsky's problem with pseudoscientific approaches: I believe your disagreements with Chomsky on politics does creep in to the discussion of AI and linguistics. Chomsky has never denied the success of ChaptGPT or google search or any of those things. The article of Norvig states this. But the core disagreement between Chomsky and his followers (such as myself) against those who prefer a statistical view of intelligence is that LLM or in fact the entire field of AI as it is right now has no real scientific basis compared to what exists in other scientific fields like physics or biology or chemistry. We don’t have a first principles of intelligence, hell we don’t even have bad theories of intelligence. Algorithms that form the basis of Google search or the statistical nature of LLM does not explain what is actually working in the real world. Just like the brain which doesn’t perform advanced calculus to calculate trajectories when it raises the hand of a body to grab a ball in the air. Look back at the Norvig article until, we have made spectacular technological progress, basically engineering and that’s great as a tool to help society. Just don’t confuse that with actual science that is supposed to explain what is actually out there. That’s the true nature of Galileo’s telescope.

krona 4y ago

(1) As it is with humans. It's not possible to attach a debugger to a human brain and determine the nature of understanding; you have to ask the human questions about that understanding and interpret the response yourself. (2) This is a strange one to me. It's a bit like saying an essential quality of a human is to experience the world as a human, and therefore nothing except a human can be human. (3) If only all humans met this arbitrary standard of secular scientific rationalism. (4) I'd say 5% of humanity fail to meet this criteria.

mjburgess 4y ago
(1) You realise that we don't investigate human understanding by looking at input-ouptut text correspondances? (This is quite a funny claim!) The reason we think people have the capacity to interpret sentences is their context-appropriate sensory-motor actions, reasoning, emotion, etc. When I say, "quick! a fire!" people: panic, run, get organised, etc. There are a very very large number of highly complex environmental, interpersonal, sensory-motor, emotional, etc. etc. interactions which follow. ChatGPT not only does not display any, but cannot. This isn't up for debate, the system has no capacity to understand text. If I say, "quick! a fire!" it will not do anything which accords with showing an understanding of that. Worse, as a matter of fact, we know its possible to generate output as-if it did understand. So we (1) have known a mechanism whereby it can fake it; and (2) know that this mechanism is the one its using. (2) All thinking possess intentionality which simply means its about something, and the reason we ask, say, "should i call the police?!" is because we are situated in an environment, which causes us to form mental states about that environment. Any system which emits, "should I call the police?" without having a backing representational state of, eg., the police, care, concern, priority, the environment which requires police, etc. *Does Not Mean* this question. It cannot. (3) The capacity to speak truthfully does not mean you speak truthfully. That capacity is simply that you can form representational states which you can introspect as being accurate or inaccurate. ChatGPT lacks that capacity. It does not form representational (intentional) states when it generates text; each sentence is not caused by such a state. (4) The capacity i named was "caring", pretty much all people care, it's necessary for goal-directed action.
- krona 4y ago
  ChatGPT has state and representation, clearly. But that's besides the point. It's been demonstrated to have the theory of mind of a 9 year old child, and this will no doubt improve. All of this is a bit like saying a car isn't a horse because it doesn't have hooves. Yes, but so what?
  - mjburgess 4y ago
    The question of whether it has a representational model of the terms it's using is open at the moment. I'm strongly inclined to deny it, but the argument on this is at least quite complex and subject to premises which are less certain than the following: It does not form a representational state prior to speaking which is the reason it speaks. In other-words, it does not say, "I like ice-cream" because it likes ice-cream. That's sufficient to invalidate the idea that it has any relevant capacities here. We even know, for example, that it randomly chooses unlikely words to make its output seem more reasonable: this mechanism alone invalidates it having a representational model. --- And on the former point: if we look at pretty much all plausible theories of meaning, the meaning of terms cannot be anything like what ChatGPT has access to. I think what ChatGPT has is something like "a statistically approximate model of word connotation". Whether a statistical approximation of part of meaning counts as "being meaningful" is a matter of debate. (I think it clearly doesnt.)

PaulHoule 4y ago

I find the morality point being interesting because it is a big deal in the Abrahamic religions that the original sin of discovering the difference between right and wrong is what made us human.

mjburgess 4y ago
I think morality is one of the strongest wedge issues against this sort of pseudoscience which says: if it appears X, it is X. ie., there are very few people who would make that claim about morality. There are many who might think, "well if it appears to be playing chess, it is!" (i disagree). But almost none who'd say, "well if it appears to defence murder, it must really think its a good idea!". The latter is clearly absurd, and we've lots of reasons to suppose the system even lacks basic capacities to form any view, preference, intention (etc.) about murder. Once you realise this, I think the house-of-cards falls; because it sets up the fatal gap between: what the output appears to be caused by vs. what it really is caused by. As soon as you ask, "but what really caused the machine to generate that text?" the answer exposes the trick.
- PaulHoule 4y ago
  Two things about morality. It’s connected with truth and logic because a correct moral decision involved getting the facts straight. In our household (which runs a riding academy) we reject the Abrahamic position and say that morality is a function of mammals (and probably birds.). That’s based on observations of animals having strong emotional reactions to violations of social expectations. The first time I got thrown off a horse, for instance, the horse was much more shook up than I was and acted contrite and extra-sweet afterwards. There was one time I teased my cat by taking the side of the neighbors’ cat (with body language) in a territory dispute and my cat seemed highly offended at me which persisted for a few hours.
cvoss 4y ago
> is what made us human. It can't be the case that the first sin is what made humans human, at least not for Christianity. Back up to 1:26-27 for what Christians take to be the thing that makes humans humans. They were fully human prior to eating of the tree. But did they know some right from wrong before eating? Yes, and they are held accountable for exactly that when they eat anyway. What exactly "knowledge of good and evil" entails, given that they are already morally accountable before that point, is an interesting topic to unpack. It must be closely related to an idolatrous belief that, not only am I in the likeness of God and so can discern right from wrong, but I am _so_ like him that I get to _decide_ what's right and wrong.

college_physics 4y ago

Something that I have only really discovered in AI related comments here on HN (and it terrifies me) is that a good number of humans have a deeply-felt need to dehumanize humans (pressumably also themselves). Sure, a fraction is purely driven by career / monetary gain prospects. They adopt the mantle of the AI fanatic to prime themselves for the wrestling in the mud that they hope will get them closer to their goals. I understand that and I am not particularly worried about that. Its a devil we know. But another fraction seems to actively enjoy the absurd suggestions advanced by the AI Jesuits.

simple-thoughts 4y ago
If I’m understanding you, the problem may be much worse than you realize. “Homo Sapiens” the wise ones have defined ourselves as the intelligent beings. But with AI demonstrating an alien intelligence, humanness and intelligence become decoupled. We can start to see, though dimly, a world where the aspects that make us uniquely human are not related to intelligence but are instead related to biology , animality, sociology, and so forth. A lot of people are going to be questioning what it means to be human.
maxdoop 4y ago
How is it dehumanizing to explore the potential similarities between new AI and LLMs and ourselves? We still have the experience of being human. Isn’t that enough to savor? The experience itself? Clinging to this idea of “I’m human, made of meat, and special” doesn’t really make for an argument as to why / how we are different. I don’t think it’s dehumanizing at all. It’s exciting, and does nothing to take away from our experience of being human.

FredPret 4y ago

He writes a great reply to this in his own comment section: "Michael #3: In a certain sense you’re right. The language models now being adopted by millions of programmers don’t write working code; they only seem-to-write-working-code. They’re not, unfortunately, already doing millions of students’ homework for them; they’re only seeming-to-do-the-homework. Even if in a few years they help me and my colleagues do our research, they won’t actually be helping, but only seeming-to-help. They won’t change civilization; they’ll only seem-to-change-it." [0] https://scottaaronson.blog/?p=7094#comment-1947189 https://scottaaronson.blog/?p=7094#comment-1947189

mjburgess 4y ago
> they’ll only seem-to-change-it Then he & I agree! :)
- raisin_churn 4y ago
  He only seems to agree.
throw_pm23 4y ago
Yes, Deep Blue didn't "really understand chess", it just seemed to, and also happened to beat Kasparov in the process.
- ChainOfFools 4y ago
  I think the proper formulation of the argument is whether Deep Blue seemed-to-understand chess better than Kasparov seemed-to-understand chess. "Understanding" will turn out to be simply a compression with optimization toward survival of the compressing algorithm. When "survival" finally resolves into a conflict between different algorithms, the winner will be that compression algorithm which compresses those that cannot compress themselves.
- FredPret 4y ago
  Kasparov only seemed to play chess. Who knows what REALLY goes on in his head. Maybe he daydreams and randomly moves the pieces and he's really extremely lucky. Maybe he's only regurgitating chess moves he saw in a book. /s

yowzadave 4y ago

> 1) It lacks the capacity to interpret sentences. It's output possess the property of seeming-to-have-interpreted. This is important, and even here on HN people are frequently ascribing to ChatGPT various qualities (“understanding”, “learning”, etc.) that misunderstand what it is doing. ChatGPT can perform a trick: it produces conversations that have the texture of human ones, skipping entirely the step of understanding the underlying ideas. Obviously a lot of people see utility in this ability, but there are limits to what this approach can do.

HDThoreaun 4y ago
I don't really agree. I wouldn't call NLP a trick, it's extremely useful. So what if chatGPT is just a chinese room? It's an effective tool and that's all it needs to be.

shrimp_emoji 4y ago

"Its", not "It's". A helpful tip: whenever you need to decide whether to use one or the other, ask yourself whether you want to say "it is"; if so, use "it's"; otherwise, use "its". Interestingly, ChatGPT knows whether to use "it's" or "its" at what seems like a superhuman level, but we don't even really know how it knows whether to use "a" vs. "an": https://clementneo.com/posts/2023/02/11/we-found-an-neuron https://clementneo.com/posts/2023/02/11/we-found-an-neuron

ahahahahah 4y ago
> how it knows whether to use "a" vs. "an": It doesn't "know" that and this is a great example of how clearly it doesn't know things even when it can produce text that if a person produced it you'd assume indicates understanding. Ask it to explain when to use "a" vs "an" and you'll get a good explanation. Give it a test, though and it'll often fail. For example: > me: Imagine that there's an animal called a "honbukle" pronounced like ending in "buckle" and starting with "on". Tell a short story about having seen one. > chatgpt: <blah blah blah> "It sounds like you saw a honbukle," they said. <blah blah blah>
- ahahahahah 4y ago
  In fact, it will reliably fail such a test because it just reproduces the most likely thing from its training corpus with no understanding of the actual rules under which to use "a" vs "an".

kybernetikos 4y ago

seeming-to vs actual is a real distinction that can be made in some cases. However, I don't think you get to make it unless you can define what 'actual' actually is. Saying that the AI can't interpret but only seem-to-have-interpreted is begging the question unless you can then go on and explain what interpret actually means. It is precisely what is most interesting about the large language models that complex seeming-tos arise from a simple process done on a large scale. If you want to say that humans have access to a qualitatively different 'actual', then you should be able to explain what that difference is, because without that deep understanding of the phenomenon, it's perfectly reasonable to ask how we know that humans do the actual rather than the seeming-to. Your number 3 is pretty surprising, given that explaining is one of few areas I'd say it's frequently superhuman in. It can also certainly generate imaginings with more fluidity than I can too. Until we have a really good understanding of how humans do these things, we shouldn't spend any time pronouncing on how there's a huge difference between a capability as ChatGPT has it and as a human has it.

mjburgess 4y ago
The actual capacities are listed: caring, imagining, and so on. Animals form mental representations of their environment, which are directly caused by that environment in conjunction with their prior models of it. On top of these states they combine motivational and emotional attitudes to form goals, to intend, to play, and so on. There is far less mystery here than we suppose. Thousands of years of religion has forced this supposition that there's a "mind/body" problem, and characterising the minds of animals is such a difficult thing; it isnt.
- kybernetikos 4y ago
  > Animals form mental representations of their environment, which are directly caused by that environment in conjunction with their prior models of it. On top of these states they combine motivational and emotional attitudes to form goals, to intend, to play, and so on. And I take it that you're arguing that ChatGPT does not have a mental representation of its environment? Obviously, we could quibble about 'mental', but for all of its limitations and problems, ChatGPT is very little more than a condensed representation of its environment. What about motivation and emotion? I don't think we have a clear enough definition of how these things are instantiated in humans to properly debate here. ChatGPT acts as if it has certain motivations (to be helpful, to avoid controversy), and has been coached to avoid expressions of emotion. If you start from a view point of 'a next-word prediction engine obviously cannot have emotion, plans, or a world model', then of course you'll conclude that ChatGPT does not have these things and cannot. If you start from the view that we don't really know what mechanisms we need to instantiate these things anyway, then you're much more likely to look at the areas that ChatGPT seems to behave similarly to humans and wonder if big chunks of our brain are actually next token prediction engines, just structured by evolution to work in particular ways. This view was not new with ChatGPT of course, predictive coding has been an interesting approach to human cognition for a while now.

elixxir123 4y ago

I think that a LLM similar to chatgpt could be trained to take a moral stance, the system could be trained in a corpus in which the defense of moral values is explored deeply. Then we could test the premise of Chomsky about the amorality of LLM, it could be an interesting challenge and experience.

mikepalmer 4y ago

> ChatGPT does not think, know, care, intend, speak, commnuicate, etc. These are statements of faith. They can't be proven or disproven by any experiment. "I think therefore I am." applies to just me. I'll never know whether you have intentionality or not. But I have faith that you do. Maybe you have faith that ChatGPT doesn't. I'm agnostic! Chomsky's thinking is a form of vitalism: that biological matter possesses a special life force or energy that nonbiological matter doesn't. Well, vitalism eventually went out of style.

rhn_mk1 4y ago

The second halves of your sentences are easy to verify to be true. But how do you go from them to the first halves? Naively, the assertions about capacities presented here contradict the observations about the outputs.

calf 4y ago

I thought this comment about what science is was also insightful because it explains simply Chomsky's problem with pseudoscientific approaches: I believe your disagreements with Chomsky on politics does creep in to the discussion of AI and linguistics. Chomsky has never denied the success of ChaptGPT or google search or any of those things. The article of Norvig states this. But the core disagreement between Chomsky and his followers (such as myself) against those who prefer a statistical view of intelligence is that LLM or in fact the entire field of AI as it is right now has no real scientific basis compared to what exists in other scientific fields like physics or biology or chemistry. We don’t have a first principles of intelligence, hell we don’t even have bad theories of intelligence. Algorithms that form the basis of Google search or the statistical nature of LLM does not explain what is actually working in the real world. Just like the brain which doesn’t perform advanced calculus to calculate trajectories when it raises the hand of a body to grab a ball in the air. Look back at the Norvig article until, we have made spectacular technological progress, basically engineering and that’s great as a tool to help society. Just don’t confuse that with actual science that is supposed to explain what is actually out there. That’s the true nature of Galileo’s telescope.