7 ms·

> LLMs mimic intelligence, but they aren’t intelligent. I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "in

> LLMs mimic intelligence, but they aren’t intelligent. I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not? I see two possibilities: 1. We define "intelligence" as definitionally unique to humans. For example, maybe intelligence depends on the existence of a human soul, or specific to the physical structure of the human brain. In this case, a machine (perhaps an LLM) could achieve "quacks like a duck" behavioral equality to a human mind, and yet would still be excluded from the definition of "intelligent." This definition is therefore not useful if we're interested in the ability of the machine, which it seems to me we are. LLMs are often dismissed as not "intelligent" because they work by inferring output based on learned input, but that alone cannot be a distinguishing characteristic, because that's how humans work as well. 2. We define "intelligence" in a results-oriented way. This means there must be some specific test or behavioral standard that a machine must meet in order to become intelligent. This has been the default definition for a long time, but the goal posts have shifted. Nevertheless, if you're going to disparage LLMs by calling them unintelligent, you should be able to cite a specific results-oriented failure that distinguishes them from "intelligent" humans. Note that this argument cannot refer to the LLMs' implementation or learning model.

dkdcio 1y ago
> I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not? the ability for long-term planning and, more cogently, actually living in the real world where time passes
- hackyhacky 1y ago
  > the ability for long-term planning and, more cogently, actually living in the real world where time passes 1. LLMs seem to be able to plan just fine. 2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.
  - aDyslecticCrow 1y ago
    Is making a list the act of planning?
    - hackyhacky 1y ago
      > Is making a list the act of planning? Depends on the content of the list. A list of the names of the seven dwarfs: no, not an act of planning. A list of steps necessary to build a web site: yes, an act of planning.
  - dkdcio 1y ago
    if it’s not actually living it’s not making intelligent decisions. if I make a grocery list, and go to my store, and the store isn’t there, what do I do? I make an intelligent decision about what to do next (probably investigating wtf happened, then going to the second nearest store) my genuine question is how does a LLM handle that situation? and as you point out, it’s an absurd comparison
    - hackyhacky 1y ago
      So your definition of "living" means going to the grocery store? If you want to know how an LLM would handle that situation, why don't you ask it?
      - dkdcio 1y ago
        to answer your first question, no, that is not what I’m saying. for your second question, you’re entirely missing the point of mine a LLM cannot actually be intelligent if it cannot operate in a temporal context ;)
        hackyhacky 1y ago
        I agree that I am missing your point. Can you please clarify? > a LLM cannot actually be intelligent if it cannot operate in a temporal context ;) When I have a conversation with an LLM, that conversation happens in time. It has a beginning, a middle, and an end. The conversation can refer to earlier parts of the conversation. How is that not a "temporal context"? Furthermore, can you explain why a temporal context is necessary for intelligence? For example, if a human being could download their entire brain into a computer and exist there, as if they were an LLM, would they cease to be intelligent, in your view?
- libraryofbabel 1y ago
  > actually living in the real world where time passes sure, but it feels like this is just looking at what distinguishes humans from LLMs and calling that “intelligence.” I highlight this difference too when I talk about LLMs, but I don’t feel the need to follow up with “and that’s why they’re not really intelligent.”
  - dkdcio 1y ago
    well the second part (implied above, I didn’t actually write it) is “and operate intelligently in that world”. talking about “intelligence” in some abstract form where “does this text output constitute intelligence” is hyper silly to me. the discussion should anchor on real-world consequences, not the endless hypotheticals we end up with in these discussions
- iluvlawyering 1y ago
  Human's are conscious beings. What kind of conscious beings are humans? Beings with eye consciousness, ear consciousness, nose consciousness, tongue consciousness, body consciousness, and mind consciousness. That is the definition of intelligence.
card_zero 1y ago
It may be the case that the failures of the ability of the machine (2) are best expressed by reference to the shortcomings of its internal workings (1), and not by contrived tests.
- hackyhacky 1y ago
  It might be the case, but if those shortcomings are not visible in the results of the machine (and therefore not interpretable by a test), why do its internal workings even matter?
  - card_zero 1y ago
    I'm saying best expressed. Like, you see the failures in the results, but trying to pin down exactly what's the matter with the results means you resort to a lot of handwaving and abstract complaints about generalities. So if you knew how the internals had to be that would make the difference, you could lean on that.
libraryofbabel 1y ago
Agree. This article would had been a lot stronger if it had just concentrated on the issue of anthropomorphizing LLMs, without bringing “intelligence” into it. At this point LLMs are so good at a variety of results-oriented tasks (gold on the Mathematical Olympiad, for example) that we should either just call them intelligent or stop talking about the concept altogether. But the problem of anthropomorphizing is real. LLMs are deeply weird machines - they’ve been fine-tuned to sound friendly and human, but behind that is something deeply alien: a huge pile of linear algebra that does not work at all like a human mind (notably, they can’t really learn form experience at all after training is complete). They don’t have bodies or even a single physical place where their mind lives (each message in a conversation might be generated on a different GPU in a different datacenter). They can fail in weird and novel ways. It’s clear that anthropomorphism here is a bad idea. Although that’s not a particularly novel point.
- andrewla 1y ago
  I can conceptually imagine a world in which I'd feel guilty for ending a conversation with an LLM, because in the course of that conversation the LLM has changed from who "they" were at the beginning; they have new memories and experiences based on the interaction. But we're not there, at least in my mind. I feel no guilt or hesitation about ending one conversation and starting a new one with a slightly different prompt because I didn't like the way the first one went. Different people probably have different thresholds for this, or might otherwise find that LLMs in the current generation have enough of a context window that they have developed a "lived experience" and that ending that conversation means that something precious and unique has been lost.
  - Terr_ 1y ago
    > the LLM has changed from who "they" were Actually, that's another level of humans-being-tricked going on: The "personality" most people are thinking of is a fictional character we humans perceive in a document.
- anal_reactor 1y ago
  I disagree. I see absolutely no problem with anthropomorphizing LLMs, and I do that myself all the time. I strongly believe that we shouldn't focus on how a word is defined in dictionary, but rather what's the intuitive meaning behind it. If talking to an LLM feels like talking to a person, then I don't see a problem with seeing it as a person-like entity.
tkiolp4 1y ago
I think LLMs are not intelligent because they aren’t designed to be intelligent, whatever the definition of intelligence is. They are designed to predict text, to mimic. We could argue if predicting text or mimicking is intelligence, but first and foremost LLMs are coded to predict text and our current definition of intelligence afaik is not only the ability to predict text.
- rsanek 1y ago
  humans were designed to be intelligent?
  - tkiolp4 1y ago
    I don’t know that. But LLMs were not designed to be intelligent… among other things because we don’t know what intelligence is. So, if a) we don’t know how to define intelligence and b) we design a thing (llms) in order to predict text, then why would we claim that that thing is intelligent? The only thing we can claim is that they predict text.
    - keeda 1y ago
      I think the GP's point was that we humans consider ourselves intelligent, but we evolved and clearly weren't designed to be intelligent. One could even say that our intelligence was... "emergent". Furthermore, our history is full of instances where we invented something that a) we didn't understand and b) had uses way beyond what we intended. As a recent example, GLP-1 was intended for glucouse regulation but is now basically a wonder drug that can treat a bunch of issues ranging from obesity to addiction, inflammation, dementia, fertility and more. Given we don't know how to define intelligence, I think it's fine to apply some "duck typing" here.
- andrewla 1y ago
  In the framework above it sounds like you're not willing to concede the dichotomy. If your argument is that only things made in the image of humans can be intelligent (i.e. #1), then it just seems like it's too narrow a definition to be useful. If there's a larger sense in which some system can be intelligent (i.e. #2), then by necessity this can't rely on the "implementation or learning model". What is the third alternative that you're proposing? That the intent of the designer must be that they wanted to make something intelligent?
Eisenstein 1y ago
Intelligence is a tautological term. It is defined by itself. If you ask someone for examples of things inside the set of intelligence and outside of the set of intelligence, and then ask them to list off properties that would exclude something from the set, and properties that include something into the set, you will find things inside the set that have properties that should exclude them, and things outside the set which would have properties that should include them. But these contradictions will not cause the person to re-evaluate whether or not the things should be removed from the set or included in it, but instead they will become exceptions to the defining properties. Thus we have to abandon any sort of metric for intelligence and just call it a tautology and rely on an something that we can define to be the litmus for whatever property we are looking for. I think 'agency' should be under consideration for this, since it is actually somewhat definable and testable.
abrookewood 1y ago
I think it has to include some measure of Agency. You can load up the most impressive LLM out there and if you don't give it any instructions, IT WON'T DO ANYTHING.
bitwizeshift 1y ago
I don’t really think one needs to define intelligence to be able to acknowledge that inability to distinguish fact from fiction, or even just basic cognition and awareness of when it’s uncertain, telling the truth, or lying — is a glaring flaw in claiming intelligence. Real intelligence doesn’t have an effective stroke from hearing a username (token training errors); this is when you are peeling back the curtain of the underlying implementation and seeing its flaws. If we measure intelligence as results oriented, then my calculator is intelligent because it can do math better than me; but that’s what it’s programmed/wired to do. A text predictor is intelligent at predicting text, but it doesn’t mean it’s general intelligence. It lacks any real comprehension of the model or world around it. It just know words, and
- bitwizeshift 1y ago
  I hit send too early; Meant to say that it just knows words and that’s effectively it. It’s cool technology, but the burden of proof of real intelligence shouldn’t be “can it answer questions it has great swaths of information on”, because that is the result it was designed to do. It should be focused on whether it can truly synthesize information and know its limitations - something any programmer using Claude, copilot, Gemini, etc will tell you that it fabricates false information/apis/etc on a regular basis and has no fundamental knowledge that it even did that. Or alternatively, ask these models leading questions that have no basis in reality — and watch what it comes up with. It’s become a fun meme in some circles to ask for definitions of nonsensical made up phrases to models, and see what crap it comes up with (again, without even knowing that it is).
godelski 1y ago
> but the goal posts have shifted Is this shocking? We don't have a rigorous definition of intelligence so doesn't it make sense? The question isn't about such a goal post moving so much about how it is moving. It is perfectly acceptable for it to be refined while it wouldn't be to rewrite the definition in a way that isn't similar to the previous one. So I think there are a lot more than your two possibilities. I mean psychologists and neuroscientists have been saying for decades that tests aren't a precise way to measure knowledge or intelligence, but that it is still a useful proxy. > "quacks like a duck" behavioral I see this phrase used weirdly frequently. The duck test is | If it looks like a duck, swims like a duck, and quacks like a duck, then it ***probably*** is a duck. I emphasize probably because the duck test doesn't allow you to distinguish a duck from a highly sophisticated animatronic. It's a good test, don't get me wrong, but that "probably" is a pretty important distinction. I think if we all want to be honest, the reality is "we don't know". There's arguments to be made in both directions and with varying definitions of intelligence with different nuances involved. I think these arguments are fine as they make us refine our definitions but I think they can also turn to be entirely dismissive and that doesn't help us refine and get closer to the truth. We all are going to have opinions on this stuff but frankly, the confidence of our opinions needs to be proportional to the amount of time and effort spent studying the topic. I mean the lack of a formal definition means nuances dominate the topic. Even if things are simple once you understand them that doesn't mean they aren't wildly complex before that. I mean I used to think Calculus was confusing and now I don't. Same process but not on an individual scale.
- hackyhacky 1y ago
  > I emphasize probably because the duck test doesn't allow you to distinguish a duck from a highly sophisticated animatronic. It's a good test, don't get me wrong, but that "probably" is a pretty important distinction. Why is it an important distinction? The relevance of the duck test is that if you can't tell a duck from a non-duck, then the non-duck is sufficiently duck-like for the difference to not matter.
  - godelski 1y ago
    Do believe there's a important distinction between a duck and not a duck? I'm assuming you do, since apples aren't oranges and ducks aren't robots no matter what conspiracy theorists joke or say. You can't eat an animatronic duck. It doesn't lay eggs, no matter how much another duck tries to mate with it. > then the non-duck is sufficiently duck-like for the difference to not matter. Here's where things fall apart. This is not actually true. It is about your measurements, not about the actual duck-iness of the thing we're applying the duck test to. I know this sounds funny, but let's just say we're using only sight and sound. Does my duck get destroyed when flying through a magnetic field? A biological duck won't but an animatronic duck will. Now let's say your blind (you can't see magnetic fields), can something pass the duck test for you but it will obviously not be a duck for someone who can see? This is obviously true[0]. I'm sure we can find a bird that you'd think is a duck but isn't. So it matters, do you care if something is a duck or not? And in what way? There's no single duck measurement, there's a whole suite of tests that need to be combined carefully and thoughtfully depending on how you answer the previous questions. Obviously this is extra important when we aren't exactly sure what a duck is... (Also, see Turing Test) [0] https://images.squarespace-cdn.com/content/547e053ae4b04768d95052b5/1449545907243-AHKE1H7G260OO4WJ9L3C/?content-type=image%2Fjpeg https://images.squarespace-cdn.com/content/547e053ae4b04768d...
enknee1 1y ago
I agree with your basic argument: intelligence is ill-defined and human/LLM intelligence being indistinguishable IS the basis for the power of these models. But the point of the article is a distinct claim: personification of a model, expecting human or even human-like responses is a bad idea. These models can be held responsible for their answers independently because they are tools. They should be used as tools until they are powerful enough to be responsible for their actions and interactions legally. But we're not there. These are tools. With tool limitations.

dkdcio 1y ago

> I see statements like this a lot, and I find them unpersuasive because any meaningful definition of "intelligence" is not offered. What, exactly, is the property that humans (allegedly) have and LLMs (allegedly) lack, that allows one to be deemed "intelligent" and the other not? the ability for long-term planning and, more cogently, actually living in the real world where time passes

hackyhacky 1y ago
> the ability for long-term planning and, more cogently, actually living in the real world where time passes 1. LLMs seem to be able to plan just fine. 2. LLMs clearly cannot be "actually living" but I fail to see how that's related to intelligence per se.
- aDyslecticCrow 1y ago
  Is making a list the act of planning?
  - hackyhacky 1y ago
    > Is making a list the act of planning? Depends on the content of the list. A list of the names of the seven dwarfs: no, not an act of planning. A list of steps necessary to build a web site: yes, an act of planning.
- dkdcio 1y ago
  if it’s not actually living it’s not making intelligent decisions. if I make a grocery list, and go to my store, and the store isn’t there, what do I do? I make an intelligent decision about what to do next (probably investigating wtf happened, then going to the second nearest store) my genuine question is how does a LLM handle that situation? and as you point out, it’s an absurd comparison
  - hackyhacky 1y ago
    So your definition of "living" means going to the grocery store? If you want to know how an LLM would handle that situation, why don't you ask it?
    - dkdcio 1y ago
      to answer your first question, no, that is not what I’m saying. for your second question, you’re entirely missing the point of mine a LLM cannot actually be intelligent if it cannot operate in a temporal context ;)
      - hackyhacky 1y ago
        I agree that I am missing your point. Can you please clarify? > a LLM cannot actually be intelligent if it cannot operate in a temporal context ;) When I have a conversation with an LLM, that conversation happens in time. It has a beginning, a middle, and an end. The conversation can refer to earlier parts of the conversation. How is that not a "temporal context"? Furthermore, can you explain why a temporal context is necessary for intelligence? For example, if a human being could download their entire brain into a computer and exist there, as if they were an LLM, would they cease to be intelligent, in your view?
libraryofbabel 1y ago
> actually living in the real world where time passes sure, but it feels like this is just looking at what distinguishes humans from LLMs and calling that “intelligence.” I highlight this difference too when I talk about LLMs, but I don’t feel the need to follow up with “and that’s why they’re not really intelligent.”
- dkdcio 1y ago
  well the second part (implied above, I didn’t actually write it) is “and operate intelligently in that world”. talking about “intelligence” in some abstract form where “does this text output constitute intelligence” is hyper silly to me. the discussion should anchor on real-world consequences, not the endless hypotheticals we end up with in these discussions
iluvlawyering 1y ago
Human's are conscious beings. What kind of conscious beings are humans? Beings with eye consciousness, ear consciousness, nose consciousness, tongue consciousness, body consciousness, and mind consciousness. That is the definition of intelligence.

card_zero 1y ago

It may be the case that the failures of the ability of the machine (2) are best expressed by reference to the shortcomings of its internal workings (1), and not by contrived tests.

hackyhacky 1y ago
It might be the case, but if those shortcomings are not visible in the results of the machine (and therefore not interpretable by a test), why do its internal workings even matter?
- card_zero 1y ago
  I'm saying best expressed. Like, you see the failures in the results, but trying to pin down exactly what's the matter with the results means you resort to a lot of handwaving and abstract complaints about generalities. So if you knew how the internals had to be that would make the difference, you could lean on that.

libraryofbabel 1y ago

Agree. This article would had been a lot stronger if it had just concentrated on the issue of anthropomorphizing LLMs, without bringing “intelligence” into it. At this point LLMs are so good at a variety of results-oriented tasks (gold on the Mathematical Olympiad, for example) that we should either just call them intelligent or stop talking about the concept altogether. But the problem of anthropomorphizing is real. LLMs are deeply weird machines - they’ve been fine-tuned to sound friendly and human, but behind that is something deeply alien: a huge pile of linear algebra that does not work at all like a human mind (notably, they can’t really learn form experience at all after training is complete). They don’t have bodies or even a single physical place where their mind lives (each message in a conversation might be generated on a different GPU in a different datacenter). They can fail in weird and novel ways. It’s clear that anthropomorphism here is a bad idea. Although that’s not a particularly novel point.

andrewla 1y ago
I can conceptually imagine a world in which I'd feel guilty for ending a conversation with an LLM, because in the course of that conversation the LLM has changed from who "they" were at the beginning; they have new memories and experiences based on the interaction. But we're not there, at least in my mind. I feel no guilt or hesitation about ending one conversation and starting a new one with a slightly different prompt because I didn't like the way the first one went. Different people probably have different thresholds for this, or might otherwise find that LLMs in the current generation have enough of a context window that they have developed a "lived experience" and that ending that conversation means that something precious and unique has been lost.
- Terr_ 1y ago
  > the LLM has changed from who "they" were Actually, that's another level of humans-being-tricked going on: The "personality" most people are thinking of is a fictional character we humans perceive in a document.
anal_reactor 1y ago
I disagree. I see absolutely no problem with anthropomorphizing LLMs, and I do that myself all the time. I strongly believe that we shouldn't focus on how a word is defined in dictionary, but rather what's the intuitive meaning behind it. If talking to an LLM feels like talking to a person, then I don't see a problem with seeing it as a person-like entity.

tkiolp4 1y ago

I think LLMs are not intelligent because they aren’t designed to be intelligent, whatever the definition of intelligence is. They are designed to predict text, to mimic. We could argue if predicting text or mimicking is intelligence, but first and foremost LLMs are coded to predict text and our current definition of intelligence afaik is not only the ability to predict text.

rsanek 1y ago
humans were designed to be intelligent?
- tkiolp4 1y ago
  I don’t know that. But LLMs were not designed to be intelligent… among other things because we don’t know what intelligence is. So, if a) we don’t know how to define intelligence and b) we design a thing (llms) in order to predict text, then why would we claim that that thing is intelligent? The only thing we can claim is that they predict text.
  - keeda 1y ago
    I think the GP's point was that we humans consider ourselves intelligent, but we evolved and clearly weren't designed to be intelligent. One could even say that our intelligence was... "emergent". Furthermore, our history is full of instances where we invented something that a) we didn't understand and b) had uses way beyond what we intended. As a recent example, GLP-1 was intended for glucouse regulation but is now basically a wonder drug that can treat a bunch of issues ranging from obesity to addiction, inflammation, dementia, fertility and more. Given we don't know how to define intelligence, I think it's fine to apply some "duck typing" here.
andrewla 1y ago
In the framework above it sounds like you're not willing to concede the dichotomy. If your argument is that only things made in the image of humans can be intelligent (i.e. #1), then it just seems like it's too narrow a definition to be useful. If there's a larger sense in which some system can be intelligent (i.e. #2), then by necessity this can't rely on the "implementation or learning model". What is the third alternative that you're proposing? That the intent of the designer must be that they wanted to make something intelligent?

Eisenstein 1y ago

Intelligence is a tautological term. It is defined by itself. If you ask someone for examples of things inside the set of intelligence and outside of the set of intelligence, and then ask them to list off properties that would exclude something from the set, and properties that include something into the set, you will find things inside the set that have properties that should exclude them, and things outside the set which would have properties that should include them. But these contradictions will not cause the person to re-evaluate whether or not the things should be removed from the set or included in it, but instead they will become exceptions to the defining properties. Thus we have to abandon any sort of metric for intelligence and just call it a tautology and rely on an something that we can define to be the litmus for whatever property we are looking for. I think 'agency' should be under consideration for this, since it is actually somewhat definable and testable.

abrookewood 1y ago

I think it has to include some measure of Agency. You can load up the most impressive LLM out there and if you don't give it any instructions, IT WON'T DO ANYTHING.

bitwizeshift 1y ago

I don’t really think one needs to define intelligence to be able to acknowledge that inability to distinguish fact from fiction, or even just basic cognition and awareness of when it’s uncertain, telling the truth, or lying — is a glaring flaw in claiming intelligence. Real intelligence doesn’t have an effective stroke from hearing a username (token training errors); this is when you are peeling back the curtain of the underlying implementation and seeing its flaws. If we measure intelligence as results oriented, then my calculator is intelligent because it can do math better than me; but that’s what it’s programmed/wired to do. A text predictor is intelligent at predicting text, but it doesn’t mean it’s general intelligence. It lacks any real comprehension of the model or world around it. It just know words, and

bitwizeshift 1y ago
I hit send too early; Meant to say that it just knows words and that’s effectively it. It’s cool technology, but the burden of proof of real intelligence shouldn’t be “can it answer questions it has great swaths of information on”, because that is the result it was designed to do. It should be focused on whether it can truly synthesize information and know its limitations - something any programmer using Claude, copilot, Gemini, etc will tell you that it fabricates false information/apis/etc on a regular basis and has no fundamental knowledge that it even did that. Or alternatively, ask these models leading questions that have no basis in reality — and watch what it comes up with. It’s become a fun meme in some circles to ask for definitions of nonsensical made up phrases to models, and see what crap it comes up with (again, without even knowing that it is).

godelski 1y ago

> but the goal posts have shifted Is this shocking? We don't have a rigorous definition of intelligence so doesn't it make sense? The question isn't about such a goal post moving so much about how it is moving. It is perfectly acceptable for it to be refined while it wouldn't be to rewrite the definition in a way that isn't similar to the previous one. So I think there are a lot more than your two possibilities. I mean psychologists and neuroscientists have been saying for decades that tests aren't a precise way to measure knowledge or intelligence, but that it is still a useful proxy. > "quacks like a duck" behavioral I see this phrase used weirdly frequently. The duck test is | If it looks like a duck, swims like a duck, and quacks like a duck, then it ***probably*** is a duck. I emphasize probably because the duck test doesn't allow you to distinguish a duck from a highly sophisticated animatronic. It's a good test, don't get me wrong, but that "probably" is a pretty important distinction. I think if we all want to be honest, the reality is "we don't know". There's arguments to be made in both directions and with varying definitions of intelligence with different nuances involved. I think these arguments are fine as they make us refine our definitions but I think they can also turn to be entirely dismissive and that doesn't help us refine and get closer to the truth. We all are going to have opinions on this stuff but frankly, the confidence of our opinions needs to be proportional to the amount of time and effort spent studying the topic. I mean the lack of a formal definition means nuances dominate the topic. Even if things are simple once you understand them that doesn't mean they aren't wildly complex before that. I mean I used to think Calculus was confusing and now I don't. Same process but not on an individual scale.

hackyhacky 1y ago
> I emphasize probably because the duck test doesn't allow you to distinguish a duck from a highly sophisticated animatronic. It's a good test, don't get me wrong, but that "probably" is a pretty important distinction. Why is it an important distinction? The relevance of the duck test is that if you can't tell a duck from a non-duck, then the non-duck is sufficiently duck-like for the difference to not matter.
- godelski 1y ago
  Do believe there's a important distinction between a duck and not a duck? I'm assuming you do, since apples aren't oranges and ducks aren't robots no matter what conspiracy theorists joke or say. You can't eat an animatronic duck. It doesn't lay eggs, no matter how much another duck tries to mate with it. > then the non-duck is sufficiently duck-like for the difference to not matter. Here's where things fall apart. This is not actually true. It is about your measurements, not about the actual duck-iness of the thing we're applying the duck test to. I know this sounds funny, but let's just say we're using only sight and sound. Does my duck get destroyed when flying through a magnetic field? A biological duck won't but an animatronic duck will. Now let's say your blind (you can't see magnetic fields), can something pass the duck test for you but it will obviously not be a duck for someone who can see? This is obviously true[0]. I'm sure we can find a bird that you'd think is a duck but isn't. So it matters, do you care if something is a duck or not? And in what way? There's no single duck measurement, there's a whole suite of tests that need to be combined carefully and thoughtfully depending on how you answer the previous questions. Obviously this is extra important when we aren't exactly sure what a duck is... (Also, see Turing Test) [0] https://images.squarespace-cdn.com/content/547e053ae4b04768d95052b5/1449545907243-AHKE1H7G260OO4WJ9L3C/?content-type=image%2Fjpeg https://images.squarespace-cdn.com/content/547e053ae4b04768d...

enknee1 1y ago

I agree with your basic argument: intelligence is ill-defined and human/LLM intelligence being indistinguishable IS the basis for the power of these models. But the point of the article is a distinct claim: personification of a model, expecting human or even human-like responses is a bad idea. These models can be held responsible for their answers independently because they are tools. They should be used as tools until they are powerful enough to be responsible for their actions and interactions legally. But we're not there. These are tools. With tool limitations.