5 ms·

I am NLP researcher who volunteers for peer review often and the anthropomorphisms in papers are indeed very common and very wrong. I have to ask authors to not

I am NLP researcher who volunteers for peer review often and the anthropomorphisms in papers are indeed very common and very wrong. I have to ask authors to not ascribe cognition to their deep learning approaches in about a third of the papers I review. People do this because mirroring cognition to machine learning lends credence that their specific modeling mechanism mimicks human understanding and so is closer "to the real thing". Obviously this is almost never the case, unless they explicitly use biomimetic methods in which case they are often outperformed by non-biomimetic state-of-the-art approaches. Thanks OP for giving me citation ammo to refer to in my obligatory "don't humanise AI" section of reviews. (It is so common I copy paste this section from a template).

gillesjacobs 4y ago
Not to shoot across the bow of CS Engineers but the trend I spot (tentatively) is that it is pure computer science folk that most often do this. In NLP you have a mix of people coming from pure CS and signal processing (the latter esp. in speech processing) and others who come from linguistics or other humanities. The CS people seem all too happy to humanise computation, probably because they had less direct teaching regarding the cognitive mechanisms behind cognition and language production.
joe_the_user 4y ago
People do this because mirroring cognition to machine learning lends credence that their specific modeling mechanism mimicks human understanding and so is closer "to the real thing". Doesn't this also involve people not having another category aside from "cognition" to put natural language processing acts in? How many neural net constructors have a rigorously developed framework describing what "cognition" is? I mean, there's a common counter argument to the "this is not cognition" position. That is: "you're just using 'cognition' as a placeholder for whatever these systems can't do". I don't think that counter-argument is true or characterizes the position well but it's important to frame one's position so it doesn't seem to be subject to this counter-argument.
- gillesjacobs 4y ago
  > Doesn't this also involve people not having another category aside from "cognition" to put natural language processing acts in? Yes, of course this might be an even more primary reason; do not attribute to malice what can be explained by laziness. However, AI researchers should be wary of their language, that point is hammered in most curricula I have seen. So at the least it is negligence. > I mean, there's a common counter argument to the "this is not cognition" position. That is: "you're just using 'cognition' as a placeholder for whatever these systems can't do". Very valid point, but we know current deep learning mechanisms do not mimick human learning, language understanding and production in any way. They are far too simplified and specific for that. Neural network activation functions are a far cry from neural spiking models and biological neural connectivity is far more complex than the networks used in deep learning. The attention mechanism that drives recent LLMs is also claimed to have some biological similarities, but upon closer inspection drawing strong analogies is not credible [1]. computer vs. human visual recognition tasks it falls apart and higher-level visual concepts. [2] 1. https://www.frontiersin.org/articles/10.3389/fncom.2020.00029/full https://www.frontiersin.org/articles/10.3389/fncom.2020.0002... 2. https://arxiv.org/abs/1906.08764 https://arxiv.org/abs/1906.08764
  - sdenton4 4y ago
    Why does nlp need to mimic human brain function to count as cognition? This seems overly reductive; I see no reason to believe that biomimicry is necessary for intelligence. The argument is overly reductive: only brains think, and this is not a brain, ergo it does not think. (To be clear, I don't think this system is an AGI; just making the point that better goal posts are needed...)
    - gillesjacobs 4y ago
      I agree and focusing on biomimetics is indeed insufficient in discussions of what constitutes intelligence. The discussion on what constitutes intelligence and cognition is too vast and complex. The central point is that haphazard and unsubstantiated authromorphising language should be barred from scientific papers. We know that an LLM does not "know", "believe", "intent", or "feels" because they lack any form of integrated, general human-like intelligence. It is trivially correct to use different terms like "models", "predicts", "outputs emotive expressions" and avoid unevidenced claims of human behaviour. I also wanted to nuance my opening post: I file the "don't humanise ML models"-comment of peer review under "minor issues". This means that it is an issue with the paper that does not prevent publication. This is not a hill I am willing to let good research die on.
fourfivefour 4y ago
bias infests research as well as seen by the replication crisis. So you being a researcher doesn't give more credence to your words especially given that the state of current research cannot fully comprehend what these ML models are doing internally. I do agree that we can't ascribe cognition to machine learning. But I also believe that we can't ascribe that it's NOT cognition. Why? Because we don't even truly understand what "Knowing" or cognition is. We can't even ascribe a quantitative similarity metric. What we are seeing is that those inputs and outputs look remarkably similar to the real thing. How similar it is internally is not a known thing. That's why even though you're an NLP researcher, I still say your argument here is just as niave as the person who claims these things are sentient. You simply don't know. No one does.
- gillesjacobs 4y ago
  In science, if you don't know, you don't make the claim, that is basic positivism and the scientific method. So basic in fact, I was thought this in elementary school. So far ad-hominem attributions of naivety. Anyone that humanises computation is not only committing an A.I. faux-pas but are going against the basic scientific method.
  - deleted 4y ago
    [deleted]
  - oneoneonetwo 4y ago
    > In science, if you don't know, you don't make the claim, that is basic positivism and the scientific method. Yes you're correct. So you can't make the claim that it's NOT cognition. That is my point. You also can't make the claim that it is cognition which was the OTHER point. Completely agree with your statement here. But it goes further then this, and your statement shows YOU don't understand science. >So basic in fact, I was thought this in elementary school. So far ad-hominem attributions of naivety. No science is complex and basically most people don't understand the scientific method and it's limitations. It's not basic at all, not even people who graduate from four year colleges in STEM fully understand the true nature of science. Or even many scientists! In science and therefore reality as we know it; nothing can be proven. This is because every subsequent observation can completely contradict an initial claim. Proof is the domain of logic and math, it doesn't exist in reality. Things can be disproven but nothing can actually be proven. That is science. This is subtle stuff, but it's legit. I'll quote Einstein if you don't believe me: "No amount of experimentation can ever prove me right; a single experiment can prove me wrong." - Einstein And a link for further investigation: https://en.wikipedia.org/wiki/Falsifiability https://en.wikipedia.org/wiki/Falsifiability Anyway all of this says that NO claim can be made about anything unless it's disproof. Which is exactly inline with what I'm saying. Still claims are made all the time anyway in academia and the majority of these claims aren't technically scientific. This occurs because we can't practically operate on anything in reality if we can't in actuality claim things are true. So we do it anyway despite lack of any form of actual proof. >Anyone that humanises computation is not only committing an A.I. faux-pas but are going against the basic scientific method. But so is dismissing any similarity to humans. You can't technically say it's wrong or right. Especially when the outputs and inputs to these models are very similar to what humans would say. This is basic preschool stuff I knew this when I was a baby! I thought everybody knew this! <Joking>.
Zababa 4y ago
I'm not really sure about the context here, but I know that I tend to humanize AIs, for example interacting with ChatGPT like with a regular human being, because I'm being nice to him and he's being nice to me in return. I don't know if it's more like being nice to a human, or more like taking good care of your tools so they will take good care of you, but it just feels better for me.
- roflyear 4y ago
  It isn't a strange thing or even a bad thing: it is a human thing to anthropomorphize things. We do it with animals even, which is some weird irony. We do it with common objects. We're just wired that way. It doesn't mean that there is "something there" with LM tho - just that we're good at tricking ourselves to think that way.
- gillesjacobs 4y ago
  It is entirely ok and normal to humanize machines, just don't do it in scientific engineering papers is all I am saying. I name my bots and machines and of course in daily discussions the loaded words ("thinking", ""believing", "meaning", "knowing") are used. Simulating any human behaviour will elicit a sympathetic response, especially if it has utility to the user. But in the context of peer-review of scientific engineering papers that is inappropiate.
  - pama 4y ago
    I don’t understand why if common usage of such terms is fine and even desirable in daily context, the same terms should not be used in scientific communication. If people are unclear or ambiguous, then of course that’s undesirable and should be corrected, however, if certain terms are fully clear within their context then I don’t see the harm of the same term having a different meaning in the study of cognition and a different meaning when the term is used in the context of the analysis of an ML algorithm. The whole field is called machine “learning” after all.
    - gillesjacobs 4y ago
deleted 4y ago
[deleted]

gillesjacobs 4y ago

Not to shoot across the bow of CS Engineers but the trend I spot (tentatively) is that it is pure computer science folk that most often do this. In NLP you have a mix of people coming from pure CS and signal processing (the latter esp. in speech processing) and others who come from linguistics or other humanities. The CS people seem all too happy to humanise computation, probably because they had less direct teaching regarding the cognitive mechanisms behind cognition and language production.

joe_the_user 4y ago

People do this because mirroring cognition to machine learning lends credence that their specific modeling mechanism mimicks human understanding and so is closer "to the real thing". Doesn't this also involve people not having another category aside from "cognition" to put natural language processing acts in? How many neural net constructors have a rigorously developed framework describing what "cognition" is? I mean, there's a common counter argument to the "this is not cognition" position. That is: "you're just using 'cognition' as a placeholder for whatever these systems can't do". I don't think that counter-argument is true or characterizes the position well but it's important to frame one's position so it doesn't seem to be subject to this counter-argument.

gillesjacobs 4y ago
> Doesn't this also involve people not having another category aside from "cognition" to put natural language processing acts in? Yes, of course this might be an even more primary reason; do not attribute to malice what can be explained by laziness. However, AI researchers should be wary of their language, that point is hammered in most curricula I have seen. So at the least it is negligence. > I mean, there's a common counter argument to the "this is not cognition" position. That is: "you're just using 'cognition' as a placeholder for whatever these systems can't do". Very valid point, but we know current deep learning mechanisms do not mimick human learning, language understanding and production in any way. They are far too simplified and specific for that. Neural network activation functions are a far cry from neural spiking models and biological neural connectivity is far more complex than the networks used in deep learning. The attention mechanism that drives recent LLMs is also claimed to have some biological similarities, but upon closer inspection drawing strong analogies is not credible [1]. computer vs. human visual recognition tasks it falls apart and higher-level visual concepts. [2] 1. https://www.frontiersin.org/articles/10.3389/fncom.2020.00029/full https://www.frontiersin.org/articles/10.3389/fncom.2020.0002... 2. https://arxiv.org/abs/1906.08764 https://arxiv.org/abs/1906.08764
- sdenton4 4y ago
  Why does nlp need to mimic human brain function to count as cognition? This seems overly reductive; I see no reason to believe that biomimicry is necessary for intelligence. The argument is overly reductive: only brains think, and this is not a brain, ergo it does not think. (To be clear, I don't think this system is an AGI; just making the point that better goal posts are needed...)
  - gillesjacobs 4y ago
    I agree and focusing on biomimetics is indeed insufficient in discussions of what constitutes intelligence. The discussion on what constitutes intelligence and cognition is too vast and complex. The central point is that haphazard and unsubstantiated authromorphising language should be barred from scientific papers. We know that an LLM does not "know", "believe", "intent", or "feels" because they lack any form of integrated, general human-like intelligence. It is trivially correct to use different terms like "models", "predicts", "outputs emotive expressions" and avoid unevidenced claims of human behaviour. I also wanted to nuance my opening post: I file the "don't humanise ML models"-comment of peer review under "minor issues". This means that it is an issue with the paper that does not prevent publication. This is not a hill I am willing to let good research die on.

fourfivefour 4y ago

bias infests research as well as seen by the replication crisis. So you being a researcher doesn't give more credence to your words especially given that the state of current research cannot fully comprehend what these ML models are doing internally. I do agree that we can't ascribe cognition to machine learning. But I also believe that we can't ascribe that it's NOT cognition. Why? Because we don't even truly understand what "Knowing" or cognition is. We can't even ascribe a quantitative similarity metric. What we are seeing is that those inputs and outputs look remarkably similar to the real thing. How similar it is internally is not a known thing. That's why even though you're an NLP researcher, I still say your argument here is just as niave as the person who claims these things are sentient. You simply don't know. No one does.

gillesjacobs 4y ago
In science, if you don't know, you don't make the claim, that is basic positivism and the scientific method. So basic in fact, I was thought this in elementary school. So far ad-hominem attributions of naivety. Anyone that humanises computation is not only committing an A.I. faux-pas but are going against the basic scientific method.
- deleted 4y ago
  [deleted]
- oneoneonetwo 4y ago
  > In science, if you don't know, you don't make the claim, that is basic positivism and the scientific method. Yes you're correct. So you can't make the claim that it's NOT cognition. That is my point. You also can't make the claim that it is cognition which was the OTHER point. Completely agree with your statement here. But it goes further then this, and your statement shows YOU don't understand science. >So basic in fact, I was thought this in elementary school. So far ad-hominem attributions of naivety. No science is complex and basically most people don't understand the scientific method and it's limitations. It's not basic at all, not even people who graduate from four year colleges in STEM fully understand the true nature of science. Or even many scientists! In science and therefore reality as we know it; nothing can be proven. This is because every subsequent observation can completely contradict an initial claim. Proof is the domain of logic and math, it doesn't exist in reality. Things can be disproven but nothing can actually be proven. That is science. This is subtle stuff, but it's legit. I'll quote Einstein if you don't believe me: "No amount of experimentation can ever prove me right; a single experiment can prove me wrong." - Einstein And a link for further investigation: https://en.wikipedia.org/wiki/Falsifiability https://en.wikipedia.org/wiki/Falsifiability Anyway all of this says that NO claim can be made about anything unless it's disproof. Which is exactly inline with what I'm saying. Still claims are made all the time anyway in academia and the majority of these claims aren't technically scientific. This occurs because we can't practically operate on anything in reality if we can't in actuality claim things are true. So we do it anyway despite lack of any form of actual proof. >Anyone that humanises computation is not only committing an A.I. faux-pas but are going against the basic scientific method. But so is dismissing any similarity to humans. You can't technically say it's wrong or right. Especially when the outputs and inputs to these models are very similar to what humans would say. This is basic preschool stuff I knew this when I was a baby! I thought everybody knew this! <Joking>.

Zababa 4y ago

I'm not really sure about the context here, but I know that I tend to humanize AIs, for example interacting with ChatGPT like with a regular human being, because I'm being nice to him and he's being nice to me in return. I don't know if it's more like being nice to a human, or more like taking good care of your tools so they will take good care of you, but it just feels better for me.

roflyear 4y ago
It isn't a strange thing or even a bad thing: it is a human thing to anthropomorphize things. We do it with animals even, which is some weird irony. We do it with common objects. We're just wired that way. It doesn't mean that there is "something there" with LM tho - just that we're good at tricking ourselves to think that way.
gillesjacobs 4y ago
It is entirely ok and normal to humanize machines, just don't do it in scientific engineering papers is all I am saying. I name my bots and machines and of course in daily discussions the loaded words ("thinking", ""believing", "meaning", "knowing") are used. Simulating any human behaviour will elicit a sympathetic response, especially if it has utility to the user. But in the context of peer-review of scientific engineering papers that is inappropiate.
- pama 4y ago
  I don’t understand why if common usage of such terms is fine and even desirable in daily context, the same terms should not be used in scientific communication. If people are unclear or ambiguous, then of course that’s undesirable and should be corrected, however, if certain terms are fully clear within their context then I don’t see the harm of the same term having a different meaning in the study of cognition and a different meaning when the term is used in the context of the analysis of an ML algorithm. The whole field is called machine “learning” after all.
  - gillesjacobs 4y ago

deleted 4y ago

[deleted]