3 ms·

Looking through some other articles from "Confirm Labs" it's striking how often concepts related to the underlying mathematical mechanism related to LLM models

Looking through some other articles from "Confirm Labs" it's striking how often concepts related to the underlying mathematical mechanism related to LLM models (i.e. linear algebra, latent embedding, optimization techniques, model alignment) are assigned very personified terms (e.g. dreaming, trustworthiness). I am not an AI researcher, my field these days is much smaller statistical models, but reading some of these papers and then reading also the discourse a lot of AI researchers engage in surrounding AGI, sentience, pain, etc... I for one have a hard time taking some of it seriously. I just don't buy some of it but of course I'm less than fully informed. My only critique is that you should not start by naming every algorithm or operation you will perform on a model after a thing that a person with a brain does and then claim to evaluate a concept such as sentience or distress from the position of an unbiased objective observer. This naming convention sows doubt among those of us who are not a part of that world. > Feature visualization, also known as "dreaming", offers insights into vision models by optimizing the inputs to maximize a neuron's activation or other internal component. However, dreaming has not been successfully applied to language models because the input space is discrete.

sterlind 2y ago
Metaphor has always been how tech concepts get named. You have "folders" on your "desk top," which you click with a "mouse." You can send "messages" through "pipes." Heck, your computer even has "memory," and you write programs using "languages." What word do you propose to replace "hallucination," for example? Remember that it ought to be descriptive and easy to remember.
- acc_297 2y ago
  Well it's interesting - many LLM researchers reject "hallucination" in favour of the term "confabulation" but I'll double down and say that I still don't like either as they are both terms from clinical psych and I'll include a quote from an article[1] interviewing Usama Fayyad (executive director of Northeastern’s Institute for Experiential AI) ''' “I could say — these models hallucinated; or, to be more precise, I could say, well, the model made an error, and we understand that these models make errors,” Fayyad says. ''' For years we called the behaviour that researchers now call "hallucination" test error or misprediction or a failure to forecast correct outputs based on "out-of-sample" inputs and those terms were accurate and would still be accurate to use on today's LLM models. Metaphors are great but can be misleading, people don't often make an argument that a computer mouse can feel pain. [1] https://news.northeastern.edu/2023/11/10/ai-chatbot-hallucinations/ https://news.northeastern.edu/2023/11/10/ai-chatbot-hallucin...
  - sterlind 2y ago
    "test error" or "misprediction" aren't very applicable terms for RL, though. is it a "test error" if the LLM suggests a bad chess move? is it a "misprediction" if the LLM can't find a mating sequence? are those "failures to forecast out-of-sample inputs," or only if the LLM suggests an illegal chess move or call an API that doesn't exist? I say that a model "hallucinates" only if it specifically presents false information as fact - e.g. that Bolognium is element 115, or that chocolate milk comes from brown cows. giving irrelevant or undesired answers isn't hallucination or confabulation. but I don't think your terms are precise enough for that nuance.
    - acc_297 2y ago
      I grant that the term has become commonplace and very catchy and no one has been able to supplant it with a better descriptive I still hold that there is good reason to be concerned with researchers claiming to objectively evaluate a silicon based sentience and jumping the gun a bit (imo) by writing articles about LLMs which "dream" and "hallucinate" when the behaviour is a consequence of essentially a non-linear regression model that does not yet show clear signs of the high order emotion/reasoning/awareness/intention Many who research LLMs avoid these terms in their publications for good reason (again imo)

sterlind 2y ago

Metaphor has always been how tech concepts get named. You have "folders" on your "desk top," which you click with a "mouse." You can send "messages" through "pipes." Heck, your computer even has "memory," and you write programs using "languages." What word do you propose to replace "hallucination," for example? Remember that it ought to be descriptive and easy to remember.

acc_297 2y ago
Well it's interesting - many LLM researchers reject "hallucination" in favour of the term "confabulation" but I'll double down and say that I still don't like either as they are both terms from clinical psych and I'll include a quote from an article[1] interviewing Usama Fayyad (executive director of Northeastern’s Institute for Experiential AI) ''' “I could say — these models hallucinated; or, to be more precise, I could say, well, the model made an error, and we understand that these models make errors,” Fayyad says. ''' For years we called the behaviour that researchers now call "hallucination" test error or misprediction or a failure to forecast correct outputs based on "out-of-sample" inputs and those terms were accurate and would still be accurate to use on today's LLM models. Metaphors are great but can be misleading, people don't often make an argument that a computer mouse can feel pain. [1] https://news.northeastern.edu/2023/11/10/ai-chatbot-hallucinations/ https://news.northeastern.edu/2023/11/10/ai-chatbot-hallucin...
- sterlind 2y ago
  "test error" or "misprediction" aren't very applicable terms for RL, though. is it a "test error" if the LLM suggests a bad chess move? is it a "misprediction" if the LLM can't find a mating sequence? are those "failures to forecast out-of-sample inputs," or only if the LLM suggests an illegal chess move or call an API that doesn't exist? I say that a model "hallucinates" only if it specifically presents false information as fact - e.g. that Bolognium is element 115, or that chocolate milk comes from brown cows. giving irrelevant or undesired answers isn't hallucination or confabulation. but I don't think your terms are precise enough for that nuance.
  - acc_297 2y ago
    I grant that the term has become commonplace and very catchy and no one has been able to supplant it with a better descriptive I still hold that there is good reason to be concerned with researchers claiming to objectively evaluate a silicon based sentience and jumping the gun a bit (imo) by writing articles about LLMs which "dream" and "hallucinate" when the behaviour is a consequence of essentially a non-linear regression model that does not yet show clear signs of the high order emotion/reasoning/awareness/intention Many who research LLMs avoid these terms in their publications for good reason (again imo)