8 ms·

The “statistical parrot” assertion is pretty thoroughly disproven by this point, but suppose we ignore the literature and just assume it’s true: what does it ma

The “statistical parrot” assertion is pretty thoroughly disproven by this point, but suppose we ignore the literature and just assume it’s true: what does it matter? “Real” people are time bombs too, for instance. Is there some predictive power that we gain by reducing LLM skills to mere token production side effects?

drapado 3y ago
I'm curious in your statement, can you point to some papers where they addressed it?
- ignoramous 3y ago
  (not op) The section A Path Forward in Managing AI Risks by Bengio et al cites a few papers: https://managing-ai-risks.com/ https://managing-ai-risks.com/
  - krainboltgreene 3y ago
    I read through that and none of the section (or entire work) ever talk about the above discussion. Further I looked at some of the many citations of on that section and none of them suggest that the OP is right. In fact a few of them I know disagree.
    - ignoramous 3y ago
      Depends. Models are matrices of floats and so there's little chance an umbrella-term like "stochastic parrot" will never not stick, even when they already show signs of syntactic, semantic world-building capability (https://www.arxiv-vanity.com/papers/2206.07682/ https://www.arxiv-vanity.com/papers/2206.07682/). If you are like me (and them: https://archive.is/cZi83 https://archive.is/cZi83) and deem instruction following, chain-of-thought prompting, computational properties of LLMs (as researchers continue to experiment with training, memory, modality, and scaling, for example, to arrive at abstract reasoning) as emergent, then we're on the same page.
      - krainboltgreene 3y ago
        Okay so just to confirm that section doesn't actually tell us anything about this and in fact this is all based on your own understanding of the mechanisms involved.
        ignoramous 3y ago
        My reading of the papers is, given enough scale, modality, and memory; there are chances (perhaps newer and different) models will be able to "generalize" our world. Also: https://archive.is/3yyZZ https://archive.is/3yyZZ / https://twitter.com/QuanquanGu/status/1721394508146057597 https://twitter.com/QuanquanGu/status/1721394508146057597 | And: https://archive.is/bW2tS https://archive.is/bW2tS / https://twitter.com/mansiege/status/1680985267262619648 https://twitter.com/mansiege/status/1680985267262619648
mjburgess 3y ago
> The “statistical parrot” assertion is pretty thoroughly disproven by this point errr... all NNs are just optimisations of an associative probability objective: P(Y|X), they are by definition "statistical parrots". There isn't anything to prove or disprove. People offering prompts as evidence are people who fundamentally do not understand the basics. NNs aren't strange empirical objects, they're specified by mathematical rules whose properties are known ahead of time. Any property of a trained NN is derivative of a property of a formula P(Answer|Prompt, TrainingData) This is an associative statistical relation, which by definition, selects elements of TrainingData by-association with the Prompt. If the basis by which you understand LLMs is putting prompts into ChatGPT you're severely underqualified for drawing any conclusions about LLMs, and radically subject to confirmation bias.
- kristiandupont 3y ago
  Well sure, but I think that same thing can be said about the human brain. Obviously at a whole different level of sophistication and all, but the two really do seem related, at least to me. >People offering prompts as evidence are people who fundamentally do not understand the basics This I agree with, but I also don't see anyone doing that in this thread?
- williamcotton 3y ago
  You’re disregarding the emergent phenomena, which are not at all understood. There was a distinct and unpredicted jump in what can loosely be described as “cognitive abilities” between GPTs 2, 3, and 4, especially after some supervised techniques like RLHF.
  - mjburgess 3y ago
    There is no "emergent phenomena" the pattern described is just the same as when you add +b to an ax+b model of linear data. ie., it's just fitting capacity. The "emergent boundary" is just an empirical measure of the necessary fitting capacity of these models on "everything ever digitised in english" given any particular functional requirement. All the language around this area is not scientific, nor are these practices. This is superstitious neophyte engineers, hopped up on scifi, by giddy VCs who love to be told they're funding captin picard.
deleted 3y ago
[deleted]
famouswaffles 3y ago
>Is there some predictive power that we gain by reducing LLM skills to mere token production side effects? No. If anything, we lose predictive power which is why it's extra silly.
intended 3y ago
Then why dont agents work? If those skills were real, why do they fizzle out on production data ?
- TeMPOraL 3y ago
  That's a good question. What is different about "production data"? What do those "production people" do that suddenly makes LLMs fail on things they work well on when not "production"?
  - deleted 3y ago
    [deleted]
  - intended 3y ago
    I can tell you who would love to hear your answer to that question. Me for starters. If it works, I can quit. Next up are Karpathy and the CTO of OpenAI. Around July and September both talked about production challenges. AI ops was the largest subcategory of fall YC startups. Every single ml and LLM ops individual who gets far enough deals with evals. I don’t know, but maybe - just maybe- the issue is that people don’t understand themselves enough, to avoid assuming too much of those emergent properties. As I recall there was also a paper that pointed out the issues with how LLMs are measured, and that the emergence of properties was not a step change once the tests were updated. Edit- found it: https://hai.stanford.edu/news/ais-ostensible-emergent-abilities-are-mirage https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...
scotty79 3y ago
> Is there some predictive power that we gain by reducing LLM skills to mere token production side effects? Yes. We can infer from this that out of sample hallucinations will be closer to the desired productions when measured by token productions metrics than by any other more informative domain relevant metrics. Which is exactly the case. If you ask llm to solve a math problem it haven't seen its response will be closer to the desired solution in its linguistic form rather than in mathematical meaning.
- famouswaffles 3y ago
  >If you ask llm to solve a math problem it haven't seen its response will be closer to the desired solution in its linguistic form rather than in mathematical meaning. This isn't true. Wild how people will confidently say nonsense about things they obviously haven't actually tested. GPT-4 can manage arbitrary arithmetic calculation closer to the real value than you could ever do without an external tool.
  - scotty79 3y ago
    I don't mean calculations. I mean math problems. Like those: https://www.mathschool.com/locations/andover/news/prepare-for-math-competitions-with-problems-from-the-russian-math-olympiad https://www.mathschool.com/locations/andover/news/prepare-fo...
    - fragmede 3y ago
      ChatGPT-4 is quite capable of solving problems like that. Here's it solving the last two problems on the Grade 7-8 Russian Math Olympiad from the linked page. https://chat.openai.com/share/e90d4711-aa38-45c5-97e7-a9a04ea2294e https://chat.openai.com/share/e90d4711-aa38-45c5-97e7-a9a04e... https://chat.openai.com/share/0c9c579a-ca9f-4aa6-bac6-624372da6a62 https://chat.openai.com/share/0c9c579a-ca9f-4aa6-bac6-624372... The two answers (84 and 225,792) agree with the answer key. I only gave it the questions and didn't give it the PDF either, so it didn't cheat and just read the answer off the answer key in the PDFs.

drapado 3y ago

I'm curious in your statement, can you point to some papers where they addressed it?

ignoramous 3y ago
(not op) The section A Path Forward in Managing AI Risks by Bengio et al cites a few papers: https://managing-ai-risks.com/ https://managing-ai-risks.com/
- krainboltgreene 3y ago
  I read through that and none of the section (or entire work) ever talk about the above discussion. Further I looked at some of the many citations of on that section and none of them suggest that the OP is right. In fact a few of them I know disagree.
  - ignoramous 3y ago
    Depends. Models are matrices of floats and so there's little chance an umbrella-term like "stochastic parrot" will never not stick, even when they already show signs of syntactic, semantic world-building capability (https://www.arxiv-vanity.com/papers/2206.07682/ https://www.arxiv-vanity.com/papers/2206.07682/). If you are like me (and them: https://archive.is/cZi83 https://archive.is/cZi83) and deem instruction following, chain-of-thought prompting, computational properties of LLMs (as researchers continue to experiment with training, memory, modality, and scaling, for example, to arrive at abstract reasoning) as emergent, then we're on the same page.
    - krainboltgreene 3y ago
      Okay so just to confirm that section doesn't actually tell us anything about this and in fact this is all based on your own understanding of the mechanisms involved.
      - ignoramous 3y ago
        My reading of the papers is, given enough scale, modality, and memory; there are chances (perhaps newer and different) models will be able to "generalize" our world. Also: https://archive.is/3yyZZ https://archive.is/3yyZZ / https://twitter.com/QuanquanGu/status/1721394508146057597 https://twitter.com/QuanquanGu/status/1721394508146057597 | And: https://archive.is/bW2tS https://archive.is/bW2tS / https://twitter.com/mansiege/status/1680985267262619648 https://twitter.com/mansiege/status/1680985267262619648

mjburgess 3y ago

> The “statistical parrot” assertion is pretty thoroughly disproven by this point errr... all NNs are just optimisations of an associative probability objective: P(Y|X), they are by definition "statistical parrots". There isn't anything to prove or disprove. People offering prompts as evidence are people who fundamentally do not understand the basics. NNs aren't strange empirical objects, they're specified by mathematical rules whose properties are known ahead of time. Any property of a trained NN is derivative of a property of a formula P(Answer|Prompt, TrainingData) This is an associative statistical relation, which by definition, selects elements of TrainingData by-association with the Prompt. If the basis by which you understand LLMs is putting prompts into ChatGPT you're severely underqualified for drawing any conclusions about LLMs, and radically subject to confirmation bias.

kristiandupont 3y ago
Well sure, but I think that same thing can be said about the human brain. Obviously at a whole different level of sophistication and all, but the two really do seem related, at least to me. >People offering prompts as evidence are people who fundamentally do not understand the basics This I agree with, but I also don't see anyone doing that in this thread?
williamcotton 3y ago
You’re disregarding the emergent phenomena, which are not at all understood. There was a distinct and unpredicted jump in what can loosely be described as “cognitive abilities” between GPTs 2, 3, and 4, especially after some supervised techniques like RLHF.
- mjburgess 3y ago
  There is no "emergent phenomena" the pattern described is just the same as when you add +b to an ax+b model of linear data. ie., it's just fitting capacity. The "emergent boundary" is just an empirical measure of the necessary fitting capacity of these models on "everything ever digitised in english" given any particular functional requirement. All the language around this area is not scientific, nor are these practices. This is superstitious neophyte engineers, hopped up on scifi, by giddy VCs who love to be told they're funding captin picard.

deleted 3y ago

[deleted]

famouswaffles 3y ago

>Is there some predictive power that we gain by reducing LLM skills to mere token production side effects? No. If anything, we lose predictive power which is why it's extra silly.

intended 3y ago

Then why dont agents work? If those skills were real, why do they fizzle out on production data ?

TeMPOraL 3y ago
That's a good question. What is different about "production data"? What do those "production people" do that suddenly makes LLMs fail on things they work well on when not "production"?
- deleted 3y ago
  [deleted]
- intended 3y ago
  I can tell you who would love to hear your answer to that question. Me for starters. If it works, I can quit. Next up are Karpathy and the CTO of OpenAI. Around July and September both talked about production challenges. AI ops was the largest subcategory of fall YC startups. Every single ml and LLM ops individual who gets far enough deals with evals. I don’t know, but maybe - just maybe- the issue is that people don’t understand themselves enough, to avoid assuming too much of those emergent properties. As I recall there was also a paper that pointed out the issues with how LLMs are measured, and that the emergence of properties was not a step change once the tests were updated. Edit- found it: https://hai.stanford.edu/news/ais-ostensible-emergent-abilities-are-mirage https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...

scotty79 3y ago

> Is there some predictive power that we gain by reducing LLM skills to mere token production side effects? Yes. We can infer from this that out of sample hallucinations will be closer to the desired productions when measured by token productions metrics than by any other more informative domain relevant metrics. Which is exactly the case. If you ask llm to solve a math problem it haven't seen its response will be closer to the desired solution in its linguistic form rather than in mathematical meaning.

famouswaffles 3y ago
>If you ask llm to solve a math problem it haven't seen its response will be closer to the desired solution in its linguistic form rather than in mathematical meaning. This isn't true. Wild how people will confidently say nonsense about things they obviously haven't actually tested. GPT-4 can manage arbitrary arithmetic calculation closer to the real value than you could ever do without an external tool.
- scotty79 3y ago
  I don't mean calculations. I mean math problems. Like those: https://www.mathschool.com/locations/andover/news/prepare-for-math-competitions-with-problems-from-the-russian-math-olympiad https://www.mathschool.com/locations/andover/news/prepare-fo...
  - fragmede 3y ago
    ChatGPT-4 is quite capable of solving problems like that. Here's it solving the last two problems on the Grade 7-8 Russian Math Olympiad from the linked page. https://chat.openai.com/share/e90d4711-aa38-45c5-97e7-a9a04ea2294e https://chat.openai.com/share/e90d4711-aa38-45c5-97e7-a9a04e... https://chat.openai.com/share/0c9c579a-ca9f-4aa6-bac6-624372da6a62 https://chat.openai.com/share/0c9c579a-ca9f-4aa6-bac6-624372... The two answers (84 and 225,792) agree with the answer key. I only gave it the questions and didn't give it the PDF either, so it didn't cheat and just read the answer off the answer key in the PDFs.