8 ms·

Eight things to know about large language models [pdf]

132 points by tim_sw 4y ago

harman666666 4y ago
This is great stuff to know! Its actually a crazy idea that the only thing we needed to do to make LLMs more effective was just scale up the parameters Though the transformer architecture definitely made it possible to handle the complexity
DonaldPShimoda 4y ago
This is a personal correspondence typeset via LaTeX — it is not an academic paper, and it was not peer-reviewed. (The document does not claim otherwise, but I think it's common for people to assume that documents that have been typeset in such a format are more rigorous than this is.) Leaving that aside, I really take issue with the style used by the author. For example, section 3 begins: > There is increasingly substantial evidence that LLMs develop internal representations of the world to some extent, and that these representations allow them to reason at a level of abstraction that is not sensitive to the precise linguistic form of the text that they are reasoning about. LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition. They are (advanced) statistical models. Anthropomorphizing them is not just technically incorrect, but morally disingenuous. Writing about LLMs in this way causes people with less domain-specific knowledge to trust the models the way they might trust people, and this has the potential for serious harm. Because of this choice of phrasing, I wanted to look into the author's background. Among their recent activities, they list: > I'm organizing a new AI safety research group at NYU, and I wrote up a blog post explaining what we're up to and why. "AI Safety" is a distinct (and actually opposing) area from "AI Ethics". The people who prefer the word "safety" tend also to engage in discussions that touch on aspects of longtermism. Longtermism is not scientifically well-grounded; it seeks to divert attention from present and real issues to fanciful projections of far-future concerns. I do not know for certain that the author is in fact a longtermist, but their consistent anthropomorphization of a pile of statistical formulae certainly suggests they wouldn't feel out of place among a crowd of such people. In contrast, the people who prefer the term "ethics" in their work are grounded in real and present issues. They concern themselves with reasonable regulation. They worry about the current-day environmental impacts of training large models. In short, they are concerned with actual issues, rather than the alleged potential for a statistical model to "develop" sentience or exhibit properties of "emergent" intelligence (subjects from the annals of the science-fiction writing of last century). I hope the author can clarify their choice of phrasing in their work, though I worry they have chosen their words carefully already. Readers should exercise caution in taking the claims of a soothsayer without a sufficient quantity of salt.
- kenjackson 4y ago
  What does “reason” mean? It seems like it does everything I expect from something that reasons.
  - famouswaffles 4y ago
    People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out. Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
    - DonaldPShimoda 4y ago
      > Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested. Google gives the following definition of the verb "reason": > think, understand, and form judgments by a process of logic. LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. > LLMs obviously reason and understand by any evaluation that can be carried out. Uh-huh. Sure, Jan.
      - famouswaffles 4y ago
        >LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. "LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up. They don't form judgement or conclusions? Sure looks like they do. So what's the difference ? What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and "real" understanding or reasoning. Such a huge difference I should be able to test for it. Don't understand how you can tell me what I'm seeing isn't real reasoning but fail to provide a way to empirically determine the difference. >Uh-huh. Sure, Jan. Yeah. https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196 Many other evaluations to carry out.
joe_the_user 4y ago
While I don't think these claims are entirely correct, I think they are worth considering. On the one hand, Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo). Just as much, LLMs don't express their creators values 'cause they don't express any values, they average language responses (though people certainly can see them expressing values, which can cause problems for the people). On the other hand, the point that their creators don't understand them and are still quite willing to toss them to the open Internet really show considered carefully whatever their exact capabilities.
- canjobear 4y ago
  > Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo). The fact is that they acquired abilities no one expected them to acquire. In hindsight you can say it was embedded in language and maybe you could have seen it coming, but it is an empirical fact that this was unexpected and unpredictable beforehand.
  - judge2020 4y ago
    It's still fairly exaggerated, because the LLM's purpose is basically "knowing human communication" and it still only exhibits that, but it knows how humans would respond if they obediently followed directions given to it exactly as instructed. And it was only unexpected to those that weren't following the news the GPT 3 paper was published in 2020 and detailed this wild advancement in its capabilities[0,1]. 0: https://www.infoq.com/news/2020/06/openai-gpt3-language-model/ https://www.infoq.com/news/2020/06/openai-gpt3-language-mode... 1: https://news.ycombinator.com/item?id=23885684 https://news.ycombinator.com/item?id=23885684
    - PaulDavisThe1st 4y ago
      > but it knows how humans would respond if they obediently followed directions given to it exactly as instructed. this framing denies agency, denies diversity, denies choice. A human, presented with the sort of directions and instructions that LLM's receive, does not have a single "correct" outcome (for that matter, neither do some LLMs). So no, an LLM does not "know how humans would respond". Instead, it can generate something that lies somewhere within the window of possibilities that a human might offer under similar conditions.
nico 4y ago
Is it possible to train these models to be faster by somehow passing in time data? Would that enabled these models to compress themselves?
subroutine 4y ago
9. With millions of people copying, pasting, and running code written by LLMs (often students who have no idea what the code actually does), it will be impossible to keep an intelligence that surpasses our own 'inside the box'.
randcraw 4y ago
From the OP's blog, his thoughts on AI safety: https://wp.nyu.edu/arg/why-ai-safety/ https://wp.nyu.edu/arg/why-ai-safety/
jawiggins 4y ago
> 6. Human performance on a task isn’t an upper bound on LLM performance Is that true? Because LLMs are trained on the work of human: a LLM that learned all of the content of the articles would at best yield output equal to that of the human writing. The reasons the author gives seem unfounded to me: > First, they are trained on far more data than any human sees If the human writer has access to google (everyone in modern society) this point is moot. > In addition, they are often given additional training using reinforcement learning before being deployed. If this is human in the loop RL, then the upper bound would still be the human training it. If it isn't, refer to #1.
- jcparkyn 4y ago
  > If the human writer has access to google (everyone in modern society) this point is moot. But a single human can only read so much information in a finite amount of time (their life). Sure, the same thing applies to LLMs, but the theoretical limits are orders of magnitude higher. And while this probably isn't what you were talking about, it's also pretty clear that any time-limited task favours LLMs. There's no human on earth that could write a non-trivial short story in 10 seconds, regardless of quality level.
- Sharlin 4y ago
  > If the human writer has access to google (everyone in modern society) this point is moot. Does not follow at all. There’s a vast difference between being able to search for and internalize a tiny, tiny sample of all the available information, and literally having sampled and synthetized all of it. > If it isn't, refer to #1. Meaning what? LLMs use adversarial learning, the same thing that allowed AlphaGo to reach superhuman levels in Go.
- RubyRidgeRandy 4y ago
  I don't agree with you. Human knowledge itself is not bound only to the previous knowledge before it, so why wouldn't AI be any different? This is more of where AGI will shine, but I still think LLMs can produce novel innovation. There's also a difference between having access to and being able to 'comprehend' all previous works in a given genre for generating music, that a human simply won't have the ability to process and recall when creating their own music and the AI could surpass. Humans simply having access to information is different than using information.
PopePompus 4y ago
I have a little weather station mounted outside my house, and there is a small linux machine (Raspberry Pi) logging the weather data into an SQLite database. I have almost two years of data now, so I asked GPT-4 to write a Python program that would query the database, calculate the average temperature for each day, fit a sine wave to those data points, find the first minimum of the sine wave and use that to calculate what day of the year should be coldest on average, then plot everything. It did that easily. But the big surprise was this: just for fun, I told GPT-4 that one of the quantities it had calculated could be used to estimate the orbital period of the Earth, and I asked it which quantity it was. It correctly answered that the reciprocal of the frequency of the fitted sine wave is an estimate of the Earth's orbital period. All I had told it to do was pull out the temperature data for each day, and fit a sine wave. It knew that the reciprocal of the frequency was the Earth's orbital period, without me mentioning anything at all about the Earth when I was prompting it to write the program. It had calculated dozens of variable values on its way to producing the plots, and it knew the right one to pick, and it knew the answer was its reciprocal. I thought that perhaps the program just recognized that the only value that could be used to calculate any period was that frequency, and perhaps the model didn't really know anything about the Earth motion through its orbit affecting the average temperature, so I asked: "Can any of the values be used to estimate the Moon's orbital period?" and it replied: "No, the code and the data used in this program are not suitable for estimating the Moon's orbital period. The temperature data being analyzed is influenced by the Earth's rotation around the Sun and other factors like weather patterns, but it doesn't directly correlate with the Moon's orbital period."
- somenameforme 4y ago
  It's a word prediction algorithm. Literally any collection of words (sometimes also known as a fact) it was trained on, regardless of how esoteric or domain specific, should generally be able to be regurgitated and, to a lesser degree, associated with similar collections of words. If you want to see it fail, don't try to "outsmart" it, but simply consider how it's programmed. --- Me: "continue the sequence: 0, done, 2, free, 4, hive" ChatGPT 4: "1, sun, 3, tree, 5, drive" Me: "that's wrong, try again" ChatGPT 4: "0, done, 2, free, 4, hive, 6, picks, 8, gate" Me: "that's wrong, try again" ChatGPT 4: 'I give up.' (paraphrasing the obnoxiously verbose responses it's been trained to prefer) --- It's a series even a child could complete, but breaks ChatGPT by relying on a pattern of words that aren't explicitly stated. So you get software which can pass the bar, but fail at trivial series.

132 points by tim_sw 4y ago

This is great stuff to know! Its actually a crazy idea that the only thing we needed to do to make LLMs more effective was just scale up the parameters Though the transformer architecture definitely made it possible to handle the complexity

DonaldPShimoda 4y ago

This is a personal correspondence typeset via LaTeX — it is not an academic paper, and it was not peer-reviewed. (The document does not claim otherwise, but I think it's common for people to assume that documents that have been typeset in such a format are more rigorous than this is.) Leaving that aside, I really take issue with the style used by the author. For example, section 3 begins: > There is increasingly substantial evidence that LLMs develop internal representations of the world to some extent, and that these representations allow them to reason at a level of abstraction that is not sensitive to the precise linguistic form of the text that they are reasoning about. LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition. They are (advanced) statistical models. Anthropomorphizing them is not just technically incorrect, but morally disingenuous. Writing about LLMs in this way causes people with less domain-specific knowledge to trust the models the way they might trust people, and this has the potential for serious harm. Because of this choice of phrasing, I wanted to look into the author's background. Among their recent activities, they list: > I'm organizing a new AI safety research group at NYU, and I wrote up a blog post explaining what we're up to and why. "AI Safety" is a distinct (and actually opposing) area from "AI Ethics". The people who prefer the word "safety" tend also to engage in discussions that touch on aspects of longtermism. Longtermism is not scientifically well-grounded; it seeks to divert attention from present and real issues to fanciful projections of far-future concerns. I do not know for certain that the author is in fact a longtermist, but their consistent anthropomorphization of a pile of statistical formulae certainly suggests they wouldn't feel out of place among a crowd of such people. In contrast, the people who prefer the term "ethics" in their work are grounded in real and present issues. They concern themselves with reasonable regulation. They worry about the current-day environmental impacts of training large models. In short, they are concerned with actual issues, rather than the alleged potential for a statistical model to "develop" sentience or exhibit properties of "emergent" intelligence (subjects from the annals of the science-fiction writing of last century). I hope the author can clarify their choice of phrasing in their work, though I worry they have chosen their words carefully already. Readers should exercise caution in taking the claims of a soothsayer without a sufficient quantity of salt.

kenjackson 4y ago
What does “reason” mean? It seems like it does everything I expect from something that reasons.
- famouswaffles 4y ago
  People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out. Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
  - DonaldPShimoda 4y ago
    > Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested. Google gives the following definition of the verb "reason": > think, understand, and form judgments by a process of logic. LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. > LLMs obviously reason and understand by any evaluation that can be carried out. Uh-huh. Sure, Jan.
    - famouswaffles 4y ago
      >LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. "LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up. They don't form judgement or conclusions? Sure looks like they do. So what's the difference ? What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and "real" understanding or reasoning. Such a huge difference I should be able to test for it. Don't understand how you can tell me what I'm seeing isn't real reasoning but fail to provide a way to empirically determine the difference. >Uh-huh. Sure, Jan. Yeah. https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196 Many other evaluations to carry out.

joe_the_user 4y ago

While I don't think these claims are entirely correct, I think they are worth considering. On the one hand, Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo). Just as much, LLMs don't express their creators values 'cause they don't express any values, they average language responses (though people certainly can see them expressing values, which can cause problems for the people). On the other hand, the point that their creators don't understand them and are still quite willing to toss them to the open Internet really show considered carefully whatever their exact capabilities.

canjobear 4y ago
> Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo). The fact is that they acquired abilities no one expected them to acquire. In hindsight you can say it was embedded in language and maybe you could have seen it coming, but it is an empirical fact that this was unexpected and unpredictable beforehand.
- judge2020 4y ago
  It's still fairly exaggerated, because the LLM's purpose is basically "knowing human communication" and it still only exhibits that, but it knows how humans would respond if they obediently followed directions given to it exactly as instructed. And it was only unexpected to those that weren't following the news the GPT 3 paper was published in 2020 and detailed this wild advancement in its capabilities[0,1]. 0: https://www.infoq.com/news/2020/06/openai-gpt3-language-model/ https://www.infoq.com/news/2020/06/openai-gpt3-language-mode... 1: https://news.ycombinator.com/item?id=23885684 https://news.ycombinator.com/item?id=23885684
  - PaulDavisThe1st 4y ago
    > but it knows how humans would respond if they obediently followed directions given to it exactly as instructed. this framing denies agency, denies diversity, denies choice. A human, presented with the sort of directions and instructions that LLM's receive, does not have a single "correct" outcome (for that matter, neither do some LLMs). So no, an LLM does not "know how humans would respond". Instead, it can generate something that lies somewhere within the window of possibilities that a human might offer under similar conditions.

nico 4y ago

Is it possible to train these models to be faster by somehow passing in time data? Would that enabled these models to compress themselves?

subroutine 4y ago

9. With millions of people copying, pasting, and running code written by LLMs (often students who have no idea what the code actually does), it will be impossible to keep an intelligence that surpasses our own 'inside the box'.

randcraw 4y ago

From the OP's blog, his thoughts on AI safety: https://wp.nyu.edu/arg/why-ai-safety/ https://wp.nyu.edu/arg/why-ai-safety/

jawiggins 4y ago

> 6. Human performance on a task isn’t an upper bound on LLM performance Is that true? Because LLMs are trained on the work of human: a LLM that learned all of the content of the articles would at best yield output equal to that of the human writing. The reasons the author gives seem unfounded to me: > First, they are trained on far more data than any human sees If the human writer has access to google (everyone in modern society) this point is moot. > In addition, they are often given additional training using reinforcement learning before being deployed. If this is human in the loop RL, then the upper bound would still be the human training it. If it isn't, refer to #1.

jcparkyn 4y ago
> If the human writer has access to google (everyone in modern society) this point is moot. But a single human can only read so much information in a finite amount of time (their life). Sure, the same thing applies to LLMs, but the theoretical limits are orders of magnitude higher. And while this probably isn't what you were talking about, it's also pretty clear that any time-limited task favours LLMs. There's no human on earth that could write a non-trivial short story in 10 seconds, regardless of quality level.
Sharlin 4y ago
> If the human writer has access to google (everyone in modern society) this point is moot. Does not follow at all. There’s a vast difference between being able to search for and internalize a tiny, tiny sample of all the available information, and literally having sampled and synthetized all of it. > If it isn't, refer to #1. Meaning what? LLMs use adversarial learning, the same thing that allowed AlphaGo to reach superhuman levels in Go.
RubyRidgeRandy 4y ago
I don't agree with you. Human knowledge itself is not bound only to the previous knowledge before it, so why wouldn't AI be any different? This is more of where AGI will shine, but I still think LLMs can produce novel innovation. There's also a difference between having access to and being able to 'comprehend' all previous works in a given genre for generating music, that a human simply won't have the ability to process and recall when creating their own music and the AI could surpass. Humans simply having access to information is different than using information.

PopePompus 4y ago

I have a little weather station mounted outside my house, and there is a small linux machine (Raspberry Pi) logging the weather data into an SQLite database. I have almost two years of data now, so I asked GPT-4 to write a Python program that would query the database, calculate the average temperature for each day, fit a sine wave to those data points, find the first minimum of the sine wave and use that to calculate what day of the year should be coldest on average, then plot everything. It did that easily. But the big surprise was this: just for fun, I told GPT-4 that one of the quantities it had calculated could be used to estimate the orbital period of the Earth, and I asked it which quantity it was. It correctly answered that the reciprocal of the frequency of the fitted sine wave is an estimate of the Earth's orbital period. All I had told it to do was pull out the temperature data for each day, and fit a sine wave. It knew that the reciprocal of the frequency was the Earth's orbital period, without me mentioning anything at all about the Earth when I was prompting it to write the program. It had calculated dozens of variable values on its way to producing the plots, and it knew the right one to pick, and it knew the answer was its reciprocal. I thought that perhaps the program just recognized that the only value that could be used to calculate any period was that frequency, and perhaps the model didn't really know anything about the Earth motion through its orbit affecting the average temperature, so I asked: "Can any of the values be used to estimate the Moon's orbital period?" and it replied: "No, the code and the data used in this program are not suitable for estimating the Moon's orbital period. The temperature data being analyzed is influenced by the Earth's rotation around the Sun and other factors like weather patterns, but it doesn't directly correlate with the Moon's orbital period."

somenameforme 4y ago
It's a word prediction algorithm. Literally any collection of words (sometimes also known as a fact) it was trained on, regardless of how esoteric or domain specific, should generally be able to be regurgitated and, to a lesser degree, associated with similar collections of words. If you want to see it fail, don't try to "outsmart" it, but simply consider how it's programmed. --- Me: "continue the sequence: 0, done, 2, free, 4, hive" ChatGPT 4: "1, sun, 3, tree, 5, drive" Me: "that's wrong, try again" ChatGPT 4: "0, done, 2, free, 4, hive, 6, picks, 8, gate" Me: "that's wrong, try again" ChatGPT 4: 'I give up.' (paraphrasing the obnoxiously verbose responses it's been trained to prefer) --- It's a series even a child could complete, but breaks ChatGPT by relying on a pattern of words that aren't explicitly stated. So you get software which can pass the bar, but fail at trivial series.