3 ms·

Not a huge fanboy of LLMs, but is the name really all that misleading? It's a language model, a model designed to produce human language, and it is large, with

Not a huge fanboy of LLMs, but is the name really all that misleading? It's a language model, a model designed to produce human language, and it is large, with billions of parameters. So it's quite descriptive.

gmac 3y ago
The point of the linked tweet (admittedly not obvious from the title here) is that AI is the misleading term.
- moffkalast 3y ago
  But it is artificial and intelligent? Also rather descriptive imo. One might also call it general.
  - t-3 3y ago
    Not intelligent. It cannot reason anything, it can only regurgitate and synthesize words without any understanding.
    - moffkalast 3y ago
      Even if that were true, baked-in reasoning is still reasoning. A simple state machine can be intelligent as well if you go by definition.
      - deleted 3y ago
        [deleted]
YeGoblynQueenne 3y ago
The term "language model" is ... well, let's be kind and not say misleading. Let's say it is over-ambitious given the current technological capabilities. What language models, model, is text, not language. More specifically, language models do not model the human language ability, i.e. the ability of humans to recognise and generate utterances in a natural language, like English. This difference is not some terminological quibble and it has very practical implications about what can be achieved by modelling text, rather than language. See, text is the product of language, but it isn't language. In technical terms, in the context of a corpus of text used to train a language model, language is a hidden, or latent variable that is not directly observable in the corpus and so cannot be modelled directly. That a hidden variable can "not be modelled directly" means that it must be inferred from observations of other variables, to which it is correlated. There are two problems with this, with respect to LLMs trained by Transformers. The first problem is that Transformers are not the right approach to model hidden variables. There are machine learning approaches that can be used to learn hidden variable models, for example various forms of Expectation-Maximisation, like the Viterbi algorithm; Hidden Markov Models; Latent Dirichlet Allocation; etc. But Transformers in particular are not a latent variable model. The second problem is that even latent variable models must be trained with examples of hidden variables in order to be able to predict the distribution of hidden variables in new data, not seen during training. What this means is that in order to train a latent variable model to predict language from text, one needs examples of language, not just text. And where are those examples of language? Remember that when I say "language" I'm talking about the human language ability, which is what, ultimately, produces all language generated by humans. Well, we do not have access to any examples of that human language ability. All we have is examples of its products, or maybe even byproducts. So we can't model that ability. tl;dr: We can't model language, only text, because we can observe no examples of language, only text.
- jiggawatts 3y ago
  If "we" can't observe examples of language, then it doesn't exist. We -- and by dear God I hope you mean we humans -- observe text. Obviously, audio also. Voice! I mean voice! Just in case you weren't aware of how humans communicate. We use our voices. Also, writing. Text, in other words. I suppose we have some hand-gestures, facial expressions, but that's about it. Some crude humans fart and burp to make a point, but this is rare. These LLMs learned from text, much the same as students -- human students -- learn from text... books. Not "language books", I want to clarify. Textbooks. We learn from web sites, and collections of texts called libraries. Also, Wikipedia, these days. You've heard of it? LLMs have read those web sites, and those books also. How is this different from what we humans do? Or is there a special group of... creatures? Beings? Others? that use something other than text and voice to communicate? Some secret language? Please tell me you're not one of... "them".
  - YeGoblynQueenne 3y ago
    Your flippant attitude is encouraged by ignorance, not knowledge. I'd be happy to help you address your lack of knowledge, but - flippant. Tant pis pour toi.
    - jiggawatts 3y ago
      Using text to transfer that knowledge? Text as in a comment here on HN? I thought you just said that can’t ever work.
- a1j9o94 3y ago
  I appreciate your perspective, but I have a slightly different view. First, it's important to recognize that the term "language model," as it's used in the context of machine learning, does not claim to fully capture the entirety of human language ability. Rather, it refers to a model's ability to predict or generate text that follows the patterns found in the language data on which it was trained. You mention that a language model can't model language, only text, because it only observes examples of text, not language. However, I would argue that these text examples are in fact representations of human language ability. The text that language models are trained on is a product of human language ability. This text, while not a perfect representation of language, is a direct outcome of language use and thus carries within it the patterns and structures that language models can learn. Moreover, models like Transformers do have a sort of latent space - the embeddings space. This is a continuous space where words and phrases are represented as vectors. The distances and directions between vectors in this space capture semantic and syntactic relationships, which suggests that these models are indeed learning some aspects of language, not just text. On top of this, many of these models are trained on diverse data sources, including transcriptions of spoken conversations. This introduces elements of pragmatics, or how language is used in real-world conversations, into the training data. Finally, the translation capabilities of these models further suggest that they are capturing something beyond mere text. They are able to translate between different languages, which implies that they are learning some underlying linguistic structures that are shared across languages. Again, it's important to stress that these models are far from capturing the full complexity of human language ability. However, to say that they only model text and not language seems to me an oversimplification. They are learning patterns and structures in the data that are intrinsically tied to human language use, and so in a sense, they are modeling aspects of language.

gmac 3y ago

The point of the linked tweet (admittedly not obvious from the title here) is that AI is the misleading term.

moffkalast 3y ago
But it is artificial and intelligent? Also rather descriptive imo. One might also call it general.
- t-3 3y ago
  Not intelligent. It cannot reason anything, it can only regurgitate and synthesize words without any understanding.
  - moffkalast 3y ago
    Even if that were true, baked-in reasoning is still reasoning. A simple state machine can be intelligent as well if you go by definition.
    - deleted 3y ago
      [deleted]

YeGoblynQueenne 3y ago

The term "language model" is ... well, let's be kind and not say misleading. Let's say it is over-ambitious given the current technological capabilities. What language models, model, is text, not language. More specifically, language models do not model the human language ability, i.e. the ability of humans to recognise and generate utterances in a natural language, like English. This difference is not some terminological quibble and it has very practical implications about what can be achieved by modelling text, rather than language. See, text is the product of language, but it isn't language. In technical terms, in the context of a corpus of text used to train a language model, language is a hidden, or latent variable that is not directly observable in the corpus and so cannot be modelled directly. That a hidden variable can "not be modelled directly" means that it must be inferred from observations of other variables, to which it is correlated. There are two problems with this, with respect to LLMs trained by Transformers. The first problem is that Transformers are not the right approach to model hidden variables. There are machine learning approaches that can be used to learn hidden variable models, for example various forms of Expectation-Maximisation, like the Viterbi algorithm; Hidden Markov Models; Latent Dirichlet Allocation; etc. But Transformers in particular are not a latent variable model. The second problem is that even latent variable models must be trained with examples of hidden variables in order to be able to predict the distribution of hidden variables in new data, not seen during training. What this means is that in order to train a latent variable model to predict language from text, one needs examples of language, not just text. And where are those examples of language? Remember that when I say "language" I'm talking about the human language ability, which is what, ultimately, produces all language generated by humans. Well, we do not have access to any examples of that human language ability. All we have is examples of its products, or maybe even byproducts. So we can't model that ability. tl;dr: We can't model language, only text, because we can observe no examples of language, only text.

jiggawatts 3y ago
If "we" can't observe examples of language, then it doesn't exist. We -- and by dear God I hope you mean we humans -- observe text. Obviously, audio also. Voice! I mean voice! Just in case you weren't aware of how humans communicate. We use our voices. Also, writing. Text, in other words. I suppose we have some hand-gestures, facial expressions, but that's about it. Some crude humans fart and burp to make a point, but this is rare. These LLMs learned from text, much the same as students -- human students -- learn from text... books. Not "language books", I want to clarify. Textbooks. We learn from web sites, and collections of texts called libraries. Also, Wikipedia, these days. You've heard of it? LLMs have read those web sites, and those books also. How is this different from what we humans do? Or is there a special group of... creatures? Beings? Others? that use something other than text and voice to communicate? Some secret language? Please tell me you're not one of... "them".
- YeGoblynQueenne 3y ago
  Your flippant attitude is encouraged by ignorance, not knowledge. I'd be happy to help you address your lack of knowledge, but - flippant. Tant pis pour toi.
  - jiggawatts 3y ago
    Using text to transfer that knowledge? Text as in a comment here on HN? I thought you just said that can’t ever work.
a1j9o94 3y ago
I appreciate your perspective, but I have a slightly different view. First, it's important to recognize that the term "language model," as it's used in the context of machine learning, does not claim to fully capture the entirety of human language ability. Rather, it refers to a model's ability to predict or generate text that follows the patterns found in the language data on which it was trained. You mention that a language model can't model language, only text, because it only observes examples of text, not language. However, I would argue that these text examples are in fact representations of human language ability. The text that language models are trained on is a product of human language ability. This text, while not a perfect representation of language, is a direct outcome of language use and thus carries within it the patterns and structures that language models can learn. Moreover, models like Transformers do have a sort of latent space - the embeddings space. This is a continuous space where words and phrases are represented as vectors. The distances and directions between vectors in this space capture semantic and syntactic relationships, which suggests that these models are indeed learning some aspects of language, not just text. On top of this, many of these models are trained on diverse data sources, including transcriptions of spoken conversations. This introduces elements of pragmatics, or how language is used in real-world conversations, into the training data. Finally, the translation capabilities of these models further suggest that they are capturing something beyond mere text. They are able to translate between different languages, which implies that they are learning some underlying linguistic structures that are shared across languages. Again, it's important to stress that these models are far from capturing the full complexity of human language ability. However, to say that they only model text and not language seems to me an oversimplification. They are learning patterns and structures in the data that are intrinsically tied to human language use, and so in a sense, they are modeling aspects of language.