3 ms·

>> That seems like a disingenuous interpretation of what I was saying. I hope not. It was not at all obvious to me what you were trying to say. It really doesn

>> That seems like a disingenuous interpretation of what I was saying. I hope not. It was not at all obvious to me what you were trying to say. It really doesn't help communication to use metaphors and analogies when discussing technical matters, especially when there is simple and commonly accepted terminology. Trust me, I've learned that the hard way (i.e. repeated rejections of my research articles with very frustrated comments by reviewers). Besides, what you say is not right. Language models are not trained to optimise "a loss function that was designed and engineered to choose a model that optimally captures meaning relations in the data". Language models of the GPT line, specifically, are trained to maximise the following probability: P(tᵢ|tᵢ₋ₖ, ..., tᵢ₋₁) Where tᵢ is some token in the training corpus and tᵢ₋ₖ, ..., tᵢ₋₁ are the tokens in a "context window" of length k. That is actually the pre-training objective. The same probability is maximised in the supervised fine-tuning phase, but where tᵢ is replaced by a label and tᵢ₋ₖ, ..., tᵢ₋₁ is a sequence of tokens representing an instance with the given label. You can find this information in the first GPT paper, here: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised... See equations (1) and (4) on page 3 (with more careful notation than mine). Note that the above probabilities do not have anything to do with "meaning". They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do. You can search the entire GPT article above and you will not find any mention of "meaning". Nobody knows how to represent meaning and how to train a neural network on examples of it. It's much easier to train a neural net on text, of which there are no end of examples. That's what OpenAI has done. >> I don't have time or patience for this pseudo-intellectual nitpicking. Sorry, I meant that "sophisticated" is a loaded term. It's the kind of thing I hear often associated with socialites and gourmets. I find that it has no place in conversations about technology. It's the kind of expression people use when they try too hard to convince of the value of some technological artifact whose value should be clear without such exaggerations. So for example, you can talk about the complexity of large language models without having to say anything about their "sophistication", and that works well because one is measurable while the other is only a matter of opinion.

Mike_12345 4y ago
It's in the tile of the paper. "Improving Language Understanding" So they call it "understanding". No problem. Understanding natural language requires building a representation of the meaning encoded in language. Otherwise what exactly do you think they are trying to "understand" about natural language here? GPT is not a spell-checker. It has a semantic model of language. The stated goal is to optimize for "natural language understanding tasks", to use the exact phrasing in the article. The article refers to "semantic similarity" which is precisely about modelling the meaning of language, by definition of the term. Here's a definition of "semantic similarity" "In the context of Natural Language Processing (NLP), "semantic similarity" refers to the measure of how closely related the meanings of two linguistic units, such as words, phrases, sentences, or documents, are to each other. The goal of assessing semantic similarity is to quantify the degree to which the meanings of these linguistic units overlap or share common conceptual features. Semantic similarity plays a crucial role in various NLP tasks, including information retrieval, text summarization, document clustering, machine translation, and sentiment analysis. By understanding and quantifying the semantic relationships between linguistic units, NLP systems can provide more accurate results and better handle the complexities of natural language." Yes, the word "meaning" is not explicitly used in the article, but it is implied. Ironically you are nitpicking over semantics that GPT can already comprehend, since it contains a model of the meaning of the word "meaning". > They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do. GPT language models are designed and engineered to perform natural language understanding tasks. This is discussed in the paper that you linked to. In order to predict or generate the next token, it uses a very super sophisticated semantic model of natural language and the previous tokens. You could also argue that a human is just "predicting the next token" when they generate a sequence of words while doing the tasks that GPT is designed to do. That misses a lot of important details. The devil is in those details.
- deleted 4y ago
  [deleted]
- YeGoblynQueenne 4y ago
  >> It's in the tile of the paper. "Improving Language Understanding" I have to confess I'm a little bit disappointed. I pointed you to the OpenAI paper that has the technical details of how their models are trained, and you chose to concentrate on the title, and the claims made about the abilities of the model, rather than the way it actually works. The technical details about the model's training make it clear that it is a language model, trained to maximise the probability of a sequence of tokens, but you keep believing that it can "understand"? Or represent meaning? I'm not even sure what you believe. The point is that you shouldn't believe. The OpenAI paper is not gospel. It's a technical report. You should even be able to implement a system like the one they describe, though at smaller scale. Maybe that will help you understand how it really works? I don't know. I think you don't want to know how things work. You just want to believe. And I can't deal with faith. Faith makes no sense when you have a technical specification. Sorry.

Mike_12345 4y ago

It's in the tile of the paper. "Improving Language Understanding" So they call it "understanding". No problem. Understanding natural language requires building a representation of the meaning encoded in language. Otherwise what exactly do you think they are trying to "understand" about natural language here? GPT is not a spell-checker. It has a semantic model of language. The stated goal is to optimize for "natural language understanding tasks", to use the exact phrasing in the article. The article refers to "semantic similarity" which is precisely about modelling the meaning of language, by definition of the term. Here's a definition of "semantic similarity" "In the context of Natural Language Processing (NLP), "semantic similarity" refers to the measure of how closely related the meanings of two linguistic units, such as words, phrases, sentences, or documents, are to each other. The goal of assessing semantic similarity is to quantify the degree to which the meanings of these linguistic units overlap or share common conceptual features. Semantic similarity plays a crucial role in various NLP tasks, including information retrieval, text summarization, document clustering, machine translation, and sentiment analysis. By understanding and quantifying the semantic relationships between linguistic units, NLP systems can provide more accurate results and better handle the complexities of natural language." Yes, the word "meaning" is not explicitly used in the article, but it is implied. Ironically you are nitpicking over semantics that GPT can already comprehend, since it contains a model of the meaning of the word "meaning". > They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do. GPT language models are designed and engineered to perform natural language understanding tasks. This is discussed in the paper that you linked to. In order to predict or generate the next token, it uses a very super sophisticated semantic model of natural language and the previous tokens. You could also argue that a human is just "predicting the next token" when they generate a sequence of words while doing the tasks that GPT is designed to do. That misses a lot of important details. The devil is in those details.

deleted 4y ago
[deleted]
YeGoblynQueenne 4y ago
>> It's in the tile of the paper. "Improving Language Understanding" I have to confess I'm a little bit disappointed. I pointed you to the OpenAI paper that has the technical details of how their models are trained, and you chose to concentrate on the title, and the claims made about the abilities of the model, rather than the way it actually works. The technical details about the model's training make it clear that it is a language model, trained to maximise the probability of a sequence of tokens, but you keep believing that it can "understand"? Or represent meaning? I'm not even sure what you believe. The point is that you shouldn't believe. The OpenAI paper is not gospel. It's a technical report. You should even be able to implement a system like the one they describe, though at smaller scale. Maybe that will help you understand how it really works? I don't know. I think you don't want to know how things work. You just want to believe. And I can't deal with faith. Faith makes no sense when you have a technical specification. Sorry.