3 ms·

It's in the tile of the paper. "Improving Language Understanding" So they call it "understanding". No problem. Understanding natural language requires building

It's in the tile of the paper. "Improving Language Understanding" So they call it "understanding". No problem. Understanding natural language requires building a representation of the meaning encoded in language. Otherwise what exactly do you think they are trying to "understand" about natural language here? GPT is not a spell-checker. It has a semantic model of language. The stated goal is to optimize for "natural language understanding tasks", to use the exact phrasing in the article. The article refers to "semantic similarity" which is precisely about modelling the meaning of language, by definition of the term. Here's a definition of "semantic similarity" "In the context of Natural Language Processing (NLP), "semantic similarity" refers to the measure of how closely related the meanings of two linguistic units, such as words, phrases, sentences, or documents, are to each other. The goal of assessing semantic similarity is to quantify the degree to which the meanings of these linguistic units overlap or share common conceptual features. Semantic similarity plays a crucial role in various NLP tasks, including information retrieval, text summarization, document clustering, machine translation, and sentiment analysis. By understanding and quantifying the semantic relationships between linguistic units, NLP systems can provide more accurate results and better handle the complexities of natural language." Yes, the word "meaning" is not explicitly used in the article, but it is implied. Ironically you are nitpicking over semantics that GPT can already comprehend, since it contains a model of the meaning of the word "meaning". > They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do. GPT language models are designed and engineered to perform natural language understanding tasks. This is discussed in the paper that you linked to. In order to predict or generate the next token, it uses a very super sophisticated semantic model of natural language and the previous tokens. You could also argue that a human is just "predicting the next token" when they generate a sequence of words while doing the tasks that GPT is designed to do. That misses a lot of important details. The devil is in those details.

deleted 4y ago
[deleted]
YeGoblynQueenne 4y ago
>> It's in the tile of the paper. "Improving Language Understanding" I have to confess I'm a little bit disappointed. I pointed you to the OpenAI paper that has the technical details of how their models are trained, and you chose to concentrate on the title, and the claims made about the abilities of the model, rather than the way it actually works. The technical details about the model's training make it clear that it is a language model, trained to maximise the probability of a sequence of tokens, but you keep believing that it can "understand"? Or represent meaning? I'm not even sure what you believe. The point is that you shouldn't believe. The OpenAI paper is not gospel. It's a technical report. You should even be able to implement a system like the one they describe, though at smaller scale. Maybe that will help you understand how it really works? I don't know. I think you don't want to know how things work. You just want to believe. And I can't deal with faith. Faith makes no sense when you have a technical specification. Sorry.
- Mike_12345 4y ago
  OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper. The loss function is involved in the training of a neural network (transformer). THE LOSS FUNCTION IS NOT THE MODEL ITSELF. The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step. This is a major blind spot in your current technical understanding of this paper and machine learning in general. The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture. "Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives. This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.

deleted 4y ago

[deleted]

YeGoblynQueenne 4y ago

>> It's in the tile of the paper. "Improving Language Understanding" I have to confess I'm a little bit disappointed. I pointed you to the OpenAI paper that has the technical details of how their models are trained, and you chose to concentrate on the title, and the claims made about the abilities of the model, rather than the way it actually works. The technical details about the model's training make it clear that it is a language model, trained to maximise the probability of a sequence of tokens, but you keep believing that it can "understand"? Or represent meaning? I'm not even sure what you believe. The point is that you shouldn't believe. The OpenAI paper is not gospel. It's a technical report. You should even be able to implement a system like the one they describe, though at smaller scale. Maybe that will help you understand how it really works? I don't know. I think you don't want to know how things work. You just want to believe. And I can't deal with faith. Faith makes no sense when you have a technical specification. Sorry.

Mike_12345 4y ago
OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper. The loss function is involved in the training of a neural network (transformer). THE LOSS FUNCTION IS NOT THE MODEL ITSELF. The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step. This is a major blind spot in your current technical understanding of this paper and machine learning in general. The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture. "Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives. This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.