3 ms·

>> It's in the tile of the paper. "Improving Language Understanding" I have to confess I'm a little bit disappointed. I pointed you to the OpenAI paper that ha

>> It's in the tile of the paper. "Improving Language Understanding" I have to confess I'm a little bit disappointed. I pointed you to the OpenAI paper that has the technical details of how their models are trained, and you chose to concentrate on the title, and the claims made about the abilities of the model, rather than the way it actually works. The technical details about the model's training make it clear that it is a language model, trained to maximise the probability of a sequence of tokens, but you keep believing that it can "understand"? Or represent meaning? I'm not even sure what you believe. The point is that you shouldn't believe. The OpenAI paper is not gospel. It's a technical report. You should even be able to implement a system like the one they describe, though at smaller scale. Maybe that will help you understand how it really works? I don't know. I think you don't want to know how things work. You just want to believe. And I can't deal with faith. Faith makes no sense when you have a technical specification. Sorry.

Mike_12345 4y ago
OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper. The loss function is involved in the training of a neural network (transformer). THE LOSS FUNCTION IS NOT THE MODEL ITSELF. The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step. This is a major blind spot in your current technical understanding of this paper and machine learning in general. The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture. "Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives. This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.
- YeGoblynQueenne 4y ago
  Right, never mind. No reason to keep this discussion going. Given the tone of the conversation I figured it's better to not leave many personal information here. I already got one "'stalker'" it appears.
  - Mike_12345 4y ago
    No I haven't taken anything personally. I am trying to explain how this stuff works. Go back and read your recent replies to me then see how condescending and ridiculous you were (accusing me of having blind faith, not wanting to know how things work, etc). I truly don't care. I'm just trying to explain something. Too bad if I have to be so blunt. No offence taken. It's actually more funny than offensive. I have a degree in Applied Mathematics and Computer Science. But so what? You don't need a degree to understand the basics. You're looking for an explicit semantic coding inside a loss function, which doesn't make any sense. Of course you won't find any coding for semantics described in the paper, because the entire point of machine learning is that it figures that out automatically. How the hell can you have a PhD in machine learning without understanding that?
- deleted 4y ago
  [deleted]

Mike_12345 4y ago

OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper. The loss function is involved in the training of a neural network (transformer). THE LOSS FUNCTION IS NOT THE MODEL ITSELF. The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step. This is a major blind spot in your current technical understanding of this paper and machine learning in general. The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture. "Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives. This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.

YeGoblynQueenne 4y ago
Right, never mind. No reason to keep this discussion going. Given the tone of the conversation I figured it's better to not leave many personal information here. I already got one "'stalker'" it appears.
- Mike_12345 4y ago
  No I haven't taken anything personally. I am trying to explain how this stuff works. Go back and read your recent replies to me then see how condescending and ridiculous you were (accusing me of having blind faith, not wanting to know how things work, etc). I truly don't care. I'm just trying to explain something. Too bad if I have to be so blunt. No offence taken. It's actually more funny than offensive. I have a degree in Applied Mathematics and Computer Science. But so what? You don't need a degree to understand the basics. You're looking for an explicit semantic coding inside a loss function, which doesn't make any sense. Of course you won't find any coding for semantics described in the paper, because the entire point of machine learning is that it figures that out automatically. How the hell can you have a PhD in machine learning without understanding that?
deleted 4y ago
[deleted]