OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actua

OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper. The loss function is involved in the training of a neural network (transformer). THE LOSS FUNCTION IS NOT THE MODEL ITSELF. The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step. This is a major blind spot in your current technical understanding of this paper and machine learning in general. The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture. "Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives. This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.