A parameter refers to any trained value in the model. If you initialize it to a random number at the start of training - it's a parameter. In the context of tr

A parameter refers to any trained value in the model. If you initialize it to a random number at the start of training - it's a parameter. In the context of transformer language models, that includes the weights and biases in the feed-forward layers, as well as the input embeddings, positional encodings, and transformers query, key, and value matrixes.