3 ms·
Generative "Pre-Trained" Transformer - GPT They did not start with a transformer that had arbitrary parameters, they started with a transformer that had been p
by RC_ITR 4y ago
Generative "Pre-Trained" Transformer - GPT
They did not start with a transformer that had arbitrary parameters, they started with a transformer that had been pre-trained.
- fenomas 4y agoPre-training refers to unsupervised training that's done before a model is fine-tuned. The model still starts out random before it's pre-trained. Here's where the Othello paper's weights are (randomly) initialized: https://github.com/likenneth/othello_world/blob/master/mingpt/model.py#L127 https://github.com/likenneth/othello_world/blob/master/mingp...