4 ms·
> what they did here is take the core pre-trained GPT model, did Supervised Fine Tuning with Othello moves They didn't start with an existing model. They train
by fenomas 4y ago
> what they did here is take the core pre-trained GPT model, did Supervised Fine Tuning with Othello moves
They didn't start with an existing model. They trained a small GPT from scratch, so the resulting model had never seen any inputs except Othello moves.
- RC_ITR 4y agoGenerative "Pre-Trained" Transformer - GPT They did not start with a transformer that had arbitrary parameters, they started with a transformer that had been pre-trained.
- fenomas 4y agoPre-training refers to unsupervised training that's done before a model is fine-tuned. The model still starts out random before it's pre-trained. Here's where the Othello paper's weights are (randomly) initialized: https://github.com/likenneth/othello_world/blob/master/mingpt/model.py#L127 https://github.com/likenneth/othello_world/blob/master/mingp...