3 ms·
You don't iterate on the model weights, you train them on billions of tokens. There was no real iteration, you set it up and wait for the GPUs to do the work. T
by spullara 3y ago
You don't iterate on the model weights, you train them on billions of tokens. There was no real iteration, you set it up and wait for the GPUs to do the work. The model architecture itself is done before you start training. If you change it, you generally have to start training from scratch.
You do get the ability to further train the model using fine-tuning or LoRas.
- monocasa 3y agoThere's absolutely state space iteration in model training. Layer sizes, composition, construction. > There was no real iteration, you set it up and wait for the GPUs to do the work. The model architecture itself is done before you start training. If you change it, you generally have to start training from scratch. That's like saying there's no design iteration in software because you type 'make' and the executable is built.
- spullara 3y agoThe point is that iteration is before training. You don't iterate post starting the training and presumably this person doesn't want to train the model themselves. If they do, they can do that but they need to get the training data and a huge amount of GPUs.
- monocasa 3y agoOnce again, yes, the training cycle is more like a compile step. Design iteration of the shape of the model absolutely occurs with partially trained weights however.