2 ms·
Set the temperature to zero and that's exactly what you get. In some NN implementations, randomness is actually pretty important to keep the gradients from get
by CamperBob2 1y ago
Set the temperature to zero and that's exactly what you get.
In some NN implementations, randomness is actually pretty important to keep the gradients from getting stuck at local minima/maxima. Is that true for LLMs, or is it not something that applies at all?
- eru 1y agoAre you talking about training?
- CamperBob2 1y agoI'm not sure, hence the question. AFAIK temperature only comes into play at inference time once the distribution is known, but I don't know if there are other places where random numbers are involved.
- eru 1y agoYes, lots of other randomness. Eg you tend to randomly shuffle your corpus to train on. If you use drop-out (https://en.wikipedia.org/wiki/Dilution_(neural_networks) https://en.wikipedia.org/wiki/Dilution_(neural_networks)) you use randomness. You might also randomly perturb your training data. Lots of other sources of randomness that you might want to try.