3 ms·
Interesting. Somehow it seems odd to add randomness (temperature) and then wash it away by averaging it out.
by infogulch 3y ago
Interesting. Somehow it seems odd to add randomness (temperature) and then wash it away by averaging it out.
- ewild 3y agotemp applies to each token so the range of temperature is significantly larger than the average being pulled
- naasking 3y agoIn optimization problems, randomness can often get you out of local minima/maxima, and so averaging out a bunch of random search paths might get you better results in the worst case. Something similar might be happening here. The training set will be biased in various ways that might create weird local min/max points and so this process could avoid those weird kinks.