3 ms·
And interesting test. I didn’t see a mention of the temperature setting used. Temperature controls the probability to pick a token that isn’t the top prediction
by codeflo 3y ago
And interesting test. I didn’t see a mention of the temperature setting used. Temperature controls the probability to pick a token that isn’t the top prediction, which leads to more creative/less robotic results.
For actual input prediction, you probably want the temperature to be zero. But even a model as good as GPT-3 becomes very boring and repetitive with those settings.
- jackcook 3y agoI used greedy sampling (temperature 0) for all of them. Since I didn't have access to logits/probabilities for Apple's model, I wasn't able to do anything else in a way that would be fair.
- smpanaro 3y agoThere is an input for temperature in the CPU model. If you can find and hook the call to predict you can probably see what is being passed. Interestingly the Neural Engine version of the model does not take a temperature input, but it does output the raw embeddings in addition to the logits.