5 ms·
There are billions of sampling strategies for language models. The problem is that it is very difficult to empirically show that one sampling strategy is better
by bjourne 2y ago
There are billions of sampling strategies for language models. The problem is that it is very difficult to empirically show that one sampling strategy is better than standard top-k or top-p sampling. Minimizing perplexity is not enough to demonstrate superiority of a particular method. The strategy suggested in the blog post has the same issue. An innovation that sounds plausible in theory, but is unproven in practice.
- danielmarkbruce 2y agoProof isn't required. It's difficult to prove because it's difficult to state clearly what is "better" and it's expensive to collect preference data (or similar). You could use common sense after looking at lots of samples and say "this method seems to work better if you are trying to optimize for X".