4 ms·
> Most models do something called top-p not beam search these days now im not so sure you know what youre talking about. gpt4 at least is rumored to use beam s
by swyx 3y ago
> Most models do something called top-p not beam search these days
now im not so sure you know what youre talking about. gpt4 at least is rumored to use beam search. as i understand this is more sophisticated than top-p, so thats noteworthy
- spmurrayzzz 3y agoWhile I agree that its an overstatement to say beam search is a minor/worthless detail, its not as cut and dry as "beam search > top-p". You will get wildly variant results with both sampling strategies depending on the objective function that any given model was trained on, in addition to the type of task domain you're expecting the model to perform. Just to cherry pick one domain: if you're doing any creative work (storytelling, roleplaying, etc), top-p may be a better sampling strategy because you will almost always get more diverse output. But since it's much less predictable, you probably don't want that sort of strategy for summarization or sentiment analysis.