3 ms·

The problem that I see with all these different sampling techniques is the way people usually judge them. There are people who claim they work better, but no ri

The problem that I see with all these different sampling techniques is the way people usually judge them. There are people who claim they work better, but no rigorous benchmarks to prove it. Lots of "it writes better" or "the prose is fresh", but that is one argument where I think LeCun is 100% right - you can't judge a generalist model by "it works on poetry" or "prose", because that's the definition of bias, and you're shooting yourself in the foot with personal anecdotes. I'd like to see this applied to coding or math. See the samplers work better in say olympiad math problems, with thorough benchmarks before and after.

Der_Einzige 2y ago
The min_p paper and many other papers are doing exactly that.
- NitpickLawyer 2y ago
  Is this [1] the paper you're referring to? Unless I'm reading Table2 (page7 - pdf version) wrong, on math, min_p is shown to score worse than top_p. For temp 0.7 it scores 1 point lower than top_p. And from temps 1.0 and up, while scoring higher than top_p for the same temp, it scores way lower (6points and up) than top_p at 0.7. So overall, if you want accurate answers (and for math you kinda do), min_p is worse overall? Unless I miss-understand something. I agree with the authors that if you want a tradeoff between accuracy and diversity, min_p might help, but if you're looking for precise answers, the results will be slightly worse. It's a tradeoff, but as I said above, people often fail to mention it as such, and instead proclaim it to be "better" across the board. [1] - https://arxiv.org/pdf/2407.01082 https://arxiv.org/pdf/2407.01082
  - deleted 2y ago
    [deleted]
ninetyninenine 2y ago
If the objective is to make a better poet or a better story book writer than this flawed metric is the only form of measure. It’s the same measure we judge human writers on so it’s not necessarily the worst.

Der_Einzige 2y ago

The min_p paper and many other papers are doing exactly that.

NitpickLawyer 2y ago
Is this [1] the paper you're referring to? Unless I'm reading Table2 (page7 - pdf version) wrong, on math, min_p is shown to score worse than top_p. For temp 0.7 it scores 1 point lower than top_p. And from temps 1.0 and up, while scoring higher than top_p for the same temp, it scores way lower (6points and up) than top_p at 0.7. So overall, if you want accurate answers (and for math you kinda do), min_p is worse overall? Unless I miss-understand something. I agree with the authors that if you want a tradeoff between accuracy and diversity, min_p might help, but if you're looking for precise answers, the results will be slightly worse. It's a tradeoff, but as I said above, people often fail to mention it as such, and instead proclaim it to be "better" across the board. [1] - https://arxiv.org/pdf/2407.01082 https://arxiv.org/pdf/2407.01082
- deleted 2y ago
  [deleted]

ninetyninenine 2y ago

If the objective is to make a better poet or a better story book writer than this flawed metric is the only form of measure. It’s the same measure we judge human writers on so it’s not necessarily the worst.