3 ms·

These sampling based techniques is a rare occasion where experimenting with consumer hardware can let you improve on SOTA models. I don't think it will last - t

by zby 2y ago

These sampling based techniques is a rare occasion where experimenting with consumer hardware can let you improve on SOTA models. I don't think it will last - the end game surely will be a trainable sampler. But for now - enjoy tinkering: https://github.com/codelion/optillm https://github.com/codelion/optillm implements a few of these techniques optillm authors suggest that the additional computations in Entropics don’t bring any better results in comparison with the simple CoT decoding (but I am not sure if they also check efficiency):https://x.com/asankhaya/status/1846736390152949966 https://x.com/asankhaya/status/1846736390152949966 It looks to me that many problems with LLMs come from something like semantic leaking, or distraction by irrelevant information (like in the GSM Symbolic paper) - maybe there is some space for improving attention too. I wrote a couple of blog posts on these subjects: https://zzbbyy.substack.com/p/semantic-leakage-quick-notes https://zzbbyy.substack.com/p/semantic-leakage-quick-notes, https://zzbbyy.substack.com/p/llms-and-reasoning https://zzbbyy.substack.com/p/llms-and-reasoning, https://zzbbyy.substack.com/p/o1-inference-time-turing-machines https://zzbbyy.substack.com/p/o1-inference-time-turing-machi...

NitpickLawyer 2y ago
The problem that I see with all these different sampling techniques is the way people usually judge them. There are people who claim they work better, but no rigorous benchmarks to prove it. Lots of "it writes better" or "the prose is fresh", but that is one argument where I think LeCun is 100% right - you can't judge a generalist model by "it works on poetry" or "prose", because that's the definition of bias, and you're shooting yourself in the foot with personal anecdotes. I'd like to see this applied to coding or math. See the samplers work better in say olympiad math problems, with thorough benchmarks before and after.
- Der_Einzige 2y ago
  The min_p paper and many other papers are doing exactly that.
  - NitpickLawyer 2y ago
    Is this [1] the paper you're referring to? Unless I'm reading Table2 (page7 - pdf version) wrong, on math, min_p is shown to score worse than top_p. For temp 0.7 it scores 1 point lower than top_p. And from temps 1.0 and up, while scoring higher than top_p for the same temp, it scores way lower (6points and up) than top_p at 0.7. So overall, if you want accurate answers (and for math you kinda do), min_p is worse overall? Unless I miss-understand something. I agree with the authors that if you want a tradeoff between accuracy and diversity, min_p might help, but if you're looking for precise answers, the results will be slightly worse. It's a tradeoff, but as I said above, people often fail to mention it as such, and instead proclaim it to be "better" across the board. [1] - https://arxiv.org/pdf/2407.01082 https://arxiv.org/pdf/2407.01082
    - deleted 2y ago
      [deleted]
- ninetyninenine 2y ago
  If the objective is to make a better poet or a better story book writer than this flawed metric is the only form of measure. It’s the same measure we judge human writers on so it’s not necessarily the worst.
trq_ 2y ago
This is incredible! I haven't seen that repo yet, thank you for pointing it out, and the writing
scellus 2y ago
Semantic leakage could be just weakness of the model, and related to claims that they don't _really_ reason. Maybe more training could help. Or maybe it's a more fundamental weakness of the attention mechanism? (There are alternatives to that now.)

by zby 2y ago

NitpickLawyer 2y ago

The problem that I see with all these different sampling techniques is the way people usually judge them. There are people who claim they work better, but no rigorous benchmarks to prove it. Lots of "it writes better" or "the prose is fresh", but that is one argument where I think LeCun is 100% right - you can't judge a generalist model by "it works on poetry" or "prose", because that's the definition of bias, and you're shooting yourself in the foot with personal anecdotes. I'd like to see this applied to coding or math. See the samplers work better in say olympiad math problems, with thorough benchmarks before and after.

Der_Einzige 2y ago
The min_p paper and many other papers are doing exactly that.
- NitpickLawyer 2y ago
  Is this [1] the paper you're referring to? Unless I'm reading Table2 (page7 - pdf version) wrong, on math, min_p is shown to score worse than top_p. For temp 0.7 it scores 1 point lower than top_p. And from temps 1.0 and up, while scoring higher than top_p for the same temp, it scores way lower (6points and up) than top_p at 0.7. So overall, if you want accurate answers (and for math you kinda do), min_p is worse overall? Unless I miss-understand something. I agree with the authors that if you want a tradeoff between accuracy and diversity, min_p might help, but if you're looking for precise answers, the results will be slightly worse. It's a tradeoff, but as I said above, people often fail to mention it as such, and instead proclaim it to be "better" across the board. [1] - https://arxiv.org/pdf/2407.01082 https://arxiv.org/pdf/2407.01082
  - deleted 2y ago
    [deleted]
ninetyninenine 2y ago
If the objective is to make a better poet or a better story book writer than this flawed metric is the only form of measure. It’s the same measure we judge human writers on so it’s not necessarily the worst.

trq_ 2y ago

This is incredible! I haven't seen that repo yet, thank you for pointing it out, and the writing

scellus 2y ago

Semantic leakage could be just weakness of the model, and related to claims that they don't _really_ reason. Maybe more training could help. Or maybe it's a more fundamental weakness of the attention mechanism? (There are alternatives to that now.)