3 ms·
Yes that's right, it seems like an area of more research. Honestly it goes counter to the Bitter Lesson (http://www.incompleteideas.net/IncIdeas/BitterLesson.h
by trq_ 2y ago
Yes that's right, it seems like an area of more research.
Honestly it goes counter to the Bitter Lesson (http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html, which stems from getting too fancy about maze traversal in Chess. But at the scale LLMs are at right now, the improvements might be worth it.
- menhguin 2y agoHi, contributor to Entropix here. This is just my opinion, but I don't think it goes counter to the Bitter Lesson at all, because it's meant to leverage model computation capabilities. Several papers have suggested that models internally compute certainty (https://arxiv.org/abs/2406.16254 https://arxiv.org/abs/2406.16254), and in my view our method simply leverages this computation and factors it explicitly into decoding. This is as opposed to pure sampling + next token prediction which basically randomly chooses a token. So if a model does 1274 x 8275 and it's not very sure of the answer, it still confidently gives an answer even though it's uncertain and needs to do more working.
- danielmarkbruce 2y ago100%. It's in line with bitter lesson learnings. Good going.
- danielmarkbruce 2y agoYeah i don't think it's counter at all. The bitter lesson calls out the fact that more computation/search wins.