3 ms·
I don't know how far "pure" next-token prediction can go with planning, although I wouldn't count them out until their performance starts noticeably plateauing.
by lucubratory 3y ago
I don't know how far "pure" next-token prediction can go with planning, although I wouldn't count them out until their performance starts noticeably plateauing. But the tree of thought architecture is a very similar concept to what you're discussing, you should definitely give it a read: https://arxiv.org/abs/2305.10601 https://arxiv.org/abs/2305.10601
It's not everything involved in traditional planning, but it may be a framework to use more traditional planning algorithms on LLM output.
- glun 3y agoTheyre already noticably plataeuing when it comes to coding, which similarly requires you to navigate tree structures and plan how to connect existing things together using shortest paths rather than inventing things out of thin air.
- lucubratory 3y agoI would love to see the evidence for plateauing when it comes to coding! Specifically, I'd like to see that new larger models are not achieving performance increases relative to previous smaller models.