2 ms·
Yes, but it's also something that proper training can fix, and that's the level at which the fix should probably be implemented. The current behavior amounts t
by antonvs 1y ago
Yes, but it's also something that proper training can fix, and that's the level at which the fix should probably be implemented.
The current behavior amounts to something like "attempt to complete the task at all costs," which is unlikely to provide good results, and in practice, often doesn't.
- samrus 1y agoBut are LLMs the right models to even be able to learn such long horizon goals and how to not cheat at them? I feel like we need a new base model where the next token prodiction itself is dynamical and RL based to be able to handle this issue properly
- antonvs 1y agoI was including RLHF in "training". And even the system prompt, really. If it's true that models can be prevented from spiraling into dead ends with "proper prompting" as the comment above claimed, then it's also true that this can be addressed earlier in the process. As it stands, this behavior isn't likely to be useful for any normal user, and it's certainly a blocker to "agentic" use.
- samrus 1y agoThe RLHF is happening too late i think. I think the reinforcement learning needs to be during the initial next token prodiction. On that note we need something to represent a complex world state than just language.