2 ms·
LLMs are trained to maximize the probability of correct guesses for the next token, not "ideas". You cannot define an idea as a training loss objective.
by minimaxir 2y ago
LLMs are trained to maximize the probability of correct guesses for the next token, not "ideas". You cannot define an idea as a training loss objective.
- mdp2021 2y agoThat is an architectural problem. If you want the post to be rephrased: it is paradoxical to have changes made near the output level, "changing words before it says them", given that the expected is to work with ideas. (And even then, selection would not be at the output level - it would be during the definition of the structure.) So, articles like this submission - while interesting from many points of view - make the elephant in the room more evident. > You cannot define an idea as a training loss objective What tells you so? If you see a technical limit, note e.g. that sentences and paragraphs can have their own position in an embedding space.
- orbital-decay 2y agoInterpretability studies offer several orthogonal ways to look at this, it's like Newtonian vs Lagrangian mechanics. Autoregressive token prediction, pattern matching, idea conceptualization, pathfinding in the extremely multidimensional space...