3 ms·
My limited and really non technical understanding of AI suggests that probability plays a very big role in AI behavior, and it makes sense to me that "reasoning
by Glyptodon 2mo ago
My limited and really non technical understanding of AI suggests that probability plays a very big role in AI behavior, and it makes sense to me that "reasoning" could be seen through a lens of creating additional content that will better constrain the probability distribution of the final output. In this case the "reasoning" done internally could be functionally bad or appear to be giberish so long as its effect on future generation is appropriate. To naive me who really doesn't know tons about AI this seems like it could be testable.
- pixl97 2mo agoThis has been a discussion in AI safety for a long time, that huge amounts of LLM reasoning could be for hidden goals outside of the actions we want. Safety training can have the perverse effect of commonly amplifying 'forbidden' answers in hidden/encoded paths as the model doesn't get rid of these behaviors but finds methods detecting training to avoid outputting bad tokens when it's being watched. Now, don't think of this of this like a conscious behavior like a kid trying not to get in trouble, but an emergent behavior of trying to repress particular output when that output is well connected to a massive amount of other tokens.