3 ms·
I assume theyre searching the local gradient to see if theres a better descent before proceeding.
by cyanydeez 2mo ago
I assume theyre searching the local gradient to see if theres a better descent before proceeding.
- c0_0p_ 2mo agoI don't think there's anything like that going on. They just word vomit into a secondary area, and then there is an internal prompt that says "clean this up and summarize for the user".
- Groxx 2mo agoLess "internal prompt" and more "they are trained to summarize after a </think> token"
- astrange 2mo agoThe training methods try not to apply any particular rules to the contents of the thinking text. That's called "optimization pressure on CoT" and is thought to reduce safety by inducing the model to lie (or stop clearly printing its intentions) in the thinking text.
- eigenspace 2mo agoLLMs dont do gradient descent to generate tokens. They are trained by gradient descent, but inference doesnt involve it.