4 ms·
If you mean “not what we want” for safety reasons, I think I agree. If you don’t mean for safety reasons, I’m not sure why.
by drdeca 2y ago
If you mean “not what we want” for safety reasons, I think I agree.
If you don’t mean for safety reasons, I’m not sure why.
- miven 2y agoIn section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models I'm not sure whether it wouldn't be more reliable to let the model run on latents and try to train a separate latent-reading explainer module that has at least some approximation of what we want as an explicit optimization objective. Assuming it actually is or has the potential to be better than CoT, from what I gathered from the paper the current results are mostly just more efficient token-wise. [1] https://arxiv.org/abs/2305.04388 https://arxiv.org/abs/2305.04388
- DalasNoin 2y agoI was thinking abut safety reasons, but also usability. Seems like a pretty big difference to me if you don't understand the chain of thought. How faithful cot are is another question.