4 ms·
Though, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
by bunderbunder 3mo ago
Though, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
- beardedwizard 3mo agoExplain it to me
- jeremyjh 3mo agoWhich direction is the correlation? I don’t think this phrase means what people assume when it’s applied to post trained instruct models - which did not exist when the paper was written. After RL it is not predicting based on samples of the original corpus - but is also chasing a reward function that does require other features. There has been a lot of subsequent research that really calls many of the statements in this article into question.