Test-Time Training is incredibly powerful. Most recently, it has been shown that Self-Attention can in fact be viewed through the lens of test-time training, wi

Test-Time Training is incredibly powerful. Most recently, it has been shown that Self-Attention can in fact be viewed through the lens of test-time training, with a kernel-smoother "learning" from context. Simply replacing that with more powerful models than a kernel-smoother result in very capable and scalable models! https://arxiv.org/abs/2407.04620 https://arxiv.org/abs/2407.04620