3 ms·
This seems somewhat related to https://arxiv.org/abs/2309.16588 https://arxiv.org/abs/2309.16588. Some Researchers found that Vision Transformers would repurpos
by famouswaffles 3y ago
This seems somewhat related to https://arxiv.org/abs/2309.16588 https://arxiv.org/abs/2309.16588.
Some Researchers found that Vision Transformers would repurpose low-informative background patches of images for internal computations. They gave them an explicit token for this to better results!
- HammadB 3y agoYeah, was thinking the same thing. Something deeper to be explored here esp due to the consistency across domains. I wonder how far we can push "externalizing" higher level information the model wants to store while in the forward pass.
- mkaic 3y agoYou beat me to it, I was just about to post this here haha. It's fascinating to me that these models can take advantage of "scratch paper" this way!
- deleted 3y ago[deleted]
- bradfox2 3y agoThis likely has a tangent to quantization research from Dettmers et al that corresponds to outlier weight values in models >6B params. Might allude to why the outliers emerge.