6 ms·

Vision Transformers Need Registers

171 points by cscurmudgeon 2y ago

superkuh 2y ago
"Attention sinks" for vision models?
deleted 2y ago
[deleted]
ttul 2y ago
According to the paper, the "registers" are additional learnable tokens that are appended to the input sequence of a Vision Transformer model during training. They are added after the patch embedding layer, with a learnable value, similar to the [CLS] token and then at the end of the Vision Transformer, the register tokens are discarded, and only the [CLS] token and patch tokens are used as image representations. The register tokens provide a place for the model to store, process and retrieve global information during the forward pass, without repurposing patch tokens for this role. Adding register tokens removes the artifacts and high-norm "outlier" tokens that otherwise appear in the feature maps of trained Vision Transformer models. Using register tokens leads to smoother feature maps, improved performance on dense prediction tasks, and enables better unsupervised object discovery compared to the same models trained without the additional register tokens. This is a neat result. For just a 2% increase in inference cost, you can significantly improve ViT model performance. Close to a free lunch.
- kadushka 2y ago
  Interesting. One other potential benefit is an easier quantization of the activations.
- macleginn 2y ago
  There was an attempt to add several CLS tokens to BERT, with less spectacular results: https://arxiv.org/pdf/2210.05043 https://arxiv.org/pdf/2210.05043
- swyx 2y ago
  are there lessons here for regular (non vision) transformers? sounds close to attention sinks/pause tokens?
- johntb86 2y ago
  For these tokens you first need to unembed the result of the final layer, the re-embed the resulting token on the next pass. Has anyone investigated passing the raw output of one pass to the input of the next?
- hasmanean 2y ago
  So is that what all the visual cues are in real life, things like fashion accessories, uniforms etc.?
- larodi 2y ago
  This all token business is very shady, and the whole probability theory. You add token here and there and magic happens. Discreet math people can not take this lightly. Stochastic regexes is one thing, but this on a completely different level of mathematical debauchery. Absolutely amazing this works.
  - programjames 2y ago
    Vision transformers are essentially just JPEG but with learned features rather than the Fourier transform.
    - larodi 2y ago
      Indeed. Kernels mashing features. Knowing jpeg helped the understanding of embedding a lot. It’s why I tell friends - talking to GPT is like talking to .ZIP files…
    - Zacharias030 2y ago
      Interesting! Can you elaborate?
      - programjames 2y ago
        The JPEG algorithm is: 1. Divide up the image into 8x8 patches 2. Take the DCT (a variant of the Fourier transform) of each patch to extract key features 3. Quantize the outputs 4. Use arithmetic encoding to compress The ViT algorithm is: 1. Divide up the image into 16x16 patches 2. Use query/key/value attention matrices to extract key features 3. Minimize cross-entropy loss between predicted and actual next tokens. (This is equivalent to trying to minimize encoding length.) ViT don't have quantization baked into the algorithm, but NNs are being moved towards quantization in general. Another user correctly pointed out that vision transformers are not necessarily autoregressive (i.e. they may use future patches to calculate values for previous patches), while arithmetic encoding usually is (so JPEG is), so the algorithms have a few differences but nothing major. ----- I think it's pretty interesting how closely related generation and compression are. ClosedAI's Sora[^1] model uses a denoising vision transformer for their state-of-the-art video generator, while JPEG has been leading image compression for the past several decades. [^1]: https://openai.com/index/sora/?video=big-sur https://openai.com/index/sora/?video=big-sur
richdougherty 2y ago
Related? "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758 > Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens. > In this work, we demonstrate that transformers trained on the next-token prediction objective can achieve improved performance on certain tasks when given filler tokens, achieving perfect accuracy whereas the no-filler, immediate-answer setting achieves only low accuracy. -- I wonder if we could get benefits from adding special computation/register tokens to text LLMs? More discussion: - https://news.ycombinator.com/item?id=40182695 https://news.ycombinator.com/item?id=40182695 - https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transformers_can_use_meaningless_filler_tokens_eg/ https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transfo...
joaquincabezas 2y ago
I was at ICLR and this was one of the best of this year, it was also evident during the poster session. Congrats to the authors!!
barbolo 2y ago
I’ve been using DinoV2 for some months now. I’ve tried the models with 4 register tokens along with CLS + patch tokens. I’ve several embeddings (tokens) from previous model (no registers) which are part of my solution, so I didn’t adopt the newer “register” models because the CLS tokens are not aligned between 0 registers and 4 registers models. It would be nice if the CLS and patch tokens were somehow aligned between those models.
ashvardanian 2y ago
We tested dozens (maybe >100) of papers/ideas over the last years in vision and multimodal perception and this is one of the rare cases, where everything worked well! Neat idea and paper! This model, for example, uses 4 register tokens, and combines them with Matryoshka-style losses for training, resulting in super-compact 64-dimensional embeddings, in case anyone is looking for CLIP alternatives: https://huggingface.co/unum-cloud/uform3-image-text-english-large https://huggingface.co/unum-cloud/uform3-image-text-english-...

171 points by cscurmudgeon 2y ago

superkuh 2y ago

"Attention sinks" for vision models?

deleted 2y ago

[deleted]

ttul 2y ago

According to the paper, the "registers" are additional learnable tokens that are appended to the input sequence of a Vision Transformer model during training. They are added after the patch embedding layer, with a learnable value, similar to the [CLS] token and then at the end of the Vision Transformer, the register tokens are discarded, and only the [CLS] token and patch tokens are used as image representations. The register tokens provide a place for the model to store, process and retrieve global information during the forward pass, without repurposing patch tokens for this role. Adding register tokens removes the artifacts and high-norm "outlier" tokens that otherwise appear in the feature maps of trained Vision Transformer models. Using register tokens leads to smoother feature maps, improved performance on dense prediction tasks, and enables better unsupervised object discovery compared to the same models trained without the additional register tokens. This is a neat result. For just a 2% increase in inference cost, you can significantly improve ViT model performance. Close to a free lunch.

kadushka 2y ago
Interesting. One other potential benefit is an easier quantization of the activations.
macleginn 2y ago
There was an attempt to add several CLS tokens to BERT, with less spectacular results: https://arxiv.org/pdf/2210.05043 https://arxiv.org/pdf/2210.05043
swyx 2y ago
are there lessons here for regular (non vision) transformers? sounds close to attention sinks/pause tokens?
johntb86 2y ago
For these tokens you first need to unembed the result of the final layer, the re-embed the resulting token on the next pass. Has anyone investigated passing the raw output of one pass to the input of the next?
hasmanean 2y ago
So is that what all the visual cues are in real life, things like fashion accessories, uniforms etc.?
larodi 2y ago
This all token business is very shady, and the whole probability theory. You add token here and there and magic happens. Discreet math people can not take this lightly. Stochastic regexes is one thing, but this on a completely different level of mathematical debauchery. Absolutely amazing this works.
- programjames 2y ago
  Vision transformers are essentially just JPEG but with learned features rather than the Fourier transform.
  - larodi 2y ago
    Indeed. Kernels mashing features. Knowing jpeg helped the understanding of embedding a lot. It’s why I tell friends - talking to GPT is like talking to .ZIP files…
  - Zacharias030 2y ago
    Interesting! Can you elaborate?
    - programjames 2y ago
      The JPEG algorithm is: 1. Divide up the image into 8x8 patches 2. Take the DCT (a variant of the Fourier transform) of each patch to extract key features 3. Quantize the outputs 4. Use arithmetic encoding to compress The ViT algorithm is: 1. Divide up the image into 16x16 patches 2. Use query/key/value attention matrices to extract key features 3. Minimize cross-entropy loss between predicted and actual next tokens. (This is equivalent to trying to minimize encoding length.) ViT don't have quantization baked into the algorithm, but NNs are being moved towards quantization in general. Another user correctly pointed out that vision transformers are not necessarily autoregressive (i.e. they may use future patches to calculate values for previous patches), while arithmetic encoding usually is (so JPEG is), so the algorithms have a few differences but nothing major. ----- I think it's pretty interesting how closely related generation and compression are. ClosedAI's Sora[^1] model uses a denoising vision transformer for their state-of-the-art video generator, while JPEG has been leading image compression for the past several decades. [^1]: https://openai.com/index/sora/?video=big-sur https://openai.com/index/sora/?video=big-sur

richdougherty 2y ago

Related? "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758 > Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens. > In this work, we demonstrate that transformers trained on the next-token prediction objective can achieve improved performance on certain tasks when given filler tokens, achieving perfect accuracy whereas the no-filler, immediate-answer setting achieves only low accuracy. -- I wonder if we could get benefits from adding special computation/register tokens to text LLMs? More discussion: - https://news.ycombinator.com/item?id=40182695 https://news.ycombinator.com/item?id=40182695 - https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transformers_can_use_meaningless_filler_tokens_eg/ https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transfo...

joaquincabezas 2y ago

I was at ICLR and this was one of the best of this year, it was also evident during the poster session. Congrats to the authors!!

barbolo 2y ago

I’ve been using DinoV2 for some months now. I’ve tried the models with 4 register tokens along with CLS + patch tokens. I’ve several embeddings (tokens) from previous model (no registers) which are part of my solution, so I didn’t adopt the newer “register” models because the CLS tokens are not aligned between 0 registers and 4 registers models. It would be nice if the CLS and patch tokens were somehow aligned between those models.

ashvardanian 2y ago

We tested dozens (maybe >100) of papers/ideas over the last years in vision and multimodal perception and this is one of the rare cases, where everything worked well! Neat idea and paper! This model, for example, uses 4 register tokens, and combines them with Matryoshka-style losses for training, resulting in super-compact 64-dimensional embeddings, in case anyone is looking for CLIP alternatives: https://huggingface.co/unum-cloud/uform3-image-text-english-large https://huggingface.co/unum-cloud/uform3-image-text-english-...