5 ms·
According to the paper, the "registers" are additional learnable tokens that are appended to the input sequence of a Vision Transformer model during training.
by ttul 2y ago
According to the paper, the "registers" are additional learnable tokens that are appended to the input sequence of a Vision Transformer model during training.
They are added after the patch embedding layer, with a learnable value, similar to the [CLS] token and then at the end of the Vision Transformer, the register tokens are discarded, and only the [CLS] token and patch tokens are used as image representations.
The register tokens provide a place for the model to store, process and retrieve global information during the forward pass, without repurposing patch tokens for this role.
Adding register tokens removes the artifacts and high-norm "outlier" tokens that otherwise appear in the feature maps of trained Vision Transformer models.
Using register tokens leads to smoother feature maps, improved performance on dense prediction tasks, and enables better unsupervised object discovery compared to the same models trained without the additional register tokens.
This is a neat result. For just a 2% increase in inference cost, you can significantly improve ViT model performance. Close to a free lunch.
- kadushka 2y agoInteresting. One other potential benefit is an easier quantization of the activations.
- macleginn 2y agoThere was an attempt to add several CLS tokens to BERT, with less spectacular results: https://arxiv.org/pdf/2210.05043 https://arxiv.org/pdf/2210.05043
- swyx 2y agoare there lessons here for regular (non vision) transformers? sounds close to attention sinks/pause tokens?
- johntb86 2y agoFor these tokens you first need to unembed the result of the final layer, the re-embed the resulting token on the next pass. Has anyone investigated passing the raw output of one pass to the input of the next?
- hasmanean 2y agoSo is that what all the visual cues are in real life, things like fashion accessories, uniforms etc.?
- larodi 2y agoThis all token business is very shady, and the whole probability theory. You add token here and there and magic happens. Discreet math people can not take this lightly. Stochastic regexes is one thing, but this on a completely different level of mathematical debauchery. Absolutely amazing this works.
- programjames 2y agoVision transformers are essentially just JPEG but with learned features rather than the Fourier transform.
- larodi 2y agoIndeed. Kernels mashing features. Knowing jpeg helped the understanding of embedding a lot. It’s why I tell friends - talking to GPT is like talking to .ZIP files…
- Zacharias030 2y agoInteresting! Can you elaborate?
- programjames 2y agoThe JPEG algorithm is: 1. Divide up the image into 8x8 patches 2. Take the DCT (a variant of the Fourier transform) of each patch to extract key features 3. Quantize the outputs 4. Use arithmetic encoding to compress The ViT algorithm is: 1. Divide up the image into 16x16 patches 2. Use query/key/value attention matrices to extract key features 3. Minimize cross-entropy loss between predicted and actual next tokens. (This is equivalent to trying to minimize encoding length.) ViT don't have quantization baked into the algorithm, but NNs are being moved towards quantization in general. Another user correctly pointed out that vision transformers are not necessarily autoregressive (i.e. they may use future patches to calculate values for previous patches), while arithmetic encoding usually is (so JPEG is), so the algorithms have a few differences but nothing major. ----- I think it's pretty interesting how closely related generation and compression are. ClosedAI's Sora[^1] model uses a denoising vision transformer for their state-of-the-art video generator, while JPEG has been leading image compression for the past several decades. [^1]: https://openai.com/index/sora/?video=big-sur https://openai.com/index/sora/?video=big-sur