3 ms·
I might be wrong but aren't embedding models usually bidirectional and not causal, so the attention mechanism itself is more expensive.
by energy123 1y ago
I might be wrong but aren't embedding models usually bidirectional and not causal, so the attention mechanism itself is more expensive.
- breadislove 1y agoyes exactly
- minimaxir 1y agoIt depends on the architecture (you very well can convert a decoder-only causal model to an embeddings model, e.g. Qwen/Mistral), but it is true the traditional embeddings models such as a BERT-based one are bidirectional, although unclear how much more compute that inherently requires. Compare to ModernBERT, which uses more modern techniques and is still bidirectional, but it is very very speedy. https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert