3 ms·
> I don’t know. After the model has been created (trained), I’m pretty sure that generating embeddings is much less computationally intensive than generating te
by minimaxir 1y ago
> I don’t know. After the model has been created (trained), I’m pretty sure that generating embeddings is much less computationally intensive than generating text.
An embedding is generated after a single pass through the model, so functionally it's the equivalent of generating a single token from an text generation model.
- energy123 1y agoI might be wrong but aren't embedding models usually bidirectional and not causal, so the attention mechanism itself is more expensive.
- breadislove 1y agoyes exactly
- minimaxir 1y agoIt depends on the architecture (you very well can convert a decoder-only causal model to an embeddings model, e.g. Qwen/Mistral), but it is true the traditional embeddings models such as a BERT-based one are bidirectional, although unclear how much more compute that inherently requires. Compare to ModernBERT, which uses more modern techniques and is still bidirectional, but it is very very speedy. https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert
- kaycebasques 1y agoWill update the post to capture that interesting tidbit, thanks And also, thanks for this post! https://minimaxir.com/2025/02/embeddings-parquet/ https://minimaxir.com/2025/02/embeddings-parquet/