5 ms·

A visual exploration of vector embeddings

185 points by pamelafox 1y ago

isjustintime 1y ago
I love the visual approaches used to explain these concepts. Words and math hurt my brain, but when accompanied by charts and diagrams, my brain hurts much less.
- cratermoon 1y ago
  If you like this you'll love Grant Sanderson's series on linear algebra and LLMs. https://www.youtube.com/watch?v=wjZofJX0v4M https://www.youtube.com/watch?v=wjZofJX0v4M
minimaxir 1y ago
Since this was oriented toward a Python audience, it may have also been useful to demonstrate on the poster how in Python you can create the embeddings (e.g. using requests/OpenAI client and hitting OpenAI's embeddings API) and calculate the similarities (e.g. using numpy) since most won't read the linked notebooks. Mostly as a good excuse to showoff Python's rare @ operator for dot products in numpy. As a tangent, what root data source are you using to calculate the movie embeddings?
- pamelafox 1y ago
  I thought I'd make this blog post be language-agnostic, but agreed that a Python-specific version would be helpful. Here's where I calculate cosine without numpy: https://github.com/pamelafox/vector-embeddings-demos/blob/main/similarity.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... And in the distance notebook, I calculate with numpy: https://github.com/pamelafox/vector-embeddings-demos/blob/main/distance_metrics.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... I didn't use the @ operator! TIL. I forget where I originally got the Disney movie titles, but it is notably just the titles. A better ranking would be based off a movie synopsis as well. Here's where I calculated their embeddings using OpenAI: https://github.com/pamelafox/vector-embeddings-demos/blob/main/prep_openai.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... Maybe I can submit a poster to Pytorch that would include the Python code as well.
godelski 1y ago
The more I've studied this stuff the less useful I actually think the vitalizations are. Pamela uses the classic approach and I'm not trying to call her wrong but I think our intuitions really fail us outside 2D and 3D. Once you move up in dimensionality things get really messy really fast. There's a contraction in variance and the meaning of distance becomes much more fuzzy. You can't differentiate your nearest neighbor from your furthest. Angles get much harder too. Everything is orthogonal, in most directions too! I'm not all that surprised "god" and "dog". I EXPECT them to be. After all, they are the reverse of one another. The question rather is about "similar in which direction?" There's no need to believe you've measured along a direction that is human meaningful. So doesn't have to be semantics. Doesn't have to be permutations either. Just like you can rotate your xy axis and travel in both directions. So these things can really trick us. At best, be very careful to not become overly reliant upon them
- pamelafox 1y ago
  I actually explicitly left the PCA graphs out of the blog post version, as I think they lose so much information as to be deceiving. That's what I told folks in person at the poster session as well. I think the other graphs I included aren't deceiving, they're just not quite as fun as an attempt to visualize the similarity space.
  - godelski 1y ago
    Yeah PCA gets tough. It isn't great for non-linear relationships and I mean that's the whole reason we use activation functions haha. And don't get me started on how people refer to t-SNE as dimensionality reduction instead of visualization... I don't think the other graphs are necessarily deceiving but I think they don't capture as much information as we often imply and I think this ends up leading people to make wrong assumptions about what is happening in the data. Embeddings and immersions get really fucking weird at high dimensions. I mean it gets weird at like 4D and insane by 10D. The spaces we're talking about are incomprehensible. Every piece of geometric intuition you have should be thrown out the window. It won't help you, it harms you. If you start digging into the high dimensional statistics and metric theory for high dimensions you'll quickly see what I'm talking about. Like the craziness of Lp distances and contraction of variance. Like you have to really dig into why we prefer L1 over L2 and why even fractional ps are of interest. We run into all kinds of problems with i.i.d. assumptions and all that. It is wild how many assumptions are being made that we generally don't even think about. They seem obvious and natural to use, but they don't work very well when D>3. I do think the visualizations become useful again once you start getting used to this again but that's more like in the way that you are interpreting it with far less generalization in meaning. I'm not trying to dunk on your post. I think it is fine. But I think our ML community needs to be having more conversations about these limits. We're really running into issues with them.
antirez 1y ago
Here I tried to use 2D visualization, and it may be more immediate: https://antirez.com/news/150 https://antirez.com/news/150
- janalsncm 1y ago
  Where can I find the original dataset for doing stylometry?
- pamelafox 1y ago
  Thanks for sharing that too! I do have a notebook that does a PCA reduction to plot a similarity space: https://github.com/pamelafox/vector-embeddings-demos/blob/main/similarity.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... But as I noted in another comment, I think it loses so much information as to be deceiving. I also find this 3d visualization to be fun: https://projector.tensorflow.org/ https://projector.tensorflow.org/ But once again, huge loss in information. I personally learn more by actually seeing the relative similarity ranking and scores within a dataset, versus trying to visualize all of the nodes on the same graph with a massive dimension simplification. That 3d visualization is what originally intrigued me though, to see how else I could visualize. :)
pamelafox 1y ago
I forgot that I also put together this little website, if you want to compare vectors for word2vec versus text-embedding-ada-002: https://pamelafox.github.io/vectors-comparison/ https://pamelafox.github.io/vectors-comparison/ (I never added text-embedding-3 to it)
podgietaru 1y ago
I did the traditional blog post about this, and used it to create an RSS Aggregator website using AWS Bedrock. https://aws.amazon.com/blogs/machine-learning/use-language-embeddings-for-zero-shot-classification-and-semantic-search-with-amazon-bedrock/ https://aws.amazon.com/blogs/machine-learning/use-language-e... The website is unfortunately down now, due to the fact I no longer work at Amazon, but the code is still readily available if you want to run it yourself. https://github.com/aws-samples/rss-aggregator-using-cohere-embeddings-bedrock https://github.com/aws-samples/rss-aggregator-using-cohere-e...
kaycebasques 1y ago
The post title reminds me of something that I researched a little a couple months back. Practically all embeddings are implemented as vectors, right? Definitionally, an embedding doesn't have to be a vector. But in practice there's not really any such thing as a non-vector embedding, is there? One thing I learned recently is that, if your embedding model supports task types (clustering, STS, retrieval, etc.), then that can have a non-trivial impact on the generated embedding for a given text: https://technicalwriting.dev/ml/embeddings/tasks/index.html https://technicalwriting.dev/ml/embeddings/tasks/index.html Parquet and Polars sound very promising for reducing embeddings storage requirements. Still haven't tinkered with them: https://minimaxir.com/2025/02/embeddings-parquet/ https://minimaxir.com/2025/02/embeddings-parquet/ And this post gave me a lot more awareness to be more careful about how exactly I'm comparing embeddings. OP's post seems to do a good job explaining common techniques, too. https://p.migdal.pl/blog/2025/01/dont-use-cosine-similarity/ https://p.migdal.pl/blog/2025/01/dont-use-cosine-similarity/
- seanhunter 1y ago
  There are lots of non-vector embeddings. “Embedding” just means one mathematical structure stuck inside another in such a way as to preserve its algebraic properties. So for example the real number line is embedded into the Cartesian plane along both axes. That’s a non-vector embedding. I think that field extensions (like the rationals in the reals) also count as an embedding although I may be wrong about that. In ML, embeddings are used as a way to work with text using the tools of linear algebra. So of course people use (and want to use) vector embeddings, because that’s what gives you what you need in order to “do linear algebra on text”.
  - kaycebasques 1y ago
    Thank you for the details. I was thinking about ML embeddings in particular, but your answer does help me understand the core mathematical nature of an embedding at a deeper level. E.g. the word "embedding" makes more sense now in the context of "one mathematical structure stuck inside another."
galaxyLogic 1y ago
> The text-embedding-ada-002 model accepts up to 8192 "tokens", where a "token" is the unit of measurement for the model (typically corresponding to a word or syllable), So the "input" is up to 8192 "units of measurement". What would that mean in practice? How are the units of measurement produced? Can they be anything?
- striking 1y ago
  They're produced from a tokenizer. Technically they could be anything (even raw bytes) but you get better results by choosing a better tokenization strategy.
- mdp2021 1y ago
  The "token" is a basic unit of meaning (for example, 'unit', 'of', 'mean', '-ing') that can have its coordinates in an embedding space, where they match scalars in the dimensions representing concepts which the machine reconstructs. Allowing more tokens for an input allows (I understand, in this context) to output the embedding coordinates not just of a unit of meaning, not just of a word, but of expressions or sentences or paragraphs ('go'; 'going'; 'going away'; 'they are going away', 'Mr. Fitzgerald and his wife are going away'...).
jamesk_au 1y ago
Does anyone have any insight (or informed guesses) that might explain the strange downward "spike" that was consistently observed at dimension 196 in OpenAI's text-embedding-ada-002 model?
persedes 1y ago
Would be interesting to see how e.g sentence transformer models compare to this. My takeaway with the e.g. openai embedding models was that they were better suited for larger chunks of texts, so getting god + dog with a higher similarity might be indicative that it's not a good model for such small text? emb = SentenceTransformer("all-MiniLM-L6-v2") embeddings = emb.encode(["dog", "god"]) cosine_similarity(embeddings) Out[16]: array([[1. , 0.41313702], [0.41313702, 1.0000004 ]], dtype=float32)
tanelpoder 1y ago
I took a completely different path to visualizing embedding vectors’ physical layout [1] - mainly to explain how the data structures, data volumes and comparison would radically differ, compared to your regular btree index searches. I made sure to mention that you can’t make any conclusions based on just human eyeballing of these vector heatmaps, but the database people I’ve demoed this to, seem to have reached some a-ha moments about understanding how radically different a vector search is compared to the usual database index lookup work: [1] https://tanelpoder.com/posts/comparing-vectors-of-the-same-rotated-image/ https://tanelpoder.com/posts/comparing-vectors-of-the-same-r...
jbellis 1y ago
[dead]
ConteMascetti71 1y ago
I have done some experimentation in the field of vector arithmetic. The results, in the field of images, are very interesting. https://github.com/vagrillo/CLIPSemanticImageArythmetics https://github.com/vagrillo/CLIPSemanticImageArythmetics

185 points by pamelafox 1y ago

I love the visual approaches used to explain these concepts. Words and math hurt my brain, but when accompanied by charts and diagrams, my brain hurts much less.

cratermoon 1y ago
If you like this you'll love Grant Sanderson's series on linear algebra and LLMs. https://www.youtube.com/watch?v=wjZofJX0v4M https://www.youtube.com/watch?v=wjZofJX0v4M

minimaxir 1y ago

Since this was oriented toward a Python audience, it may have also been useful to demonstrate on the poster how in Python you can create the embeddings (e.g. using requests/OpenAI client and hitting OpenAI's embeddings API) and calculate the similarities (e.g. using numpy) since most won't read the linked notebooks. Mostly as a good excuse to showoff Python's rare @ operator for dot products in numpy. As a tangent, what root data source are you using to calculate the movie embeddings?

pamelafox 1y ago
I thought I'd make this blog post be language-agnostic, but agreed that a Python-specific version would be helpful. Here's where I calculate cosine without numpy: https://github.com/pamelafox/vector-embeddings-demos/blob/main/similarity.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... And in the distance notebook, I calculate with numpy: https://github.com/pamelafox/vector-embeddings-demos/blob/main/distance_metrics.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... I didn't use the @ operator! TIL. I forget where I originally got the Disney movie titles, but it is notably just the titles. A better ranking would be based off a movie synopsis as well. Here's where I calculated their embeddings using OpenAI: https://github.com/pamelafox/vector-embeddings-demos/blob/main/prep_openai.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... Maybe I can submit a poster to Pytorch that would include the Python code as well.

godelski 1y ago

The more I've studied this stuff the less useful I actually think the vitalizations are. Pamela uses the classic approach and I'm not trying to call her wrong but I think our intuitions really fail us outside 2D and 3D. Once you move up in dimensionality things get really messy really fast. There's a contraction in variance and the meaning of distance becomes much more fuzzy. You can't differentiate your nearest neighbor from your furthest. Angles get much harder too. Everything is orthogonal, in most directions too! I'm not all that surprised "god" and "dog". I EXPECT them to be. After all, they are the reverse of one another. The question rather is about "similar in which direction?" There's no need to believe you've measured along a direction that is human meaningful. So doesn't have to be semantics. Doesn't have to be permutations either. Just like you can rotate your xy axis and travel in both directions. So these things can really trick us. At best, be very careful to not become overly reliant upon them

pamelafox 1y ago
I actually explicitly left the PCA graphs out of the blog post version, as I think they lose so much information as to be deceiving. That's what I told folks in person at the poster session as well. I think the other graphs I included aren't deceiving, they're just not quite as fun as an attempt to visualize the similarity space.
- godelski 1y ago
  Yeah PCA gets tough. It isn't great for non-linear relationships and I mean that's the whole reason we use activation functions haha. And don't get me started on how people refer to t-SNE as dimensionality reduction instead of visualization... I don't think the other graphs are necessarily deceiving but I think they don't capture as much information as we often imply and I think this ends up leading people to make wrong assumptions about what is happening in the data. Embeddings and immersions get really fucking weird at high dimensions. I mean it gets weird at like 4D and insane by 10D. The spaces we're talking about are incomprehensible. Every piece of geometric intuition you have should be thrown out the window. It won't help you, it harms you. If you start digging into the high dimensional statistics and metric theory for high dimensions you'll quickly see what I'm talking about. Like the craziness of Lp distances and contraction of variance. Like you have to really dig into why we prefer L1 over L2 and why even fractional ps are of interest. We run into all kinds of problems with i.i.d. assumptions and all that. It is wild how many assumptions are being made that we generally don't even think about. They seem obvious and natural to use, but they don't work very well when D>3. I do think the visualizations become useful again once you start getting used to this again but that's more like in the way that you are interpreting it with far less generalization in meaning. I'm not trying to dunk on your post. I think it is fine. But I think our ML community needs to be having more conversations about these limits. We're really running into issues with them.

antirez 1y ago

Here I tried to use 2D visualization, and it may be more immediate: https://antirez.com/news/150 https://antirez.com/news/150

janalsncm 1y ago
Where can I find the original dataset for doing stylometry?
pamelafox 1y ago
Thanks for sharing that too! I do have a notebook that does a PCA reduction to plot a similarity space: https://github.com/pamelafox/vector-embeddings-demos/blob/main/similarity.ipynb https://github.com/pamelafox/vector-embeddings-demos/blob/ma... But as I noted in another comment, I think it loses so much information as to be deceiving. I also find this 3d visualization to be fun: https://projector.tensorflow.org/ https://projector.tensorflow.org/ But once again, huge loss in information. I personally learn more by actually seeing the relative similarity ranking and scores within a dataset, versus trying to visualize all of the nodes on the same graph with a massive dimension simplification. That 3d visualization is what originally intrigued me though, to see how else I could visualize. :)

pamelafox 1y ago

I forgot that I also put together this little website, if you want to compare vectors for word2vec versus text-embedding-ada-002: https://pamelafox.github.io/vectors-comparison/ https://pamelafox.github.io/vectors-comparison/ (I never added text-embedding-3 to it)

podgietaru 1y ago

I did the traditional blog post about this, and used it to create an RSS Aggregator website using AWS Bedrock. https://aws.amazon.com/blogs/machine-learning/use-language-embeddings-for-zero-shot-classification-and-semantic-search-with-amazon-bedrock/ https://aws.amazon.com/blogs/machine-learning/use-language-e... The website is unfortunately down now, due to the fact I no longer work at Amazon, but the code is still readily available if you want to run it yourself. https://github.com/aws-samples/rss-aggregator-using-cohere-embeddings-bedrock https://github.com/aws-samples/rss-aggregator-using-cohere-e...

kaycebasques 1y ago

The post title reminds me of something that I researched a little a couple months back. Practically all embeddings are implemented as vectors, right? Definitionally, an embedding doesn't have to be a vector. But in practice there's not really any such thing as a non-vector embedding, is there? One thing I learned recently is that, if your embedding model supports task types (clustering, STS, retrieval, etc.), then that can have a non-trivial impact on the generated embedding for a given text: https://technicalwriting.dev/ml/embeddings/tasks/index.html https://technicalwriting.dev/ml/embeddings/tasks/index.html Parquet and Polars sound very promising for reducing embeddings storage requirements. Still haven't tinkered with them: https://minimaxir.com/2025/02/embeddings-parquet/ https://minimaxir.com/2025/02/embeddings-parquet/ And this post gave me a lot more awareness to be more careful about how exactly I'm comparing embeddings. OP's post seems to do a good job explaining common techniques, too. https://p.migdal.pl/blog/2025/01/dont-use-cosine-similarity/ https://p.migdal.pl/blog/2025/01/dont-use-cosine-similarity/

seanhunter 1y ago
There are lots of non-vector embeddings. “Embedding” just means one mathematical structure stuck inside another in such a way as to preserve its algebraic properties. So for example the real number line is embedded into the Cartesian plane along both axes. That’s a non-vector embedding. I think that field extensions (like the rationals in the reals) also count as an embedding although I may be wrong about that. In ML, embeddings are used as a way to work with text using the tools of linear algebra. So of course people use (and want to use) vector embeddings, because that’s what gives you what you need in order to “do linear algebra on text”.
- kaycebasques 1y ago
  Thank you for the details. I was thinking about ML embeddings in particular, but your answer does help me understand the core mathematical nature of an embedding at a deeper level. E.g. the word "embedding" makes more sense now in the context of "one mathematical structure stuck inside another."

galaxyLogic 1y ago

> The text-embedding-ada-002 model accepts up to 8192 "tokens", where a "token" is the unit of measurement for the model (typically corresponding to a word or syllable), So the "input" is up to 8192 "units of measurement". What would that mean in practice? How are the units of measurement produced? Can they be anything?

striking 1y ago
They're produced from a tokenizer. Technically they could be anything (even raw bytes) but you get better results by choosing a better tokenization strategy.
mdp2021 1y ago
The "token" is a basic unit of meaning (for example, 'unit', 'of', 'mean', '-ing') that can have its coordinates in an embedding space, where they match scalars in the dimensions representing concepts which the machine reconstructs. Allowing more tokens for an input allows (I understand, in this context) to output the embedding coordinates not just of a unit of meaning, not just of a word, but of expressions or sentences or paragraphs ('go'; 'going'; 'going away'; 'they are going away', 'Mr. Fitzgerald and his wife are going away'...).

jamesk_au 1y ago

Does anyone have any insight (or informed guesses) that might explain the strange downward "spike" that was consistently observed at dimension 196 in OpenAI's text-embedding-ada-002 model?

persedes 1y ago

Would be interesting to see how e.g sentence transformer models compare to this. My takeaway with the e.g. openai embedding models was that they were better suited for larger chunks of texts, so getting god + dog with a higher similarity might be indicative that it's not a good model for such small text? emb = SentenceTransformer("all-MiniLM-L6-v2") embeddings = emb.encode(["dog", "god"]) cosine_similarity(embeddings) Out[16]: array([[1. , 0.41313702], [0.41313702, 1.0000004 ]], dtype=float32)

tanelpoder 1y ago

I took a completely different path to visualizing embedding vectors’ physical layout [1] - mainly to explain how the data structures, data volumes and comparison would radically differ, compared to your regular btree index searches. I made sure to mention that you can’t make any conclusions based on just human eyeballing of these vector heatmaps, but the database people I’ve demoed this to, seem to have reached some a-ha moments about understanding how radically different a vector search is compared to the usual database index lookup work: [1] https://tanelpoder.com/posts/comparing-vectors-of-the-same-rotated-image/ https://tanelpoder.com/posts/comparing-vectors-of-the-same-r...

jbellis 1y ago

[dead]

ConteMascetti71 1y ago

I have done some experimentation in the field of vector arithmetic. The results, in the field of images, are very interesting. https://github.com/vagrillo/CLIPSemanticImageArythmetics https://github.com/vagrillo/CLIPSemanticImageArythmetics