8 ms·
LLM Embeddings Explained: A Visual and Intuitive Guide
- carschno 1y agoNice explanations! A (more advanced) aspect which I find missing would be the difference between encoder-decoder transformer models (BERT) and "decoder-only", generative models, with respect to the embeddings.
- dust42 1y agoMinor correction, BERT is an encoder (not encoder-decoder), ChatGPT is a decoder. Encoders like BERT produce better results for embeddings because they look at the whole sentence, while GPTs look from left to right: Imagine you're trying to understand the meaning of a word in a sentence, and you can read the entire sentence before deciding what that word means. For example, in "The bank was steep and muddy," you can see "steep and muddy" at the end, which tells you "bank" means the side of a river (aka riverbank), not a financial institution. BERT works this way - it looks at all the words around a target word (both before and after) to understand its meaning. Now imagine you have to understand each word as you read from left to right, but you're not allowed to peek ahead. So when you encounter "The bank was..." you have to decide what "bank" means based only on "The" - you can't see the helpful clues that come later. GPT models work this way because they're designed to generate text one word at a time, predicting what comes next based only on what they've seen so far. Here is a link also from huggingface, about modernBERT which has more info: https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert Also worth a look: neoBERT https://huggingface.co/papers/2502.19587 https://huggingface.co/papers/2502.19587
- jasonjayr 1y agoAs an extreme example that can (intentionally) confuse even human readers, see https://en.wikipedia.org/wiki/Garden-path_sentence https://en.wikipedia.org/wiki/Garden-path_sentence
- xxpor 1y agoComplete LLM internals noob here: Wouldn't this make GPTs awful at languages like German with separable word prefixes? E.g. Er macht das Fenster. vs Er macht das Fenster auf. (He makes the window. vs He opens the window.)
- Ey7NFZ3P0nzAe 1y agoOr exceptionally good at german because they have to keep better track of what is meant and anticipate more? No I don't think it makes any noticeable difference :)
- xxpor 1y agoI'm probably way too English brained :D
- ubutler 1y agoFurther to @dust42, BERT is an encoder, GPT is a decoder, and T5 is an encoder-decoder. Encoder-decoders are not in vogue. Encoders are favored for classification, extraction (eg, NER and extractive QA) and information retrieval. Decoders are favored for text generation, summarization and translation. Recent research (see, eg, the Ettin paper: https://arxiv.org/html/2507.11412v1 https://arxiv.org/html/2507.11412v1 ) seems to confirm the previous understanding that encoders are indeed better for “encoder task” and vice-versa. Fundamentally, both are transformers and so an encoder could be turned into a decoder or a decoder could be turned into an encoder. The design difference comes down to bidirectional (ie, all tokens can attend to all other tokens) versus autoregressive attention (ie, the current token can only attend to the previous tokens).
- microtonal 1y agoUntil we got highly optimized decoder implementations, decoders for prefill were often even implemented by using the same implementation as an encoder, but logit-masking inputs using a causal mask before the attention softmax so that tokens could not attend to future tokens.
- namibj 1y agoYou can use an encoder style architecture with decoder style output heads up top for denoising diffusion mode mask/blank filling. They seem to be somewhat more expensive on short sequences than GPT style decoder-only models when you batch them, as you need fewer passes over the content and until sequence length blows up your KV cache throughout cost, fewer passes are cheaper. But for situations that don't get request batching or where the context length is so heavy that you'd prefer to get to exploit memory locality on the attention computation, you'd benefit from diffusion mode decoding. A nice side effect of the diffusion mode is that it's natural reliance on the bidirectional attention from the encoder layers provides much more flexible (and, critically, context-aware) understanding so as mentioned, later words can easily modulate earlier words like with "bank [of the river]"/"bank [in the park]"/"bank [got robbed]" or the classic of these days: telling an agent it did wrong and expecting it to in-context learn from the mistake (in practice decoder-only models basically merely get polluted from that, so you have to re-wind the conversation, because the later correction has literally no way of backwards-affecting the problematic tokens). That said, the recent surge in training "reasoning" models to utilize thinking tokens that often get cut out of further conversation context, and all via a reinforcement learning process that's not merely RLHF/preference-conditioning, is actually quite related: discrete denoising diffusion models can be trained as a RL scheme during pre training where the training step is provided the outcome goal and a masked version as the input query, and then trained to manage the work done in the individual steps on it's own to where it eventually produces the outcome goal, crucially without prescribing any order of filling in the masked tokens or how many to do in which step. A recent paper on the matter: https://openreview.net/forum?id=MJNywBdSDy https://openreview.net/forum?id=MJNywBdSDy
- petesergeant 1y agoI wrote a simpler explanation still, that follows a similar flow, but approaches it from more of a "problems to solve" perspective: https://sgnt.ai/p/embeddings-explainer/ https://sgnt.ai/p/embeddings-explainer/
- k__ 1y agoAwesome, thanks! If I understand this correctly, there are three major problems with LLMs right now. 1. LLMs reduce a very high-dimensional vector space into a very low-dimensional vector space. Since we don't know what the dimensions in the low-dimensional vector space mean, we can only check that the outputs are correct most of the time. What research is happening to resolve this? 2. LLMs use written texts to facilitate this reduction. So, they don't learn from reality, but from what humans written down about reality. It seems like Keen Technologies tries to avoid this issue, by using (simple) robots with sensors for training, instead of human text. Which seems a much slower process, but could yield more accurate models in the long run. 3. LLMs holds internal state as a vector that reflects the meaning and context of the "conversation". Which explains, why the quality of responses deteriorates with longer conversations, if one vector is "stamped over" again and again, the meaning of the first "stamps" will get blurred. Are there alternative ways of holding state or is the only way around this to back up that state vector at every point an revert if things go awry?
- agentcoops 1y agoApologies if this comes across as too abstract, but I think your comment raises really important questions. (1) While studying the properties of the mathematical objects produced is important, I don't think we should understand the situation you describe as a problem to be solved. In old supervised machine learning methods, human beings were tasked with defining the rather crude 'features' of relevance in a data/object domain, so each dimension had some intuitive significance (often binary 'is tall', 'is blue' etc). The question now is really about learning the objective geometry of meaning, so the dimensions of the resultant vector don't exactly have to be 'meaningful' in the same way -- and, counter-intuitive as it may seem, this is progress. Now the question is of the necessary dimensionality of the mathematical space in which semantic relations can be preserved -- and meaning /is/ in some fundamental sense the resultant geometry. (2) This is where the 'Platonic hypothesis' research [1] is so fascinating: empirically we have found that the learned structures from text and image converge. This isn't saying we don't need images and sensor robots, but it appears we get the best results when training across modalities (language and image, for example). This is really fascinating for how we understand language. While any particular text might get things wrong, the language that human beings have developed over however many thousands of years really does seem to do a good job of breaking out the relevant possible 'features' of experience. The convergence of models trained from language and image suggests a certain convergence between what is learnable from sensory experience of the world and the relations that human beings have slowly come to know through the relations between words. [1] https://phillipi.github.io/prh/ https://phillipi.github.io/prh/ and https://arxiv.org/pdf/2405.07987 https://arxiv.org/pdf/2405.07987
- dotancohen 1y agoOne of the first sentences of the page clearly states: > This blog post is recommended for desktop users. That said, there is a lot of content here that could have been mobile-friendly with very little effort. The first image, of embeddings, is a prime example. It has been a very long time since I've seen any online content, let alone a blog post, that requires a desktop browser
- fastball 1y ago> Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- CamperBob2 1y agoActive malevolence in page design (for instance, look what this site does to your back button -- even Firefox can't make sense of it) is interesting, just because it is still fairly uncommon to see. Simple incompetence, not so much. But whoever wrote this wanted to kick sand in the user's face.
- stirfish 1y agoThis is interesting, and I'm curious how it came to be that way. >If your ears are more important than your eyes, you can listen to the podcast version of this article generated by NotebookLM. It looks like an LLM would read it to you; I wonder if one could have made it mobile-friendly.
- lynx97 1y agoShameless plug: If you want to experiment with semantic search for the pages you visit: https://github.com/mlang/llm-embed-proxy https://github.com/mlang/llm-embed-proxy -- a intercepting proxy as a `llm` plugin.
- mdaniel 1y agoI was going to suggest removing the extraneous network hop to pure.md but based on the notice I presume this is actually a consumer of it, so driving traffic there is a feature? https://github.com/mlang/llm-embed-proxy/blob/master/llm_embed_proxy/script.py#L184 https://github.com/mlang/llm-embed-proxy/blob/master/llm_emb...
- lynx97 1y agoThis is really just a PoC. pure.md is a pragmatic solution, since it gives good results. I was looking at markitdown but didn't find a way to disable href targets (noisy) nor did my tests of youtube transcripts work with markitdown. Keeping it on my list to monitor. Whatever works best is going to be used.
- smcleod 1y agoSeems to be down? Lots of console errors with the likes of "Content-Security-Policy: The page’s settings blocked an inline style (style-src-elem) from being applied because it violates the following directive: “style-src 'self'”." etc...
- nycdatasci 1y agoIf you want to see many more than 50 words and also have an appreciation for 3D data visualization check out embedding projector (no affiliation): https://projector.tensorflow.org/ https://projector.tensorflow.org/
- bob_theslob646 1y agoIf someone enjoyed learning about this, where should I suggest they start to learn more about embeddings?
- ayhanfuat 1y agoVicki Boykis wrote a small book about it: https://vickiboykis.com/what_are_embeddings/ https://vickiboykis.com/what_are_embeddings/
- rithikrolex 1y ago[flagged]
- rithikrolex 1y ago[flagged]
- boulevard 1y agoThis is a great visual guide! I’ve also been working on a similar concept focused on deep understanding - a visual + audio + quiz-driven lesson on LLM embeddings, hosted on app.vidyaarthi.ai. https://app.vidyaarthi.ai/ai-tutor?session_id=C2Wr46JFIqslX7vC0198I&action=replay&shared=true https://app.vidyaarthi.ai/ai-tutor?session_id=C2Wr46JFIqslX7... Our goal is to make abstract concepts more intuitive and interactive — kind of like a "learning-by-doing" approach. Would love feedback from folks here. (Not trying to self-promote — just sharing a related learning tool we’ve put a lot of thought into.)
- rithikrolex 1y ago[flagged]
- rithikrolex 1y ago[flagged]
- rithikrolex 1y ago[flagged]
- amelius 1y agoIf LLMs are so smart, then why can't they run directly on 8bit ascii input rather than tokens based on embeddings?
- pornel 1y agoThis isn't about smarts, but about performance and memory usage. Tokens are a form of compression, and working on uncompressed representation would require more memory and more processing power.
- amelius 1y agoThe opposite is true. Ascii and English are pretty good at compressing. I can say "cat" with just 24 bits. Your average LLM token embedding uses on the order of kilobits internally.
- blutfink 1y agoThe LLM can also “say” “cat” with few bits. Note that the meaning of the word as stored in your brain takes more than 24 bits.
- amelius 1y agoNo, an LLM really uses __much__ more bits per token. First, the embedding typically uses thousands of dimensions. Then, the value along each dimension is represented with a floating point number which will take 16 bits (can be smaller though with higher quantization).
- blutfink 1y agoOf course an LLM uses more space internally for a token. But so do humans. My point was that you compared how the LLM represents a token internally versus how “English” transmits a word. That’s a category error.
- montebicyclelo 1y agoNice tutorial — the contextual vs static embeddings is the important point; many are familiar with word2vec (static), but contextual embeddings are more powerful for many tasks. (However, there seems to be some serious back-button / browser history hijacking on this page.. Just scolling down the page appends a ton to my browser history, which is lame.)
- joaquincabezas 1y agoThe culprit seems to be: https://huggingface.co/spaces/hesamation/primer-llm-embedding/blob/72a6559aa81df3f4ae287382b756fe28676d2df0/src/syncHFSpacesURLHash.js#L109 https://huggingface.co/spaces/hesamation/primer-llm-embeddin... So someone, at some point, thought this was a feature
- hxtk 1y agoI thought that the point of replaceState was precisely to avoid appending elements to the history, and instead replace the most recent one, so I think I must be missing something if that line causes lots of additional history items.
- khalic 1y agoWhat a didactic and well built article! My thanks to the author
- eric-burel 1y agoAuthor's profile on Huggingface: https://huggingface.co/hesamation https://huggingface.co/hesamation HN mods suggested me to repost after a less successful share. I especially liked this article because the author goes through different types of embeddings rather than sticking to the definition.
- khalic 1y agoThank you very much my dear, he seems to have a way with words, his last post about context summarizes many concepts I have internalized but not formalized yet.
- zmmmmm 1y agoIt really surprises me that embeddings seem to be one of the least discussed parts of the LLM stack. Intuitively you would think that they would have enormous influence over the network's ability to infer semantic connections. But it doesn't seem that people talk about it too much.
- KasianFranks 1y agoAgreed. Vector embeddings along with which distance calculations you choose.
- gbacon 1y agoTend to avoid Euclidean distance.
- crystal_revenge 1y agoWhen the vectors are normalized to unit length cosine similarity and Euclidean distance are equivalent. This an optimization that many vector dbs use in retrieval since it is typically much faster to compute Euclidean distance rather than cosine.
- elite_barnacle 1y agoabsolutely. the first time i learned more deeply about embeddings i was like "whoa... at least a third of the magic of LLMs comes from embeddings". Understanding that words were already semantically arranged in such a useful pattern demystified LLMs a little bit for me. they're still wonderous, but it feels like the curtain has been rolled back a tiny bit for me
- ttul 1y agoThe weird thing about high-dimensional spaces is that most values are orthogonal to each other and most are also very far apart. It’s remarkable that you can still cluster concepts using dimension-reduction techniques when there are 50,000 dimensions to play with.
- TZubiri 1y agoI tried the openai embeddings model, but it seemed very old and uncared for, like a 2023 release iirc? Also the results were not great. Are there any good embeddings api providers?
- ryneandal 1y agoI haven't done exhaustive testing of all top-performing models on the HF Embedding Leaderboard (https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard) but I have tested a number of them extensively in the past month or two. The two best API provider models I've tested are: - JinaAI (https://jina.ai/embeddings/ https://jina.ai/embeddings/) v3 and v4 performed well in my testing. - Google's Gemini-001 model (https://ai.google.dev/gemini-api/docs/models#gemini-embedding https://ai.google.dev/gemini-api/docs/models#gemini-embeddin...). Overall, both were surpassed by Qwen3-8b (https://huggingface.co/Qwen/Qwen3-Embedding-8B https://huggingface.co/Qwen/Qwen3-Embedding-8B). Note, this was specifically regarding English and Code embedding generation/retrieval, with reranking.
- visarga 1y agoI think it is more informative to simply visualize a word cloud or even to show top-k results for a query. Something like https://projector.tensorflow.org/ https://projector.tensorflow.org/ just type a word in, select UMAP projection.
- xg15 1y ago> While we can use pretrained models such as Word2Vec to generate embeddings for machine learning models, LLMs commonly produce their own embeddings that are part of the input layer and are updated during training. So out of interest: During inference, the embedding is simply a lookup table "token ID -> embedding vector". Mathematically, you could represent this as encoding the token ID as a (very very long) one-hot vector, then passing that through a linear layer to get the embedding vector. The linear layer would contain exactly the information from the lookup table. My question: Is this also how the embeddings are trained? I.e. just treat them as a linear layer and include them in the normal backpropagation of the model?
- montebicyclelo 1y agoSo, they are included in the normal backpropagation of the model. But there is no one-hot encoding, because, although you are correct that it is equivalent, it would be very inefficient to do it that way. You can make indexing differentiable, i.e. gradient descent flows back to the vectors that were selected, which is more efficient than a one-hot matmul. (If you're curious about the details, there's an example of making indexing differentiable in my minimal deep learning library here: https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92350401da2ab891c9098727/smallpebble/smallpebble.py#L216 https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...)
- xg15 1y agoAh, that makes sense, thanks a lot!
- asjir 1y agoTo expand upon the other comment: Indexing and multiplying with one-hot embeddings are equivalent. IF N is vocab size and L is sequence length, you'd need to create a NxL matrix, and multiply it with the embedding matrix. But since your NxL matrix will be sparse with only a single 1 per column, it'd make sense to represent it internally as just one number per column, representing the index at which 1 is. At which point if you defined new multiplication by this matrix, it would basically just index with this number. And just like you write a special forward pass, you can write a special backward pass so that backpropagation would reach it.
- vkawasth 1y agoOne can visulaize how embeddings transform using Alpha Complexes. https://www.preprints.org/manuscript/202505.0097/v1 https://www.preprints.org/manuscript/202505.0097/v1 As embeddings transfer through various layers, you can see what contribution each layer of transformer is making to classification. There are 3 types of holes that form 1-d, 2-d 3-d... each is telling the shape of data (embedding) as it traverses... It can help is reducing layers/reducing backprop. Some layers are more important than others... You will see none of this using Vietoris Rips!
- k90k90k90 1y agohttps://g.co/gemini/share/893c0a4af623 https://g.co/gemini/share/893c0a4af623 in case you want to play and visually understand the traditional PE;