7 ms·

Embeddings are underrated (2024)

484 points by jxmorris12 1y ago

jacobr1 1y ago
I may have missed it ... but were any direct applications to tech writers discussed in this article? Embeddings are fascinating and very important for things like LLMs or semantic search, but the author seems to imply more direct utility.
- PaulHoule 1y ago
  Semantic search and classification and clustering. For the first, there is a substantial breakthrough in IR every 10 years or so you take what you can get. (I got so depressed reading TREC proceedings which seemed to prove that "every obvious idea to improve search relevance doesn't work" and it wasn't until I found a summary of the first ten years that I learned that the first ten years had turned up one useful result, BM2.5) As for classification, it is highly practical to put a text through an embedding and then run the embedding through a classical ML algorithm out of https://scikit-learn.org/stable/supervised_learning.html https://scikit-learn.org/stable/supervised_learning.html This works so consistently that I'm considering not packing in a bag-of-words classifier in a text classification library I'm working on. People who hold court on Huggingface forums tends to believe you can do better with fine-tuned BERT, and I'd agree you can do better with that, but training time is 100x and maybe you won't. 20 years ago you could make bag-of-word vectors and put them through a clustering algorithm https://scikit-learn.org/stable/modules/clustering.html https://scikit-learn.org/stable/modules/clustering.html and it worked but you got awful results. With embeddings you can use a very simple and fast algorithm like https://scikit-learn.org/stable/modules/clustering.html#k-means https://scikit-learn.org/stable/modules/clustering.html#k-me... and get great clusters. I'd disagree with the bit that it takes "a lot of linear algebra" to find nearby vectors, it can be done with a dot product so I'd say it is "a little linear algebra"
- podgietaru 1y ago
  I built an rss aggregator with semantic search using embeddings. The main usage was being able to categorise based on any randomly created category. So you could have arbitrary categories https://github.com/aws-samples/rss-aggregator-using-cohere-embeddings-bedrock https://github.com/aws-samples/rss-aggregator-using-cohere-e... Unfortunately I no longer work at AWS so the infrastructure that was running it is down.
- kaycebasques 1y ago
  > were any direct applications to tech writers discussed in this article No, it was supposed to be a teaser post followed up by more posts and projects exploring the different applications of embeddings in technical writing (TW). But alas, life happened, and I'm now a proud new papa with a 3-month old baby :D I do have other projects and embeddings-related posts in the pipeline. Suffice to say, embeddings can help us make progress on all 3 of the "intractable" challengs of TW mentioned here: https://technicalwriting.dev/strategy/challenges.html https://technicalwriting.dev/strategy/challenges.html
  - jacobr1 1y ago
    Thanks for sharing regardless. It was a good overview for those less familiar with the material.
  - kaycebasques 1y ago
    Also re: a direct application I forgot to mention this: https://www.tdcommons.org/dpubs_series/8057/ https://www.tdcommons.org/dpubs_series/8057/ (It finally published last week after being in review purgatory for months)
- sansseriff 1y ago
  It would be great to semantically search through literature with embeddings. At least one person I know if is trying to generate a vector database of all arxiv papers. The big problem I see is attribution and citations. An embedding is just a vector. It doesn't contain any citation back to the source material or modification date or certificate of authenticity. So when using embeddings in RAG, they only serve to link back to a particular page of source material. Using embeddings as links doesn't dramatically change the way citation and attribution are handled in technical writing. You still end up citing a whole paper or a page of a paper. I think GraphRAG [1] is a more useful thing to build on for technical literature. There's ways to use graphs to cite a particular concept of a particular page of an academic paper. And for the 'citations' to act as bidirectional links between new and old scientific discourse. But I digress [1] https://microsoft.github.io/graphrag/ https://microsoft.github.io/graphrag/
  - kaycebasques 1y ago
    IMO, for technical writing, citing a page or section within a page is usually good enough. I rarely need to cite a particular concept. But I've never even thought of the possibility of more granular concept-level citations and will definitely be pondering it more!
lblume 1y ago
Semantic search seems like a more promising usecase than simple related articles. A big problem with classical keyword-based search is that synonyms are not reflected at all. With semantic search you can search for what you mean, not what words you expect to find on the site you are looking for.
- kgeist 1y ago
  In my benchmarks for a service which is now running in production, hybrid search based on both keywords and embeddings performed the best. Sometimes you need exact keyword matches; other times, synonyms are more useful. Hybrid search combines both sets of results into a single, unified set. OpenSearch has built-in support for this approach.
- PaulHoule 1y ago
  A case related to that is "more like this" which in my mind breaks down into two forks: (1) Sometimes your query is a short document. Say you wanted to know if there were any patents similar to something you invented. You'd give a professional patent searcher a paragraph or a few paragraphs describing the invention, you can give a "semantic search engine" the paragraph -- I helped build one that did about as well as the professional using embeddings before this was cool. (2) Even Salton's early works on IR talked about "relevance feedback" where you'd mark some documents in your results as relevant, some as irrelevant. With bag-of-words this doesn't really work well (it can take 1000 samples for a bag-of-words classifier to "wake up") but works much better with embeddings. The thing is that embeddings are "hunchy" and not really the right data structure to represent things like "people who are between 5 feet and 6 feet tall and have been on more than 1000 airplane flights in their life" (knowledge graph/database sorts of queries) or "the thread that links the work of Derrida and Badiou" (could be spelled out logically in some particular framework but doing that in general seems practically intractable)
  - ted_dunning 1y ago
    Regarding item (2), Salton did talk about this and many systems from the early TREC times implemented them. In my own work, I found that relevance feedback worked very well with only a few dozen judgements. See Chapter 7 starting on page 83 in my dissertation: https://arxiv.org/abs/1207.1847 https://arxiv.org/abs/1207.1847 Note also that the evaluation of Luduan in that chapter compared against state-of-the-art (for the time) back-of-words systems but also an early word embedding retrieval system (Convectis from HNC).
ncruces 1y ago
Previous discussion: https://news.ycombinator.com/item?id=42013762 https://news.ycombinator.com/item?id=42013762
- dang 1y ago
  Thanks! Macroexpanded: Embeddings are underrated - https://news.ycombinator.com/item?id=42013762 https://news.ycombinator.com/item?id=42013762 - Nov 2024 (176 comments)
kaycebasques 1y ago
Hello, I wrote this. Thank you for reading! The post was previously discussed 6 months ago: https://news.ycombinator.com/item?id=42013762 https://news.ycombinator.com/item?id=42013762 To be clear, when I said "embeddings are underrated" I was only arguing that my fellow technical writers (TWs) were not paying enough attention to a very useful new tool in the TW toolbox. I know that the statement sounds silly to ML practitioners, who very much don't "underrate" embeddings. I know that the post is light on details regarding how exactly we apply embeddings in TW. I have some projects and other blog posts in the pipeline. Short story long, embeddings are important because they can help us make progress on the 3 intractable challenges of TW: https://technicalwriting.dev/strategy/challenges.html https://technicalwriting.dev/strategy/challenges.html
- rybosome 1y ago
  Thanks for the write-up! I’m curious how you found the quality of the results? This gets into evals which ML folks love, but even just with “vibes” do the results eyeball as reasonable to you?
  - kaycebasques 1y ago
    By results I assume that you're asking about the related pages experiment? The results were definitely promising. A lot of the calculated related pages were totally reasonable. E.g. if I'm reading https://www.sphinx-doc.org/en/master/development/html_themes/index.html https://www.sphinx-doc.org/en/master/development/html_themes... then it's very reasonable to assume that I may also be interested in https://www.sphinx-doc.org/en/master/usage/theming.html https://www.sphinx-doc.org/en/master/usage/theming.html
- sgbeal 1y ago
  > I know that the post is light on details regarding how exactly we apply embeddings in TW. More significantly, after having read the first 6 or 8 paragraphs, i still have no clue what an "embedding" is. From the 3rd paragraph: > Here’s an overview of how you use embeddings and how they work. But no mention of what they are (unless perhaps it's buried far deeper in the article).
petesergeant 1y ago
I wrote an embeddings explainer a few days ago if anyone is interested: https://sgnt.ai/p/embeddings-explainer/ https://sgnt.ai/p/embeddings-explainer/ Very little maths and lots of dogs involved.
podgietaru 1y ago
I wrote a blog post about embedding - and a sample application to show their uses. https://aws.amazon.com/blogs/machine-learning/use-language-embeddings-for-zero-shot-classification-and-semantic-search-with-amazon-bedrock/ https://aws.amazon.com/blogs/machine-learning/use-language-e... https://github.com/aws-samples/rss-aggregator-using-cohere-embeddings-bedrock https://github.com/aws-samples/rss-aggregator-using-cohere-e... I really enjoy working with embedding. They’re truly fascinating as a representation of meaning - but also a very cheap and effective way to perform very cheap things like categorisation and clustering.
- btbuildem 1y ago
  How would you approach using them in a specialized discipline (think technical jargon, acronyms etc) where traning a model from scratch is practically impossible because everyone (customers, solution providers) fiercely guards their data? A generic embedding model does not have enough specificity to cluster the specialized terms or "code names" of specific entities (these differ across orgs but represent the same sets of concepts within the domain). A more specific model cannot be trained because the data is not available. Quite the conundrum!
  - minimaxir 1y ago
    You can fine-tune existing embedding models.
tyho 1y ago
> The 2D map analogy was a nice stepping stone for building intuition but now we need to cast it aside, because embeddings operate in hundreds or thousands of dimensions. It’s impossible for us lowly 3-dimensional creatures to visualize what “distance” looks like in 1000 dimensions. Also, we don’t know what each dimension represents, hence the section heading “Very weird multi-dimensional space”.5 One dimension might represent something close to color. The king - man + woman ≈ queen anecdote suggests that these models contain a dimension with some notion of gender. And so on. Well Dude, we just don’t know. nit. This suggests that the model contains a direction with some notion of gender, not a dimension. Direction and dimension appear to be inextricably linked by definition, but with some handwavy maths, you find that the number of nearly orthogonal dimensions within n dimensional space is exponential with regards to n. This helps explain why spaces on the order of 1k dimensions can "fit" billions of concepts.
- drc500free 1y ago
  Is this because we can essentially treat each dimension like a binary digit, so we get 2^n directions we can encode? Or am I barking up totally the wrong tree?
  - emaro 1y ago
    Basically, but it gets even better. If you allow directions of 'meaning' do wiggle a little bit (say, between 89 and 91 degrees to all other directions), you get a lot more degrees of freedom. In 3 dimensions, you still only get 3 meaningful directions with that wiggle-freedom. However in high-dimensional spaces, this small additional freedom allows you to fit a lot more almost orthogonal directions than the number of strictly orthogonal ones. That means in a 1000-dimensional space you can fit a huge number >> 1000 of binary concepts.
- kaycebasques 1y ago
  Oh yes, this makes a lot of sense, thank you for the "nit" (which doesn't feel like a nit to me, it feels like an important conceptual correction). When I was writing the post I definitely paused at that part, knowing that something was off about describing the model as having a dimension that maps to gender. As you said, since the models are general-purpose and work so well in so many domains, there's no way that there's a 1-to-1 correspondence between concepts and dimensions. I think your comment is also clicking for me now because I previously did not really understand how cosine similarity worked, but then watched videos like this and understand it better now: https://youtu.be/e9U0QAFbfLI https://youtu.be/e9U0QAFbfLI I will eventually update the post to correct this inaccuracy, thank you for improving my own wetware's conceptual model of embeddings
jbellis 1y ago
Great to see embeddings getting some love outside the straight-up-ML space! I had a non-traditional use case recently, as well. I wanted to debounce the API calls I'm making to gemini flash as the user types his instructions, and I decided to try a very lightweight embeddings model, light enough to run on CPU and way too underpowered to attempt vector search with. It works pretty well! https://brokk.ai/blog/brokk-under-the-hood https://brokk.ai/blog/brokk-under-the-hood
stefanka 1y ago
I like that this looks like a very ethical and "fair" use of the LLM technology
- nameless912 1y ago
  Yeah, the ethics around _training_ models that generate embeddings is still suspect to me, but the use of embeddings as a cheap, efficient way to provide semantic similarity seems very valuable. I've started dipping my toes in doing real, honest-to-goodness "machine learning" at work and it's mostly involved having OpenAI create embeddings for support logs my team generates, and we're starting to get value out of being able to cluster certain types of issues together, which I'm excited by. But this kind of stuff is truly augmentative: representing complex ideas in easily-searchable vector spaces, making connections in datasets too vast for humans to comb through alone, that's actual value.
minimaxir 1y ago
> I don’t know. After the model has been created (trained), I’m pretty sure that generating embeddings is much less computationally intensive than generating text. An embedding is generated after a single pass through the model, so functionally it's the equivalent of generating a single token from an text generation model.
- energy123 1y ago
  I might be wrong but aren't embedding models usually bidirectional and not causal, so the attention mechanism itself is more expensive.
  - breadislove 1y ago
    yes exactly
  - minimaxir 1y ago
    It depends on the architecture (you very well can convert a decoder-only causal model to an embeddings model, e.g. Qwen/Mistral), but it is true the traditional embeddings models such as a BERT-based one are bidirectional, although unclear how much more compute that inherently requires. Compare to ModernBERT, which uses more modern techniques and is still bidirectional, but it is very very speedy. https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert
- kaycebasques 1y ago
  Will update the post to capture that interesting tidbit, thanks And also, thanks for this post! https://minimaxir.com/2025/02/embeddings-parquet/ https://minimaxir.com/2025/02/embeddings-parquet/
jasonjmcghee 1y ago
Another very cool attribute of embeddings and embedding search is that they are resource cheap enough that you can perform them client side. ONNX models can be loaded and executed with transformer.js https://github.com/huggingface/transformers.js/ https://github.com/huggingface/transformers.js/ You can even build and statically host indices like hnsw for embeddings. I put together a little open source demo for this here https://jasonjmcghee.github.io/portable-hnsw/ https://jasonjmcghee.github.io/portable-hnsw/ (it's a prototype / hacked together approximation of hnsw, but you could implement the real thing) Long story short, represent indices as queryable parquet files and use duckdb to query them. Depending on how you host, it's either free or nearly free. I used Github Pages so it's free. R2 with cloudflare would only cost the size what you store (very cheap- no egress fees).
- qq99 1y ago
  I was wondering about this. I was hesitant to add embedding-based search to my app because I didn't want to incur the latency to the embedding API provider blocking every search on initial render. Granted, you can cache the embeddings for common searches. OTOH, I also don't want to render something without them, perform the embedding async, and then have to reify the results list once the embedding arrives. Seems hard to sensibly do that from a UX perspective. To render locally, you need access to the model right? I just wonder how good those embeddings will be compared to those from OpenAI/Google/etc in terms of semantic search. I do like the free/instant aspect though
  - jasonjmcghee 1y ago
    checkout MTEB (https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard) many of the open source ones are actually _better_. I've had a particularly good experiences with nomic, bge, gte, and all-MiniLM-L6-v2. All are hundreds of MB (except all-minilm which is like 87MB)
    - simonw 1y ago
      I love all-MiniLM-L6-v2 - 87MB is tiny enough that you could just load it into RAM in a web application process on a small VM. From my experiments with it the results are Good Enough for a lot of purposes. https://simonwillison.net/2023/Sep/4/llm-embeddings/#embedding-in-bulk https://simonwillison.net/2023/Sep/4/llm-embeddings/#embeddi...
charcircuit 1y ago
How are they underrated when they have been been used by the top sites for over a decade? The author doesn't really explain why he thinks they are underrated despite them being behind almost every search and recommendation users receive on their computers.
- minimaxir 1y ago
  Underrated is more a relative term, and embeddings are definitely underrated to all the other uses for the LLM boom.
  - charcircuit 1y ago
    Embeddings were popularized preLLM. They didn't come from the LLM boom.
- esafak 1y ago
  The author is addressing technical writers, not engineers. https://news.ycombinator.com/item?id=43964290 https://news.ycombinator.com/item?id=43964290
daxfohl 1y ago
I wonder if this could be used to find redundant code
milindsoni 1y ago
[dead]
milindsoni 1y ago
I was using transformer.js to generate and use embeddings with small models in the browser itself, its quite useful to implement any kind of semantic search.
breadislove 1y ago
embeddings and consequently retrieval is probably the most overlooked topic in this whole AI thing. great to see people tinkering with it
adefa 1y ago
Here’s a CLI I’m experimenting with https://github.com/TrevorS/rhizome https://github.com/TrevorS/rhizome that indexes local repos with Tree‑sitter, stores ONNX embeddings in SQLite, and answers semantic queries offline; for example, `rhizome search "pull apart"` surfaces relevant snippets across projects: rhizome search --limit 2 "pull apart" Model already exists at \~/.rhizome/models/bge-small-en-v1.5.onnx \~/Projects/rhizome/src/chunking.rs:458\:fn rust\_no\_structural\_items\_fallback() { \~/Projects/rhizome/src/lib.rs:2\:pub mod chunking;
ubj 1y ago
> Because we always get back the same amount of numbers no matter how big or small the input text, we now have a way to mathematically compare any two pieces of arbitrary text to each other. I think there needs to be some more clarification here. Hash functions also return the same sized output no matter how big or small the input text. However, mathematically comparing two hashes is going to have a much different meaning than mathematically comparing two embeddings. I'd recommend emphasizing that embeddings are training dependent--the quality of comparison will depend on the quality and type of training used to produce the embedding. There isn't some single "universal embedding" that allows for meaningful comparison of arbitrary text.
- kaycebasques 1y ago
  Thanks for the feedback. I've been reading the research papers behind how these models are created e.g. Gecko [1] and NV-Embed [2] and Gemini Embedding [3] so I'm starting to grok what you're getting at and will find a way to make the post more accurate on this front [1] https://arxiv.org/abs/2403.20327 https://arxiv.org/abs/2403.20327 [2] https://arxiv.org/abs/2405.17428 https://arxiv.org/abs/2405.17428 [3] https://arxiv.org/abs/2503.07891 https://arxiv.org/abs/2503.07891
bawolff 1y ago
> I could tell you exactly how I think we might advance the state of the art in technical writing with embeddings, but where’s the fun in that? You now know why they’re such an interesting and useful new tool in the technical writer toolbox… go connect the rest of the dots yourself! Wow, that's bold. I guess "good" technical writing no longer includes a thesis statement. Seriously though, why would this be useful for technical writing? Sure you could make some similar pages widget however i dont think i've ever wanted that when reading technical docs, let alone writing them.
- simonw 1y ago
  Related documents aside, technical documentation benefits from really great search. Embeddings are a _very_ useful tool for building better search - they can handle "fuzzy" matches, where a user can say things like "that feature that lets me run a function against every column of data" because they can't remember the name of the feature. With embeddings you can implement a hybrid approach, where you mix both keyword search (still necessary because embeddings can miss things that use jargon they weren't trained on) and vector similarity search. I wish I had good examples to point to for this!
  - kaycebasques 1y ago
    In-site search is super important. I suspect that many docs maintainers don't realize how heavily it's used. Many docs sites don't even track in-site search queries! One of the things I love about Sphinx is that it has a decent, client-side, JS-powered offline search. I recently hacked together a workflow for making it search-as-you-type [1]. jasonjmcghee's comment [2] has got me pondering whether we can augment it with transformer.js embeddings. [1] https://github.com/orgs/sphinx-doc/discussions/13222 https://github.com/orgs/sphinx-doc/discussions/13222 [2] https://news.ycombinator.com/item?id=43964913 https://news.ycombinator.com/item?id=43964913
- kaycebasques 1y ago
  > I guess "good" technical writing no longer includes a thesis statement. Thesis is outlined in the second paragraph: > What embeddings offer to technical writers is the ability to discover connections between texts at previously impossible scales. I think it's fair, however, to say that this post is ineffective because it does not provide concrete examples of the thesis in action. My only excuse is that I never intended for this to be a standalone post but life got in the way (in the best possible way!) https://news.ycombinator.com/item?id=43964584 https://news.ycombinator.com/item?id=43964584 > why would this be useful for technical writing? You're not going to like this answer, because it's also vague. There are 3 intractable challenges in technical writing. Embeddings can help us make progress on all 3: https://technicalwriting.dev/strategy/challenges.html https://technicalwriting.dev/strategy/challenges.html See also https://www.tdcommons.org/dpubs_series/8057/ https://www.tdcommons.org/dpubs_series/8057/
jas8425 1y ago
If embeddings are roughly the equivalent of a hash at least insofar as they transform a large input into some kind of "content-addressed distillation" (ignoring the major difference that a hash is opaque whereas an embedding has intrinsic meaning), has there been any research done on "cracking" them? That is, starting from an embedding and working backwards to generate a piece of text that is semantically close by? I could imagine an LLM inference pipeline where the next token ranking includes its similarity to the target embedding, or perhaps instead the change in direction towards/away from the desired embedding that adding it would introduce. Put another way, the author gives the example: > embedding("king") - embedding("man") + embedding("woman") ≈ embedding("queen") What if you could do that but for whole bodies of text? I'm imagining being able to do "semantic algebra" with whole paragraphs/articles/books. Instead of just prompting an LLM to "adjust the tone to be more friendly", you could have the core concept of "friendly" (or some more nuanced variant thereof) and "add" it to your existing text, etc.
- luke-stanley 1y ago
  "starting from an embedding and working backwards to generate a piece of text that is semantically close by?" Apparently this is called embedding inversion and Universal Zero-shot Embedding Inversion https://arxiv.org/abs/2504.00147 https://arxiv.org/abs/2504.00147 Going incrementally closer and closer to the target with some means to vary seems to be the most general way, there are lots of ways to be more optimal though. Image diffusion with CLIP embeddings and such is kinda related too.
  - luke-stanley 1y ago
    I meant to say: Apparently this is called "embedding inversion", and that "Universal Zero-shot Embedding Inversion" is a related paper that covers a lot of the basics. Recently I learned that a ArXiv RAG agent by ArXiv Labs is a really cool way for people wanting to find out about research: https://www.alphaxiv.org/assistant https://www.alphaxiv.org/assistant Though I had ran into "inversion" before, the AlphaXiv Assistant introduced me to "embedding inversion".
- jerjerjer 1y ago
computerex 1y ago
This article really rubbed me the wrong way. > I could tell you exactly how I think we might advance the state of the art in technical writing with embeddings, but where’s the fun in that? You now know why they’re such an interesting and useful new tool in the technical writer toolbox… go connect the rest of the dots yourself! I read the article because of the title, only to find the above.
- kaycebasques 1y ago
  You're not alone. I've gotten lots of feedback that the post buries the lede [1] and is ineffective because it doesn't provide more concrete examples. I will fix it eventually, I promise! See here for context: https://news.ycombinator.com/item?id=43964584 https://news.ycombinator.com/item?id=43964584 [1] It's perhaps not even appropriate to say "buried the lede" because that implies the lede is dug back up at some point, whereas this post buries the lede and then forgets where the lede was buried!!
archeantus 1y ago
Great read. Brain started fry when it talked about infinite dimensions but then I imagined it looking like the black hole scene at the end of interstellar and that helped a bit (though it’s not like that scene made any sense either)
jonathanrmumm 1y ago
Embeddings are a new jump to universality, like the alphabet or numbers. https://thebeginningofinfinity.xyz/Jump%20to%20Universality https://thebeginningofinfinity.xyz/Jump%20to%20Universality
tucnak 1y ago
Surprised they never mentioned jina.ai models such as jina-embeddings-v3 at 8K context and outperforming most "contenders" in the MTEB or jina-clip-v2 (multimodal) or "late chunking," also known as mean pooling: https://jina.ai/news/late-chunking-in-long-context-embedding-models/ https://jina.ai/news/late-chunking-in-long-context-embedding... The article feels incomplete
- kaycebasques 1y ago
  Wasn't on my radar! Thanks for the pointer. I'll look into it. It's hard to keep up with all the embedding models out there.
gnarlouse 1y ago
The article did not go the direction that I imagined, which I loved. What if we could use embedding spaces to clean up documentation to be more explicit and direct? The biggest thing that stuck with me from my technical writing class was “get to the point”. Finding ways to narrow word choices in an almost 1984’esque ingSoc seems appropriate? Ish?
- energy123 1y ago
  Neural nets aren't invertible so I don't see how embeddings can help reduce verbosity. It's not like we could take the embedding of a piece of verbose text and figure out less verbose phrasings that have an embedding with high similarity without using some expensive search. At which point it seems better to just use a standard LLM for that.
  - kaycebasques 1y ago
    Yes, I have experimented with using ML for style concerns such as consistent capitalization and verbosity and can confirm: no need for embeddings here. You can get the job done with only a text generation model. I actually can't visualize how an embeddings-powered implementation would work here
simianwords 1y ago
I don't think any of the current consumer LLM tools use embeddings for web search. Instead they do it at the text level. The evidence for this is the COT summary with ChatGPT - I have seen something where the the LLM uses quotes to grep on the web. Embeddings seem good in theory but in practice its probably best to ask an LLM to do a deep search instead by giving it instructions like "use synonyms and common typos and grep". Does any one know any live example of a consumer product using embeddings?
- zhobbs 1y ago
  My understanding is that modern search engines are using embeddings / vector search under the hood. So even if LLM's aren't directly passing a vector to the search engine, my assumption is that the search engine is converting to a vector and searching. "You interact with embeddings every time you complete a Google Search" from https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings https://cloud.google.com/vertex-ai/generative-ai/docs/embedd...
  - simianwords 1y ago
    Fair, and maybe key point here is that it uses embeddings to help with the search results along with many manual heuristics in place. I hardly think google search works just by dumping embeddings then doing KNN's and calling it a day.
- dcre 1y ago
  I believe they use the LLMs to generate a set of things to search for and then run those through existing search engines, which are totally opaque and use whatever array of techniques SOTA search engines use. They are almost certainly not "grepping" the internet.
  - simianwords 1y ago
    yes that's what i meant thanks for clarifying. the grepping part is definitely done at least in spirit where the COT includes quotes. if i were searching for top 10 cars that are manufactured in South America for example, the COT might show: "Brazil" car manufacture This forces Brazil to be included in the keywords, at least that's how google (used to?) works.
antirez 1y ago
For a coincidence two hours ago I posted a video on my channel where it can be seen how king-man+woman also works with CLIP image embeddings. This may be obvious for people that worked with CLIP extensively, or that tried embeddings math in other embedding spaces, but it really surprised me. https://youtu.be/r6TJfGUhv6s?si=wG6h1kdigiPrNFdk https://youtu.be/r6TJfGUhv6s?si=wG6h1kdigiPrNFdk Video is in English but please pardon my and my friend Italian accents...
gweinberg 1y ago
I don't understand why some people consider the concept of many dimensions to be so mysterious. It's numbers specifying something like degrees of freedom. If I wanted to specify the position of my body sitting at my desk, I might say use two angles to specify the what is happening at each joint, and so would probably need a couple hundred to fully specify my position. A human being probably could not look at the numbers and see "he's sitting at a desk", but I don't see the conceptual difficulty".
- JohnKemeny 1y ago
  You're right that higher dimensions are just more degrees of freedom, and mathematically it's just more numbers. But what's counterintuitive is how geometry behaves differently as dimensions grow. Things like distance, volume, and angles don’t scale the way we expect. For example, in high dimensions, almost all the volume of a sphere concentrates near its surface, and random vectors tend to be nearly orthogonal—something that rarely happens in 2D or 3D. These effects matter in practice. In high-dimensional spaces like word embeddings, even unrelated points can seem equidistant, making basic tasks like clustering or similarity search much harder. So it's not that higher dimensions are mysterious per se, but that they defy the spatial intuitions we've developed from living in three.
mkasberg 1y ago
I started using embeddings to find similar posts on my blog about a year ago. The results were awesome: https://www.mikekasberg.com/blog/2024/04/23/better-related-posts-in-jekyll-using-ai.html https://www.mikekasberg.com/blog/2024/04/23/better-related-p... Such a simple tool to implement with so much power in certain situations!
ColinEberhardt 1y ago
Agreed, embeddings are a very useful tool. Started using them for article recommendations on our blog a few years ago, with a measured increase in engagement: https://blog.scottlogic.com/2022/02/23/word-embedding-recommendations.html https://blog.scottlogic.com/2022/02/23/word-embedding-recomm...
- wussboy 1y ago
  Where there any other metrics that improved besides engagement? Or is that your only metric? If it is your only metric what does it really prove?
daturkel 1y ago
I gave a very short talk (now a blog post) about embeddings and we use them to bridge the gap between human notions of understanding and digital representations. It might be of interest to people who enjoyed this post: https://danturkel.com/2025/03/10/ignite-machine-understanding-vectors.html https://danturkel.com/2025/03/10/ignite-machine-understandin...
Lerc 1y ago
There's a talk by Leland McInnes (on my phone so can't find it rn) where he shows how to make quite effective embeddings through off-the-shelf analytical processes rather than using a large trained model.
- kanak 1y ago
  Is it this one? https://www.youtube.com/watch?v=Z6s7PrfJlQ0&t=3084s https://www.youtube.com/watch?v=Z6s7PrfJlQ0&t=3084s > MIA Meeting: Low dimensional embeddings of words and documents (and how they might apply to single-cell data) Leland McInnes Tutte Institute for Mathematics and Computing Over the last decade the field of Natural Language Processing (NLP) has been overtaken by neural networks and deep learning. The latest models and algorithms, from word2vec to Google’s Universal Sentence Encoder and BERT, can perform seemingly magical feats and provide powerful tools to understand and analyze text documents. This talk will seek to pick apart word and document embedding techniques from NLP, removing the neural networks and instead working with linear algebra and dimension reduction on large sparse matrices of counts that is very similar to a large amount of single-cell data. Such an approach can achieve results on par with neural network approaches, but are both simpler to understand the inner workings of, and are generalizable to much more diverse domains than NLP, including, hopefully, single-cell research. In particular all we really need to apply these kinds of techniques are large sparse matrices of counts and some notion of locality or cooccurrence of the feature columns of that matrix. The goal of this talk is to hopefully open up some of the ideas that have been so effective in NLP and make them usable and accessible to a wider audience working with more diverse types of data, and initiate a conversation about what is required to make such techniques useful for the single-cell community.
  - Lerc 1y ago
    That looks like it. Been meaning to give it a try on a few different data types. Not enough hours in the day.
paulnovacovici 1y ago
Completely agree. I’ve been experimenting with embeddings by building Recallify, primarily to help me quickly retrieve obscure things I’ve read online. Even at just 1024 dimensions, it’s impressive how effectively embeddings capture and surface ideas based purely on semantic similarity, rather than exact keyword recall (which I’m pretty terrible at remembering). It’s been a game changer for turning fuzzy mental concepts into actionable insights. Beta testing an iOS app for it if anyone is interested: https://recallify.app/ https://recallify.app/
eabeezxjc 1y ago
please look at rust package.The amount is striking and is constantly increasing not decreasing!
deleted 1y ago
[deleted]
joaoli131 1y ago
Embeddings are still underrated—even in RAG. Legal text is deeply hierarchical and full of pointers (“Art. 5 CF”, “see Art 34”). One vector per article leaves too much on the table. Things that moved the needle for us: – *Multi-layer embeds* vectors for every paragraph and every structural level (chapter → book). Retriever picks the right granularity. (arXiv:2411.07739) – *Propositional queries* strip speech-act fluff (“could you please…”) before embedding. Similarity + top-k recall jump. (arXiv:2503.10654) – *Poly-vector retrieval* two vectors per norm—one for content, one for the label/nickname. Handles “what does the CDC say?” and internal cross-refs. (arXiv:2504.10508) *TL;DR* If your corpus has hierarchy or aliases, stop thinking “one doc = one embedding.” Plenty of juice to squeeze before heavier tricks. [1] https://arxiv.org/abs/2411.07739 https://arxiv.org/abs/2411.07739 [2] https://arxiv.org/abs/2503.10654 https://arxiv.org/abs/2503.10654 [3] https://arxiv.org/abs/2504.10508 https://arxiv.org/abs/2504.10508
nakedneuron 1y ago
Question for the experts: As embeddings in question reflect content is there another vector for style? I was wondering if stylometry research would profit from embeddings becoming more easily accessible than ever. I dug my head into this matter some time ago and believe this would be the right tool.
- n3t 1y ago
  If you have a dataset for which you know which elements have similar/same style, you can create your own embeddings for style.
- navar 1y ago
  Embeddings as a tool have been around for longer than LLMs. They were (and are) ubiquitous in, e.g., recommender systems. It sounds maybe this would be more in-line with what you are looking for. In this case, check out https://github.com/benfred/implicit https://github.com/benfred/implicit - I have used it in the past with great success.
navar 1y ago
[dead]
ADAMOsong 1y ago
I HAVE MY EX BACK TO ME WITH THE SOLUTION FROM Dr. SHAUN MURPHY. I came in contact with Dr. SHAUN MURPHY. Through a testimony written about him, I have also encountered many testimonies about how he has been helping others with their life. To get back with an ex is one of the innermost feelings many people would love to experience, especially as those memories with our ex always cloud our mind when someone else does some of those things our ex used to do. I was a single parent for almost 6 years, and though my ex-husband was far away from my kids, I still wish someday he would return to me. I encountered this spell doctor, Dr. SHAUN MURPHY. It helped me with my wishes, and I am happy to say I am back with my ex-husband. He assured me that the result manifested within 48 hours, and our love and connection were restored amazingly. If you want to get back with your ex successfully, contact this great spell caster with HIS EMAIL: doctormurphhy@yahoo.com‎
mijoharas 1y ago
So, just the other day I was looking for a "grep" like tool that'll allow me to find files based on embedding similarity from a set of local files. I'm assuming it'll have an indexing process, and then a little vector db locally, and then when I grep it can search that and return me results. Does anything like that exist? I couldn't find anything like it.

484 points by jxmorris12 1y ago

I may have missed it ... but were any direct applications to tech writers discussed in this article? Embeddings are fascinating and very important for things like LLMs or semantic search, but the author seems to imply more direct utility.

PaulHoule 1y ago
Semantic search and classification and clustering. For the first, there is a substantial breakthrough in IR every 10 years or so you take what you can get. (I got so depressed reading TREC proceedings which seemed to prove that "every obvious idea to improve search relevance doesn't work" and it wasn't until I found a summary of the first ten years that I learned that the first ten years had turned up one useful result, BM2.5) As for classification, it is highly practical to put a text through an embedding and then run the embedding through a classical ML algorithm out of https://scikit-learn.org/stable/supervised_learning.html https://scikit-learn.org/stable/supervised_learning.html This works so consistently that I'm considering not packing in a bag-of-words classifier in a text classification library I'm working on. People who hold court on Huggingface forums tends to believe you can do better with fine-tuned BERT, and I'd agree you can do better with that, but training time is 100x and maybe you won't. 20 years ago you could make bag-of-word vectors and put them through a clustering algorithm https://scikit-learn.org/stable/modules/clustering.html https://scikit-learn.org/stable/modules/clustering.html and it worked but you got awful results. With embeddings you can use a very simple and fast algorithm like https://scikit-learn.org/stable/modules/clustering.html#k-means https://scikit-learn.org/stable/modules/clustering.html#k-me... and get great clusters. I'd disagree with the bit that it takes "a lot of linear algebra" to find nearby vectors, it can be done with a dot product so I'd say it is "a little linear algebra"
podgietaru 1y ago
I built an rss aggregator with semantic search using embeddings. The main usage was being able to categorise based on any randomly created category. So you could have arbitrary categories https://github.com/aws-samples/rss-aggregator-using-cohere-embeddings-bedrock https://github.com/aws-samples/rss-aggregator-using-cohere-e... Unfortunately I no longer work at AWS so the infrastructure that was running it is down.
kaycebasques 1y ago
> were any direct applications to tech writers discussed in this article No, it was supposed to be a teaser post followed up by more posts and projects exploring the different applications of embeddings in technical writing (TW). But alas, life happened, and I'm now a proud new papa with a 3-month old baby :D I do have other projects and embeddings-related posts in the pipeline. Suffice to say, embeddings can help us make progress on all 3 of the "intractable" challengs of TW mentioned here: https://technicalwriting.dev/strategy/challenges.html https://technicalwriting.dev/strategy/challenges.html
- jacobr1 1y ago
  Thanks for sharing regardless. It was a good overview for those less familiar with the material.
- kaycebasques 1y ago
  Also re: a direct application I forgot to mention this: https://www.tdcommons.org/dpubs_series/8057/ https://www.tdcommons.org/dpubs_series/8057/ (It finally published last week after being in review purgatory for months)
sansseriff 1y ago
It would be great to semantically search through literature with embeddings. At least one person I know if is trying to generate a vector database of all arxiv papers. The big problem I see is attribution and citations. An embedding is just a vector. It doesn't contain any citation back to the source material or modification date or certificate of authenticity. So when using embeddings in RAG, they only serve to link back to a particular page of source material. Using embeddings as links doesn't dramatically change the way citation and attribution are handled in technical writing. You still end up citing a whole paper or a page of a paper. I think GraphRAG [1] is a more useful thing to build on for technical literature. There's ways to use graphs to cite a particular concept of a particular page of an academic paper. And for the 'citations' to act as bidirectional links between new and old scientific discourse. But I digress [1] https://microsoft.github.io/graphrag/ https://microsoft.github.io/graphrag/
- kaycebasques 1y ago
  IMO, for technical writing, citing a page or section within a page is usually good enough. I rarely need to cite a particular concept. But I've never even thought of the possibility of more granular concept-level citations and will definitely be pondering it more!

lblume 1y ago

Semantic search seems like a more promising usecase than simple related articles. A big problem with classical keyword-based search is that synonyms are not reflected at all. With semantic search you can search for what you mean, not what words you expect to find on the site you are looking for.

kgeist 1y ago
In my benchmarks for a service which is now running in production, hybrid search based on both keywords and embeddings performed the best. Sometimes you need exact keyword matches; other times, synonyms are more useful. Hybrid search combines both sets of results into a single, unified set. OpenSearch has built-in support for this approach.
PaulHoule 1y ago
A case related to that is "more like this" which in my mind breaks down into two forks: (1) Sometimes your query is a short document. Say you wanted to know if there were any patents similar to something you invented. You'd give a professional patent searcher a paragraph or a few paragraphs describing the invention, you can give a "semantic search engine" the paragraph -- I helped build one that did about as well as the professional using embeddings before this was cool. (2) Even Salton's early works on IR talked about "relevance feedback" where you'd mark some documents in your results as relevant, some as irrelevant. With bag-of-words this doesn't really work well (it can take 1000 samples for a bag-of-words classifier to "wake up") but works much better with embeddings. The thing is that embeddings are "hunchy" and not really the right data structure to represent things like "people who are between 5 feet and 6 feet tall and have been on more than 1000 airplane flights in their life" (knowledge graph/database sorts of queries) or "the thread that links the work of Derrida and Badiou" (could be spelled out logically in some particular framework but doing that in general seems practically intractable)
- ted_dunning 1y ago
  Regarding item (2), Salton did talk about this and many systems from the early TREC times implemented them. In my own work, I found that relevance feedback worked very well with only a few dozen judgements. See Chapter 7 starting on page 83 in my dissertation: https://arxiv.org/abs/1207.1847 https://arxiv.org/abs/1207.1847 Note also that the evaluation of Luduan in that chapter compared against state-of-the-art (for the time) back-of-words systems but also an early word embedding retrieval system (Convectis from HNC).

ncruces 1y ago

Previous discussion: https://news.ycombinator.com/item?id=42013762 https://news.ycombinator.com/item?id=42013762

dang 1y ago
Thanks! Macroexpanded: Embeddings are underrated - https://news.ycombinator.com/item?id=42013762 https://news.ycombinator.com/item?id=42013762 - Nov 2024 (176 comments)

kaycebasques 1y ago

Hello, I wrote this. Thank you for reading! The post was previously discussed 6 months ago: https://news.ycombinator.com/item?id=42013762 https://news.ycombinator.com/item?id=42013762 To be clear, when I said "embeddings are underrated" I was only arguing that my fellow technical writers (TWs) were not paying enough attention to a very useful new tool in the TW toolbox. I know that the statement sounds silly to ML practitioners, who very much don't "underrate" embeddings. I know that the post is light on details regarding how exactly we apply embeddings in TW. I have some projects and other blog posts in the pipeline. Short story long, embeddings are important because they can help us make progress on the 3 intractable challenges of TW: https://technicalwriting.dev/strategy/challenges.html https://technicalwriting.dev/strategy/challenges.html

rybosome 1y ago
Thanks for the write-up! I’m curious how you found the quality of the results? This gets into evals which ML folks love, but even just with “vibes” do the results eyeball as reasonable to you?
- kaycebasques 1y ago
  By results I assume that you're asking about the related pages experiment? The results were definitely promising. A lot of the calculated related pages were totally reasonable. E.g. if I'm reading https://www.sphinx-doc.org/en/master/development/html_themes/index.html https://www.sphinx-doc.org/en/master/development/html_themes... then it's very reasonable to assume that I may also be interested in https://www.sphinx-doc.org/en/master/usage/theming.html https://www.sphinx-doc.org/en/master/usage/theming.html
sgbeal 1y ago
> I know that the post is light on details regarding how exactly we apply embeddings in TW. More significantly, after having read the first 6 or 8 paragraphs, i still have no clue what an "embedding" is. From the 3rd paragraph: > Here’s an overview of how you use embeddings and how they work. But no mention of what they are (unless perhaps it's buried far deeper in the article).

petesergeant 1y ago

I wrote an embeddings explainer a few days ago if anyone is interested: https://sgnt.ai/p/embeddings-explainer/ https://sgnt.ai/p/embeddings-explainer/ Very little maths and lots of dogs involved.

podgietaru 1y ago

I wrote a blog post about embedding - and a sample application to show their uses. https://aws.amazon.com/blogs/machine-learning/use-language-embeddings-for-zero-shot-classification-and-semantic-search-with-amazon-bedrock/ https://aws.amazon.com/blogs/machine-learning/use-language-e... https://github.com/aws-samples/rss-aggregator-using-cohere-embeddings-bedrock https://github.com/aws-samples/rss-aggregator-using-cohere-e... I really enjoy working with embedding. They’re truly fascinating as a representation of meaning - but also a very cheap and effective way to perform very cheap things like categorisation and clustering.

btbuildem 1y ago
How would you approach using them in a specialized discipline (think technical jargon, acronyms etc) where traning a model from scratch is practically impossible because everyone (customers, solution providers) fiercely guards their data? A generic embedding model does not have enough specificity to cluster the specialized terms or "code names" of specific entities (these differ across orgs but represent the same sets of concepts within the domain). A more specific model cannot be trained because the data is not available. Quite the conundrum!
- minimaxir 1y ago
  You can fine-tune existing embedding models.

tyho 1y ago

> The 2D map analogy was a nice stepping stone for building intuition but now we need to cast it aside, because embeddings operate in hundreds or thousands of dimensions. It’s impossible for us lowly 3-dimensional creatures to visualize what “distance” looks like in 1000 dimensions. Also, we don’t know what each dimension represents, hence the section heading “Very weird multi-dimensional space”.5 One dimension might represent something close to color. The king - man + woman ≈ queen anecdote suggests that these models contain a dimension with some notion of gender. And so on. Well Dude, we just don’t know. nit. This suggests that the model contains a direction with some notion of gender, not a dimension. Direction and dimension appear to be inextricably linked by definition, but with some handwavy maths, you find that the number of nearly orthogonal dimensions within n dimensional space is exponential with regards to n. This helps explain why spaces on the order of 1k dimensions can "fit" billions of concepts.

drc500free 1y ago
Is this because we can essentially treat each dimension like a binary digit, so we get 2^n directions we can encode? Or am I barking up totally the wrong tree?
- emaro 1y ago
  Basically, but it gets even better. If you allow directions of 'meaning' do wiggle a little bit (say, between 89 and 91 degrees to all other directions), you get a lot more degrees of freedom. In 3 dimensions, you still only get 3 meaningful directions with that wiggle-freedom. However in high-dimensional spaces, this small additional freedom allows you to fit a lot more almost orthogonal directions than the number of strictly orthogonal ones. That means in a 1000-dimensional space you can fit a huge number >> 1000 of binary concepts.
kaycebasques 1y ago
Oh yes, this makes a lot of sense, thank you for the "nit" (which doesn't feel like a nit to me, it feels like an important conceptual correction). When I was writing the post I definitely paused at that part, knowing that something was off about describing the model as having a dimension that maps to gender. As you said, since the models are general-purpose and work so well in so many domains, there's no way that there's a 1-to-1 correspondence between concepts and dimensions. I think your comment is also clicking for me now because I previously did not really understand how cosine similarity worked, but then watched videos like this and understand it better now: https://youtu.be/e9U0QAFbfLI https://youtu.be/e9U0QAFbfLI I will eventually update the post to correct this inaccuracy, thank you for improving my own wetware's conceptual model of embeddings

jbellis 1y ago

Great to see embeddings getting some love outside the straight-up-ML space! I had a non-traditional use case recently, as well. I wanted to debounce the API calls I'm making to gemini flash as the user types his instructions, and I decided to try a very lightweight embeddings model, light enough to run on CPU and way too underpowered to attempt vector search with. It works pretty well! https://brokk.ai/blog/brokk-under-the-hood https://brokk.ai/blog/brokk-under-the-hood

stefanka 1y ago

I like that this looks like a very ethical and "fair" use of the LLM technology

nameless912 1y ago
Yeah, the ethics around _training_ models that generate embeddings is still suspect to me, but the use of embeddings as a cheap, efficient way to provide semantic similarity seems very valuable. I've started dipping my toes in doing real, honest-to-goodness "machine learning" at work and it's mostly involved having OpenAI create embeddings for support logs my team generates, and we're starting to get value out of being able to cluster certain types of issues together, which I'm excited by. But this kind of stuff is truly augmentative: representing complex ideas in easily-searchable vector spaces, making connections in datasets too vast for humans to comb through alone, that's actual value.

minimaxir 1y ago

> I don’t know. After the model has been created (trained), I’m pretty sure that generating embeddings is much less computationally intensive than generating text. An embedding is generated after a single pass through the model, so functionally it's the equivalent of generating a single token from an text generation model.

energy123 1y ago
I might be wrong but aren't embedding models usually bidirectional and not causal, so the attention mechanism itself is more expensive.
- breadislove 1y ago
  yes exactly
- minimaxir 1y ago
  It depends on the architecture (you very well can convert a decoder-only causal model to an embeddings model, e.g. Qwen/Mistral), but it is true the traditional embeddings models such as a BERT-based one are bidirectional, although unclear how much more compute that inherently requires. Compare to ModernBERT, which uses more modern techniques and is still bidirectional, but it is very very speedy. https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert
kaycebasques 1y ago
Will update the post to capture that interesting tidbit, thanks And also, thanks for this post! https://minimaxir.com/2025/02/embeddings-parquet/ https://minimaxir.com/2025/02/embeddings-parquet/

jasonjmcghee 1y ago

Another very cool attribute of embeddings and embedding search is that they are resource cheap enough that you can perform them client side. ONNX models can be loaded and executed with transformer.js https://github.com/huggingface/transformers.js/ https://github.com/huggingface/transformers.js/ You can even build and statically host indices like hnsw for embeddings. I put together a little open source demo for this here https://jasonjmcghee.github.io/portable-hnsw/ https://jasonjmcghee.github.io/portable-hnsw/ (it's a prototype / hacked together approximation of hnsw, but you could implement the real thing) Long story short, represent indices as queryable parquet files and use duckdb to query them. Depending on how you host, it's either free or nearly free. I used Github Pages so it's free. R2 with cloudflare would only cost the size what you store (very cheap- no egress fees).

qq99 1y ago
I was wondering about this. I was hesitant to add embedding-based search to my app because I didn't want to incur the latency to the embedding API provider blocking every search on initial render. Granted, you can cache the embeddings for common searches. OTOH, I also don't want to render something without them, perform the embedding async, and then have to reify the results list once the embedding arrives. Seems hard to sensibly do that from a UX perspective. To render locally, you need access to the model right? I just wonder how good those embeddings will be compared to those from OpenAI/Google/etc in terms of semantic search. I do like the free/instant aspect though
- jasonjmcghee 1y ago
  checkout MTEB (https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard) many of the open source ones are actually _better_. I've had a particularly good experiences with nomic, bge, gte, and all-MiniLM-L6-v2. All are hundreds of MB (except all-minilm which is like 87MB)
  - simonw 1y ago
    I love all-MiniLM-L6-v2 - 87MB is tiny enough that you could just load it into RAM in a web application process on a small VM. From my experiments with it the results are Good Enough for a lot of purposes. https://simonwillison.net/2023/Sep/4/llm-embeddings/#embedding-in-bulk https://simonwillison.net/2023/Sep/4/llm-embeddings/#embeddi...

charcircuit 1y ago

How are they underrated when they have been been used by the top sites for over a decade? The author doesn't really explain why he thinks they are underrated despite them being behind almost every search and recommendation users receive on their computers.

minimaxir 1y ago
Underrated is more a relative term, and embeddings are definitely underrated to all the other uses for the LLM boom.
- charcircuit 1y ago
  Embeddings were popularized preLLM. They didn't come from the LLM boom.
esafak 1y ago
The author is addressing technical writers, not engineers. https://news.ycombinator.com/item?id=43964290 https://news.ycombinator.com/item?id=43964290

daxfohl 1y ago

I wonder if this could be used to find redundant code

milindsoni 1y ago

[dead]

milindsoni 1y ago

I was using transformer.js to generate and use embeddings with small models in the browser itself, its quite useful to implement any kind of semantic search.

breadislove 1y ago

embeddings and consequently retrieval is probably the most overlooked topic in this whole AI thing. great to see people tinkering with it

adefa 1y ago

Here’s a CLI I’m experimenting with https://github.com/TrevorS/rhizome https://github.com/TrevorS/rhizome that indexes local repos with Tree‑sitter, stores ONNX embeddings in SQLite, and answers semantic queries offline; for example, `rhizome search "pull apart"` surfaces relevant snippets across projects: rhizome search --limit 2 "pull apart" Model already exists at \~/.rhizome/models/bge-small-en-v1.5.onnx \~/Projects/rhizome/src/chunking.rs:458\:fn rust\_no\_structural\_items\_fallback() { \~/Projects/rhizome/src/lib.rs:2\:pub mod chunking;

ubj 1y ago

> Because we always get back the same amount of numbers no matter how big or small the input text, we now have a way to mathematically compare any two pieces of arbitrary text to each other. I think there needs to be some more clarification here. Hash functions also return the same sized output no matter how big or small the input text. However, mathematically comparing two hashes is going to have a much different meaning than mathematically comparing two embeddings. I'd recommend emphasizing that embeddings are training dependent--the quality of comparison will depend on the quality and type of training used to produce the embedding. There isn't some single "universal embedding" that allows for meaningful comparison of arbitrary text.

kaycebasques 1y ago
Thanks for the feedback. I've been reading the research papers behind how these models are created e.g. Gecko [1] and NV-Embed [2] and Gemini Embedding [3] so I'm starting to grok what you're getting at and will find a way to make the post more accurate on this front [1] https://arxiv.org/abs/2403.20327 https://arxiv.org/abs/2403.20327 [2] https://arxiv.org/abs/2405.17428 https://arxiv.org/abs/2405.17428 [3] https://arxiv.org/abs/2503.07891 https://arxiv.org/abs/2503.07891

bawolff 1y ago

> I could tell you exactly how I think we might advance the state of the art in technical writing with embeddings, but where’s the fun in that? You now know why they’re such an interesting and useful new tool in the technical writer toolbox… go connect the rest of the dots yourself! Wow, that's bold. I guess "good" technical writing no longer includes a thesis statement. Seriously though, why would this be useful for technical writing? Sure you could make some similar pages widget however i dont think i've ever wanted that when reading technical docs, let alone writing them.

simonw 1y ago
Related documents aside, technical documentation benefits from really great search. Embeddings are a _very_ useful tool for building better search - they can handle "fuzzy" matches, where a user can say things like "that feature that lets me run a function against every column of data" because they can't remember the name of the feature. With embeddings you can implement a hybrid approach, where you mix both keyword search (still necessary because embeddings can miss things that use jargon they weren't trained on) and vector similarity search. I wish I had good examples to point to for this!
- kaycebasques 1y ago
  In-site search is super important. I suspect that many docs maintainers don't realize how heavily it's used. Many docs sites don't even track in-site search queries! One of the things I love about Sphinx is that it has a decent, client-side, JS-powered offline search. I recently hacked together a workflow for making it search-as-you-type [1]. jasonjmcghee's comment [2] has got me pondering whether we can augment it with transformer.js embeddings. [1] https://github.com/orgs/sphinx-doc/discussions/13222 https://github.com/orgs/sphinx-doc/discussions/13222 [2] https://news.ycombinator.com/item?id=43964913 https://news.ycombinator.com/item?id=43964913
kaycebasques 1y ago
> I guess "good" technical writing no longer includes a thesis statement. Thesis is outlined in the second paragraph: > What embeddings offer to technical writers is the ability to discover connections between texts at previously impossible scales. I think it's fair, however, to say that this post is ineffective because it does not provide concrete examples of the thesis in action. My only excuse is that I never intended for this to be a standalone post but life got in the way (in the best possible way!) https://news.ycombinator.com/item?id=43964584 https://news.ycombinator.com/item?id=43964584 > why would this be useful for technical writing? You're not going to like this answer, because it's also vague. There are 3 intractable challenges in technical writing. Embeddings can help us make progress on all 3: https://technicalwriting.dev/strategy/challenges.html https://technicalwriting.dev/strategy/challenges.html See also https://www.tdcommons.org/dpubs_series/8057/ https://www.tdcommons.org/dpubs_series/8057/

jas8425 1y ago

If embeddings are roughly the equivalent of a hash at least insofar as they transform a large input into some kind of "content-addressed distillation" (ignoring the major difference that a hash is opaque whereas an embedding has intrinsic meaning), has there been any research done on "cracking" them? That is, starting from an embedding and working backwards to generate a piece of text that is semantically close by? I could imagine an LLM inference pipeline where the next token ranking includes its similarity to the target embedding, or perhaps instead the change in direction towards/away from the desired embedding that adding it would introduce. Put another way, the author gives the example: > embedding("king") - embedding("man") + embedding("woman") ≈ embedding("queen") What if you could do that but for whole bodies of text? I'm imagining being able to do "semantic algebra" with whole paragraphs/articles/books. Instead of just prompting an LLM to "adjust the tone to be more friendly", you could have the core concept of "friendly" (or some more nuanced variant thereof) and "add" it to your existing text, etc.

luke-stanley 1y ago
"starting from an embedding and working backwards to generate a piece of text that is semantically close by?" Apparently this is called embedding inversion and Universal Zero-shot Embedding Inversion https://arxiv.org/abs/2504.00147 https://arxiv.org/abs/2504.00147 Going incrementally closer and closer to the target with some means to vary seems to be the most general way, there are lots of ways to be more optimal though. Image diffusion with CLIP embeddings and such is kinda related too.
- luke-stanley 1y ago
  I meant to say: Apparently this is called "embedding inversion", and that "Universal Zero-shot Embedding Inversion" is a related paper that covers a lot of the basics. Recently I learned that a ArXiv RAG agent by ArXiv Labs is a really cool way for people wanting to find out about research: https://www.alphaxiv.org/assistant https://www.alphaxiv.org/assistant Though I had ran into "inversion" before, the AlphaXiv Assistant introduced me to "embedding inversion".
jerjerjer 1y ago

computerex 1y ago

This article really rubbed me the wrong way. > I could tell you exactly how I think we might advance the state of the art in technical writing with embeddings, but where’s the fun in that? You now know why they’re such an interesting and useful new tool in the technical writer toolbox… go connect the rest of the dots yourself! I read the article because of the title, only to find the above.

kaycebasques 1y ago
You're not alone. I've gotten lots of feedback that the post buries the lede [1] and is ineffective because it doesn't provide more concrete examples. I will fix it eventually, I promise! See here for context: https://news.ycombinator.com/item?id=43964584 https://news.ycombinator.com/item?id=43964584 [1] It's perhaps not even appropriate to say "buried the lede" because that implies the lede is dug back up at some point, whereas this post buries the lede and then forgets where the lede was buried!!

archeantus 1y ago

Great read. Brain started fry when it talked about infinite dimensions but then I imagined it looking like the black hole scene at the end of interstellar and that helped a bit (though it’s not like that scene made any sense either)

jonathanrmumm 1y ago

Embeddings are a new jump to universality, like the alphabet or numbers. https://thebeginningofinfinity.xyz/Jump%20to%20Universality https://thebeginningofinfinity.xyz/Jump%20to%20Universality

tucnak 1y ago

Surprised they never mentioned jina.ai models such as jina-embeddings-v3 at 8K context and outperforming most "contenders" in the MTEB or jina-clip-v2 (multimodal) or "late chunking," also known as mean pooling: https://jina.ai/news/late-chunking-in-long-context-embedding-models/ https://jina.ai/news/late-chunking-in-long-context-embedding... The article feels incomplete

kaycebasques 1y ago
Wasn't on my radar! Thanks for the pointer. I'll look into it. It's hard to keep up with all the embedding models out there.

gnarlouse 1y ago

The article did not go the direction that I imagined, which I loved. What if we could use embedding spaces to clean up documentation to be more explicit and direct? The biggest thing that stuck with me from my technical writing class was “get to the point”. Finding ways to narrow word choices in an almost 1984’esque ingSoc seems appropriate? Ish?

energy123 1y ago
Neural nets aren't invertible so I don't see how embeddings can help reduce verbosity. It's not like we could take the embedding of a piece of verbose text and figure out less verbose phrasings that have an embedding with high similarity without using some expensive search. At which point it seems better to just use a standard LLM for that.
- kaycebasques 1y ago
  Yes, I have experimented with using ML for style concerns such as consistent capitalization and verbosity and can confirm: no need for embeddings here. You can get the job done with only a text generation model. I actually can't visualize how an embeddings-powered implementation would work here

simianwords 1y ago

I don't think any of the current consumer LLM tools use embeddings for web search. Instead they do it at the text level. The evidence for this is the COT summary with ChatGPT - I have seen something where the the LLM uses quotes to grep on the web. Embeddings seem good in theory but in practice its probably best to ask an LLM to do a deep search instead by giving it instructions like "use synonyms and common typos and grep". Does any one know any live example of a consumer product using embeddings?

zhobbs 1y ago
My understanding is that modern search engines are using embeddings / vector search under the hood. So even if LLM's aren't directly passing a vector to the search engine, my assumption is that the search engine is converting to a vector and searching. "You interact with embeddings every time you complete a Google Search" from https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings https://cloud.google.com/vertex-ai/generative-ai/docs/embedd...
- simianwords 1y ago
  Fair, and maybe key point here is that it uses embeddings to help with the search results along with many manual heuristics in place. I hardly think google search works just by dumping embeddings then doing KNN's and calling it a day.
dcre 1y ago
I believe they use the LLMs to generate a set of things to search for and then run those through existing search engines, which are totally opaque and use whatever array of techniques SOTA search engines use. They are almost certainly not "grepping" the internet.
- simianwords 1y ago
  yes that's what i meant thanks for clarifying. the grepping part is definitely done at least in spirit where the COT includes quotes. if i were searching for top 10 cars that are manufactured in South America for example, the COT might show: "Brazil" car manufacture This forces Brazil to be included in the keywords, at least that's how google (used to?) works.

antirez 1y ago

For a coincidence two hours ago I posted a video on my channel where it can be seen how king-man+woman also works with CLIP image embeddings. This may be obvious for people that worked with CLIP extensively, or that tried embeddings math in other embedding spaces, but it really surprised me. https://youtu.be/r6TJfGUhv6s?si=wG6h1kdigiPrNFdk https://youtu.be/r6TJfGUhv6s?si=wG6h1kdigiPrNFdk Video is in English but please pardon my and my friend Italian accents...

gweinberg 1y ago

I don't understand why some people consider the concept of many dimensions to be so mysterious. It's numbers specifying something like degrees of freedom. If I wanted to specify the position of my body sitting at my desk, I might say use two angles to specify the what is happening at each joint, and so would probably need a couple hundred to fully specify my position. A human being probably could not look at the numbers and see "he's sitting at a desk", but I don't see the conceptual difficulty".

JohnKemeny 1y ago
You're right that higher dimensions are just more degrees of freedom, and mathematically it's just more numbers. But what's counterintuitive is how geometry behaves differently as dimensions grow. Things like distance, volume, and angles don’t scale the way we expect. For example, in high dimensions, almost all the volume of a sphere concentrates near its surface, and random vectors tend to be nearly orthogonal—something that rarely happens in 2D or 3D. These effects matter in practice. In high-dimensional spaces like word embeddings, even unrelated points can seem equidistant, making basic tasks like clustering or similarity search much harder. So it's not that higher dimensions are mysterious per se, but that they defy the spatial intuitions we've developed from living in three.

mkasberg 1y ago

I started using embeddings to find similar posts on my blog about a year ago. The results were awesome: https://www.mikekasberg.com/blog/2024/04/23/better-related-posts-in-jekyll-using-ai.html https://www.mikekasberg.com/blog/2024/04/23/better-related-p... Such a simple tool to implement with so much power in certain situations!

ColinEberhardt 1y ago

Agreed, embeddings are a very useful tool. Started using them for article recommendations on our blog a few years ago, with a measured increase in engagement: https://blog.scottlogic.com/2022/02/23/word-embedding-recommendations.html https://blog.scottlogic.com/2022/02/23/word-embedding-recomm...

wussboy 1y ago
Where there any other metrics that improved besides engagement? Or is that your only metric? If it is your only metric what does it really prove?

daturkel 1y ago

I gave a very short talk (now a blog post) about embeddings and we use them to bridge the gap between human notions of understanding and digital representations. It might be of interest to people who enjoyed this post: https://danturkel.com/2025/03/10/ignite-machine-understanding-vectors.html https://danturkel.com/2025/03/10/ignite-machine-understandin...

Lerc 1y ago

There's a talk by Leland McInnes (on my phone so can't find it rn) where he shows how to make quite effective embeddings through off-the-shelf analytical processes rather than using a large trained model.

kanak 1y ago
Is it this one? https://www.youtube.com/watch?v=Z6s7PrfJlQ0&t=3084s https://www.youtube.com/watch?v=Z6s7PrfJlQ0&t=3084s > MIA Meeting: Low dimensional embeddings of words and documents (and how they might apply to single-cell data) Leland McInnes Tutte Institute for Mathematics and Computing Over the last decade the field of Natural Language Processing (NLP) has been overtaken by neural networks and deep learning. The latest models and algorithms, from word2vec to Google’s Universal Sentence Encoder and BERT, can perform seemingly magical feats and provide powerful tools to understand and analyze text documents. This talk will seek to pick apart word and document embedding techniques from NLP, removing the neural networks and instead working with linear algebra and dimension reduction on large sparse matrices of counts that is very similar to a large amount of single-cell data. Such an approach can achieve results on par with neural network approaches, but are both simpler to understand the inner workings of, and are generalizable to much more diverse domains than NLP, including, hopefully, single-cell research. In particular all we really need to apply these kinds of techniques are large sparse matrices of counts and some notion of locality or cooccurrence of the feature columns of that matrix. The goal of this talk is to hopefully open up some of the ideas that have been so effective in NLP and make them usable and accessible to a wider audience working with more diverse types of data, and initiate a conversation about what is required to make such techniques useful for the single-cell community.
- Lerc 1y ago
  That looks like it. Been meaning to give it a try on a few different data types. Not enough hours in the day.

paulnovacovici 1y ago

Completely agree. I’ve been experimenting with embeddings by building Recallify, primarily to help me quickly retrieve obscure things I’ve read online. Even at just 1024 dimensions, it’s impressive how effectively embeddings capture and surface ideas based purely on semantic similarity, rather than exact keyword recall (which I’m pretty terrible at remembering). It’s been a game changer for turning fuzzy mental concepts into actionable insights. Beta testing an iOS app for it if anyone is interested: https://recallify.app/ https://recallify.app/

eabeezxjc 1y ago

please look at rust package.The amount is striking and is constantly increasing not decreasing!

deleted 1y ago

[deleted]

joaoli131 1y ago

Embeddings are still underrated—even in RAG. Legal text is deeply hierarchical and full of pointers (“Art. 5 CF”, “see Art 34”). One vector per article leaves too much on the table. Things that moved the needle for us: – *Multi-layer embeds* vectors for every paragraph and every structural level (chapter → book). Retriever picks the right granularity. (arXiv:2411.07739) – *Propositional queries* strip speech-act fluff (“could you please…”) before embedding. Similarity + top-k recall jump. (arXiv:2503.10654) – *Poly-vector retrieval* two vectors per norm—one for content, one for the label/nickname. Handles “what does the CDC say?” and internal cross-refs. (arXiv:2504.10508) *TL;DR* If your corpus has hierarchy or aliases, stop thinking “one doc = one embedding.” Plenty of juice to squeeze before heavier tricks. [1] https://arxiv.org/abs/2411.07739 https://arxiv.org/abs/2411.07739 [2] https://arxiv.org/abs/2503.10654 https://arxiv.org/abs/2503.10654 [3] https://arxiv.org/abs/2504.10508 https://arxiv.org/abs/2504.10508

nakedneuron 1y ago

Question for the experts: As embeddings in question reflect content is there another vector for style? I was wondering if stylometry research would profit from embeddings becoming more easily accessible than ever. I dug my head into this matter some time ago and believe this would be the right tool.

n3t 1y ago
If you have a dataset for which you know which elements have similar/same style, you can create your own embeddings for style.
navar 1y ago
Embeddings as a tool have been around for longer than LLMs. They were (and are) ubiquitous in, e.g., recommender systems. It sounds maybe this would be more in-line with what you are looking for. In this case, check out https://github.com/benfred/implicit https://github.com/benfred/implicit - I have used it in the past with great success.

navar 1y ago

[dead]

ADAMOsong 1y ago

I HAVE MY EX BACK TO ME WITH THE SOLUTION FROM Dr. SHAUN MURPHY. I came in contact with Dr. SHAUN MURPHY. Through a testimony written about him, I have also encountered many testimonies about how he has been helping others with their life. To get back with an ex is one of the innermost feelings many people would love to experience, especially as those memories with our ex always cloud our mind when someone else does some of those things our ex used to do. I was a single parent for almost 6 years, and though my ex-husband was far away from my kids, I still wish someday he would return to me. I encountered this spell doctor, Dr. SHAUN MURPHY. It helped me with my wishes, and I am happy to say I am back with my ex-husband. He assured me that the result manifested within 48 hours, and our love and connection were restored amazingly. If you want to get back with your ex successfully, contact this great spell caster with HIS EMAIL: doctormurphhy@yahoo.com‎

mijoharas 1y ago

So, just the other day I was looking for a "grep" like tool that'll allow me to find files based on embedding similarity from a set of local files. I'm assuming it'll have an indexing process, and then a little vector db locally, and then when I grep it can search that and return me results. Does anything like that exist? I couldn't find anything like it.