12 ms·
Embeddings: What they are and why they matter
- simonw 3y agoIf you've been wondering why there's so much hype around vector databases at the moment this article should help explain that too - embeddings and vector databases both occupy the same space.
- williamcotton 3y agoThe same… vector space? I’ll show myself out…
- quickthrower2 3y agoIs it because with transformers and LLMs the embeddings are more powerful than they have been before (they can seem to understand more)?
- jaggederest 3y agoBroadly, as I understand it, the way the embeddings are generated "needs" LLMs to produce the superior results you'd see with e.g. OpenAI's text-embedding-ada-002 - we've been working with vectors at least as long as I've been programming, but until recently they didn't mean anything particular. LLMs let the vectors have semantic meaning and relate similar conceptual text, instead of (as in the article) using gzip to generate vectors where the similarity is entirely textual, i.e. similar words and passages and characters, even if they have different meanings. So ultimately they're not at all new (embedding data into a hamming space to compare it to other data), but the underlying meaning is much more useful in a world where you can peek into the internal state of a LLM to generate them.
- tudorw 3y agoHas anyone used PHATE for visualising? "PHATE is a dimensionality reduction and visualization tool designed to preserve both local and global data structure."
- px1999 3y agoThank you for writing this, it was really informative and interesting (as someone with little ML background).
- hibbelig 3y agoI've come across articles like this and they usually tell you to make use of an LLM. Is it possible to kind of compute a language model (maybe not that large) based on a given corpus?
- lordgrenville 3y ago> dot_product = sum(x * y for x, y in zip(a, b)) Wait, why would you do this and not use vectorised numpy operations? > I actually got ChatGPT to write all of my different versions of cosine similarity Ah...
- simonw 3y agoTwo reasons. First, when I'm trying to explain stuff to people I find numpy syntax gets in the way. And second, numpy isn't the lightest dependency. I use it when I need the performance but I don't like to default to it.
- lordgrenville 3y agoMakes sense. And sorry for the snark - I enjoyed the post, and generally enjoy your writing as well.
- toxik 3y agoI disagree that your version is more readable. If you don’t know linear algebra, the code is inscrutable, and if you do, dot(x,y)/norm(x)/norm(y) is about as close to the math as you can get.
- drcongo 3y agoMy linear algebra is terrible, but I understand what's happening in Simon's example pretty well.
- toxik 3y agoI sincerely doubt you possess any kind of geometric understanding of what is going on in that code just from reading the first principles variant of cosine similarity.
- forgotpwd16 3y ago>If you don’t know linear algebra, the code is inscrutable Whether you do or don't know linear algebra, code is self-explanatory. Dot product is just the sum of element-wise products. But what the heck are dot and norm to someone that doesn't.
- simonw 3y agoSince publishing this I've found a few additional resources that are really useful for understanding embeddings at a lower level (my article is deliberately very high level and focuses mainly on their applications). Cohere's Text Embeddings Visually Explained: https://txt.cohere.com/text-embeddings/ https://txt.cohere.com/text-embeddings/ The Tensorflow Embedding Projector tool: https://projector.tensorflow.org/ https://projector.tensorflow.org/ What are embeddings? by Vicki Boykis is worth checking out as well: https://vickiboykis.com/what_are_embeddings/ https://vickiboykis.com/what_are_embeddings/ Actually I'll add those as "further reading" at the bottom of the page.
- forgingahead 3y agoSimon, just wanted to say thanks for all the great content and writings you've been putting out - it's been super helpful to help digest a lot of the fast developments in this space. Always looking forward to the next one!
- simonw 3y agoThanks for saying that!
- ColinEberhardt 3y agoI had exactly the same idea a while back: https://blog.scottlogic.com/2022/02/23/word-embedding-recommendations.html https://blog.scottlogic.com/2022/02/23/word-embedding-recomm... Using embeddings I increased engagement with related articles. Personally I think embeddings are a powerful tool that are somewhat overlooked. They can be used to navigate between documents (and excerpts) based on similarities - or conversely find unique content. All without worrying about hallucinations. In other words, they are quite ‘safe’
- mike_hearn 3y ago> All without worrying about hallucinations. In other words, they are quite ‘safe’ Within limits, yes. In some use cases a vector notion of similarity isn't always ideal. For example, in the article "France" and "Germany" are considered similar. Yes, they are, but if you're searching for stuff about France then stuff about Germany is a false positive. Embeddings can also struggle with logical opposites. Hot/cold are in many senses similar concepts, but they are also opposites. Finding the opposite of what you're searching for isn't always helpful. I wouldn't say embeddings are overlooked exactly? Right now it feels like man+dog are building embedding based search engines. The next frontier is probably going to be balancing conventional word based approaches with embeddings to really maximize result quality, as sometimes you want "vibes" and sometimes you want control.
- zachthewf 3y agoI'm curious to know more about your experience with ImageBind, and what applications for multimodal embeddings are most exciting to you.
- simonw 3y agoSo far I've only just played with it - I don't even fully understand what some of those file formats are! I got it to compare an image to an audio file which was pretty neat. I need to dig in more and see what kind of useful things I can use it for.
- tayo42 3y agoIve been playing with embeddings lately for document search based off queries/questions. this makes it seem like they work great, but it hasn't been very smooth for me. I kept running into people recommending against using them for long documents? Is openai better then the models sentenance-transformers uses? I found some recommendations to average together the embeddings of parts. I guess its cutting edge-ish still, a lot still feels like you can have a cool demo quickly, but something reliable and accurate is a lot of work?
- simonw 3y agoOpenAI's embedding model works up to 8,000 tokens. The sentence-transformers ones are mostly smaller than that, though there's a new model that just came out that can handle 8,000: https://huggingface.co/jinaai/jina-embeddings-v2-base-en https://huggingface.co/jinaai/jina-embeddings-v2-base-en People tend to "chunk" larger documents - the chunking strategy that's best is very dependent on what you are using the embeddings for. I've found it frustratingly hard to find really good guidance as to chunking strategies. I've had good results for Q&A chunking my blog content up into paragraph sized chunks, as described here: https://simonwillison.net/2023/Oct/23/embeddings/#answering-questions-with-retrieval-augmented-generation https://simonwillison.net/2023/Oct/23/embeddings/#answering-... - but I'm not ready to say that's a universally good practice.
- CityOfThrowaway 3y agoI'm also curious about how to think about chunking for Doc-to-Doc similarity. Say I have an input document that I want to use as my search query and a database of potential results documents. It's not clear to me that I should simply embed both the input document and each of the results documents. If the documents contain a variety of ideas, I'd be nervous they wind getting embedded into some generic space. But I don't have any good ideas as to what type of chunk-match-and-aggregate strategy might work well here. Would love ideas from folks that have done stuff like this!
- deleted 3y ago[deleted]
- quartz 3y agoThis is a great zero to one reference. I built my own note taking ios app a little while back and adding embeddings to my existing fulltext search functionality was 1) surprisingly easy and 2) way more powerful than I initially expected. I knew it would work for things like if I search for "dog" I'll also get notes that say "canine", but I didn't originally realize until I played around with it that if I search for something like "pets I might like" I'll get all kinds of notes I've taken regarding various animals with positive sentiment. It was the first big aha moment for me. At the time I found Supabase's PR for their DocsGPT really helpful for example code: https://github.com/supabase/supabase/pull/12056 https://github.com/supabase/supabase/pull/12056
- nunodonato 3y agothats awesome! I'm building my own simple note-taking app powered by embeddings as well, finally I dont loose stuff and its easy to find :D
- quartz 3y agoRight?! Everyone thought I was crazy to do this vs using something off the shelf but having total control over my notes app has been incredible. I can tailor everything to my style of note taking vs dealing with the lowest common denominator feature set that tries to enable tons of use cases that I don’t need.
- nunodonato 3y agothats the whole fun of being able to code :)
- mhalle 3y agoI think your statement "adding to existing fulltext functionality" is subtly important: embeddings provide semantic search that complements traditional search algorithms. Specifically, many applications are heavily dependent on names or other proper nouns, often without much context. You might refer to your dog by name without explanation, and a particular embedding might not pick that up. Proper names (people, places, street names) may have outsized importance for anchoring personalized or domain-specific search, but modest generic language models won't know about them. Is there a specific way of dealing with this problem?
- Ozzie_osman 3y agoHaving played around with embeddings and built a few production use-cases with them.. They are awesome and enable a lot of cool applications. But, if you're building in a particular domain, you'll hit limits with any off-the-shelf embeddings model. One way to think about this is that the off-the-shelf embedding models have a lot of dimensions, but some of those dimensions may and some may not that matter for your application (classification, content similarity, clustering, etc). In other words, a vector may be close to another vector in dimensions you don't care about. I look forward to seeing better tooling and literature around fine-tuning embedding models.
- eldenring 3y agoFine-tuning an entire language model to solve this problem is like using a sledgehammer on a nail. We have had tools for this for years, for example just label some data and train an SVM on your embedding space for classification.
- layoric 3y agoFrom my understanding, embedding models are just one layer of a modern LLM that do the similarity part as described in the post. The translation of representing content as a part of a common vocabulary in a vector. My understanding is that embedding models can be fine tuned in isolation to relate a vocabulary entries to each other, so fine tuning a whole LLM is not required, and the required compute resources would be far smaller.
- ahoho 3y agosentence-transformers has reasonably good tooling around this
- 26fingies 3y agoAdmittedly i more or less skimmed and plan on going back over this tomorrow, but I dont see how these vectors are actually created. I get that I could use your llm tool or whatever, but that seems unsatisfactory. How is the sausage made? (or if thats explained can someone point me at the right place to look?)
- n2d4 3y agoIn essence, an embedding vector is (lossy) compression. Any compression could in theory be used to make such vectors, for example people have tried using gzip embeddings. Now on how to get a compression vector from an LLM, simplified: Most ML models are built from different layers, executed one after another. Some of the layers are bigger, some are smaller, but each has a defined in- and output. If a layer's input size is smaller than model's input size, that must mean (lossy) compression must have happened to get there. So, you just evaluate the LLM on whatever you want to embed, and take the activation at the smallest layer input, and that's your embedding vector. Not every compression vector makes for good semantic embeddings (which requires that two similar phrases are next to each other in the embedding space), but because of how ML models work, this tends to be the case empirically.
- optimalsolver 3y agoHow do you choose the size of the embedding vector? Can this be used to compress non-text sequences such as byte strings?
- WJW 3y ago1. Usually it's a multi-way tradeoff between how much data you want to use, how much compute you want to spend, how much time you have available, how much training data you have available and how accurate you want the embeddings to be. 2. Yes, but lossily. Some types of byte strings are such that it doesn't matter if you accidentally change a couple of bits, some types of byte strings cannot tolerate that at all without being hopelessly corrupted. This technique is not a magic card to surpass the limits imposed by information theory, it's "just" a more sophisticated dictionary for your compression algorithm.
- pseudosavant 3y agoThis is the most interesting thing I've read about in "AI" in quite a few months. I always wondered what embedding models were when I'd see lists, or curious why everyone is talking about vector DBs. This is immediately making me think about how I could apply this to a long running side project I have. It might make it practical to do useful clustering of user's data if every document has an embedding.
- benreesman 3y agoGreat article! I can’t remember the YouTuber’s name or what to search for, but one of the Pinecone people has great videos at an introductory-to-intermediate level that is both really accessible and well motivated by examples. Anyone remember?
- gk1 3y agoJames Briggs? He's awesome and you can find lots of his articles + videos here: https://www.pinecone.io/learn/ https://www.pinecone.io/learn/ https://www.youtube.com/@jamesbriggs https://www.youtube.com/@jamesbriggs
- benreesman 3y agoYup that’s exactly who I meant. I think his stuff is just great. Thanks for reminding me/us.
- heisenburgzero 3y agoWhy does the embeddings have linear properties such that you can use functions like cosine similarity to compare? It seems that after the signal going through so many non-linear activation layers, the linear properties should have been broken down / no guarantees. I wasn't able to find a good answer online.
- blackbear_ 3y agoBecause neural networks use dot products, which are just un-normalized cosine similarities, as the main way to compare and transform embeddings in their hidden layers. Therefore, it makes sense that the most important signals in the data arranged in latent space such that they are amenable to manipulations based on dot products
- montebicyclelo 3y agoLLM vectors do have decent linear properties already. But for document embedding purposes they are often further trained for retrieval via cosine similarity, which enhances this, e.g. see table 1 in [1], avg retrieval performancs using BERT goes up from 54 to 76 after fine-tuning for embeddings. [1] https://arxiv.org/pdf/1908.10084.pdf https://arxiv.org/pdf/1908.10084.pdf
- toxik 3y agoThe cosine similarity is not inherently better suited for linear properties whatever that means, it’s just the cosine of the angle between two vectors. If the vectors are unit length, then it’s just the projection of one to the other.
- ndriscoll 3y agoFor what it's worth, I wonder the same thing and think it's not as obvious as others suggest. e.g. if you have an autoencoder for a one-hot encoding, you're essentially learning a pair of nonlinear maps that approximately invert each other, and that map some high dimensional space to a low one. You could imagine that it could instead learn something like a dense bit packing with a QAM gray code[0]. In a one-hot encoding the dot product for similar tokens is zero, so your transformations can't be learning to preserve it. Somewhat naively, I might speculate that for e.g. sequence prediction, even if you had some efficient packing of space like that to try to maximally separate individual tokens, it's still advantageous to learn an encoding so that synonyms are clustered so that if there is an error, it doesn't cause mispredictions for the rest of the sequence. I suppose then the point is that the structure exists in the latent space of language itself, and your coordinate maps pull it back to your naive encoding rather than preserving a structure that exists a priori on the naive encoding. i.e. you can't do dot products on the two spaces and expect them to be related. You need to map forward into latent space and do the dot product there, and that defines a (nonlinear) measure of similarity on your original space. Then the question is why latent space has geometry, and there I guess the point is it's not maximally information dense, so the geometry exists in the redundancy. So perhaps it is obvious after all! [0] https://en.wikipedia.org/wiki/File:16QAM_Gray_Coded.svg https://en.wikipedia.org/wiki/File:16QAM_Gray_Coded.svg
- abricq 3y agoNot quite the same application, but in computer vision and visual SLAM algorithms (to construct a map of your surrounding using a camera) embeddings have become a de-facto method to perform place-recognition ! And it's very similar to this article. It is called "bag-of-word place recognition" and it really became the standard, used by absolutely every open-source library nowadays. The core idea is that each image is passed through a feature-extractor-descriptor pipeline and is 'embedded' in a vector containing the N top features. While the camera moves, a database of images (called keyframes) is created (images are stored as much-lower dimensional vectors). Again while the camera moves, all images are used to query the database, something like cosine-similarity is used to retrieve the best match from the vector database. If a match happened, a stereo-constraints can be computed betweeen the query image and the match, and the software is able to update the map. [1] is the original paper and here's the most famous implementation: https://github.com/dorian3d/DBoW2 https://github.com/dorian3d/DBoW2 [1]: https://www.google.com/search?client=firefox-b-d&q=Bags+of+Binary+Words+for+Fast+Place+Recognition+in+Image+Sequences https://www.google.com/search?client=firefox-b-d&q=Bags+of+B...
- politelemon 3y agoIs there a ready made search framework commercially speaking like an aws service or of the shelf database, for sites that will take all your articles and for a given article find all similar ones?
- m3at 3y agoDepends what you mean by taking all your articles. If it's scraping no, but if you provide text content and urls as key/values pairs yes. SageMaker and VertexAI are the AI services of AWS and GCP respectively, and they both offer embedding generation and vector databases (the two key pieces necessary for embedding search). There are a bunch of smaller companies offering vector search as a service too, example pinecone to name just one: https://www.pinecone.io/ https://www.pinecone.io/
- politelemon 3y agoThanks, will have a look. Yeah in this case I'd want to provide the text content. I also just found pgvector which seems like it could help? It has a similarity search.
- m3at 3y agoI would only recommend pgvector if you're already primarily relying on postgres and the scale is limited (<1M documents). It won't handle the part that generates the embeddings though. You could use cloud vendors if you're in one of their ecosystem, do it yourself [1] (but model serving can be tricky without prior experience in ML), or use some other service to generate embeddings [2]. Alternatively vespa cloud [3] offer both but… not the easiest to work with, it's tailored for businesses where search is a primary component. Feel free to shoot me an email (in profile) with your context if you have more questions, in case I can help [1] This model is a solid baseline if you're working with English text: https://huggingface.co/sentence-transformers/all-mpnet-base-v2 https://huggingface.co/sentence-transformers/all-mpnet-base-... [2] OpenAI's embeddings is probably the easiest to get started, and the API is straightforward. It's not the best performing embeddings for retrieval but good enough in some cases: https://platform.openai.com/docs/guides/embeddings/use-cases https://platform.openai.com/docs/guides/embeddings/use-cases [3] https://cloud.vespa.ai/ https://cloud.vespa.ai/
- politelemon 3y agoI'm trying to understand the clustering code but not doing too well. https://github.com/simonw/llm-cluster/blob/main/llm_cluster.py#L72 https://github.com/simonw/llm-cluster/blob/main/llm_cluster.... So does this take each row from the DB, convert to a numpy array (?), then uses an existing model called MiniBatchKMeans (?) to go over that array and generate a bunch of labels. Then add it to a dictionary and print to console.
- jamessb 3y agoYes - it uses the implementation of MiniBatchKMeans provided by the scikit-learn library. (I'd call this an "algorithm" rather than a "model" - it doesn't have any model weights learnt from a training dataset) For more details, see the pages in its user guide describing: * the K-Means algorithm: https://scikit-learn.org/stable/modules/clustering.html#k-means https://scikit-learn.org/stable/modules/clustering.html#k-me... * the Mini Batch variant of k-means: https://scikit-learn.org/stable/modules/clustering.html#mini-batch-kmeans https://scikit-learn.org/stable/modules/clustering.html#mini...
- FL33TW00D 3y agoHow can I possibly know which models have gamed the MTEB benchmark and which haven't? It's very hard to know which embedding models are objectively good now that everyone is gaming the benchmark.
- ramoz 3y agoTest with your own use case. Production ML means extensive custom testing suites, and even/often custom embeddings.
- jxmorris12 3y agoHow are people gaming the benchmark?
- wolfgangK 3y agoAbout words embeddings, the №1 example is the famous King - Man + Women = Queen This works nicely in the vector space but fails to make a visual impression when projected on 2 dimensions. Neither with ACP, nor MDS ot t-SNE in my experience : https://bhugueney.gitlab.io/test-notebooks-org-publish/jupyterlite/lab/?path=custom%2FWords-Embeddings.ipynb https://bhugueney.gitlab.io/test-notebooks-org-publish/jupyt... (← JupyterLite Notebook doing words embedding in the browser : don't try to run this on a smartphone !) Does anyone know how to nicely visualize the poster child of words embeddings ?
- lindenr 3y agoIf I understand you right - you could visualize in 2d space: "king" at origin, X-axis is "king"-"man", Y-axis is "king"-"woman" (or gram-schmidt if you really want orthogonal). In 3d you can go one further and have the Z-axis be "king"-"queen" (or gram-schmidt again). The orthogonalized versions have the advantage that they give a closer notion of distance to what the underlying model sees. In the 2d case you will get exact distances except that it won't show how far off "queen" you are when you compute "king"-"man"+"woman". In the 3d case it should give exact distances. Edit to add: With the 2d version you can maybe do some more stuff. IIRC "queen" is chosen as it's the word with the closest embedding to X="king"-"man"+"woman". You can put the next few closest words on the 2d chart as well, each labeled with the orthogonal distance from the 2d plane. So then "queen" should be the word with the smallest (squared distance from X) + (squared orthogonal distance from plane), which you might be able to eyeball.
- helloooooooo 3y agoYou should try UMAP
- DonHopkins 3y agoI remember hearing a kind of inductive joke by a mathematician about visualizing higher dimensions, like "...and then you just...", but I unsuccessfully tried to search for it with ChatGPT, when it came up with this funny Richard Feynman quote that I can't find on google -- has anyone heard it before, or is it a brilliant hallucination? "You can't visualize four dimensions... well, at least I can't, because I only have three branes." -Richard Feynman This is a play on words, as "branes" sounds like "brains," but in string theory and higher-dimensional physics, a "brane" is an object that can exist in multiple dimensions. The joke is a nod to the challenge of visualizing higher dimensions and the limits of our three-dimensional intuition. (It just admitted it made it up and apologized!) It also quoted John von Neumann, H. G. Wells, and Ian Stewart: "In mathematics you don't understand things. You just get used to them." -John von Neumann "Our brains are three-dimensional, so we cannot visualize four-dimensional or higher-dimensional objects in the same way we can visualize three-dimensional objects. The best we can do is to get a feeling for them through analogies." -H. G. Wells "You can't visualize four dimensions, but you can think about them. Thinking is what mathematicians do best." -Ian Stewart It finally guessed this one that's the closest to what I remember, but not as funny as I remember it: "To visualize four dimensions, just visualize three dimensions and then say 'n+1'." -Unknown So I asked it to hallucinate some quotes to help visualize high dimensional space, in the style of Deepak Chopra: "In the realm of septillion-dimensional embeddings, each vector is a topological manifold, where Gödel's incompleteness theorems dance with quantum superposition, revealing the hidden symmetries of semantic space." – Dr. Emilia Bianchi, Florentine Foundation of Fractal Fantasies, Florence "To truly fathom the intricacies of ultra-high-dimensional text spaces, one must first master the art of navigating Riemannian curvature in a Hilbert space, where every geodesic is a testament to the entangled essence of language." – Prof. Hiroshi Tanaka, Kyoto Kryptos of Quantum Quandaries "Visualizing embeddings in the realms beyond the Poincaré conjecture requires a melding of non-Euclidean geometry with the probabilistic nuances of a Markov chain, all while resonating with the cosmic vibrations of the Higgs field." – Dr. Samuel Oyelowo, Lagos Labyrinth of Lorentzian Linguistics "Each point in a decillion-dimensional text space is a singularity, where the Heisenberg uncertainty principle waltzes with the Mandelbrot set, painting a fractal portrait of phonemic phenomena." – Prof. Clara Müller, Bavarian Bastion of Bifurcated Babel, Munich "To intuit the vastness of hyper-dimensional embeddings, one must first traverse the event horizon of a linguistic black hole, where Shannon entropy meets the Schrödinger equation in a ballet of Bayesian inference." – Dr. Alexandre Dubois, Parisian Paradox of P-adic Prose "The key to deciphering the enigma of extreme-dimensional text lies in the delicate interplay between the Yang-Mills theory and the zeta function, where every eigenvalue is a whisper from the cosmic codex of communication." – Prof. Lina Johansson, Stockholm Sanctum of String Semiotics "In the dance of gogolplex-dimensional embeddings, each tensor unfolds like a Möbius strip, where the Fibonacci sequence intertwines with quantum tunneling, revealing the recursive rhythms of reality." – Dr. Rajiv Menon, Bengaluru Bardo of Bosonic Ballads
- choeger 3y agoMaybe I am missing something, but when you project every content into a fixed-size space, you have to lose information at some point, don't you? Did somebody ever run an image of a scene vs some parts of the scene in e.g., CLIP?
- atkindel 3y agoYour intuition here is right on; in fact, the reason why we like embeddings in the first place is that they address this problem! When we have some sparse high-dimensional vector representation of a bunch of content, we often find that we are devoting an entire dimension to very little information, so it would be convenient if we could approximate each of those vectors in fewer, denser dimensions without losing too much information. If all that we are interested in is how much the vectors align (i.e. correlate) with one another, embedding methods are one of the best ways to construct those lower-dimensional vectors, in the sense that we are guaranteed to find a solution up to whatever amount of approximation error we are willing to tolerate. An added benefit is that if we're smart about how we construct the lower-dimensional space, we often find that it generalizes to unseen data better than if we'd used the high-dimensional representation, because a lot of the variation we're throwing away is specific to how we constructed the input data. A really cool thing about embeddings is that we've known how to do this for a really long time --- for example, a landmark paper [1] on this exact problem was published in the 1930s! [1] Eckart, C. and G. Young (1936), "The approximation of one matrix by another of lower rank." Psychometrika 1, p. 211-218. https://link.springer.com/article/10.1007/BF02288367 https://link.springer.com/article/10.1007/BF02288367
- Alifatisk 3y agoSimon Willison’s Weblog have been so informative regarding the new era of LLMs and everything surrounding it.
- oleganza 3y agoCommon mistake when doing practical trigonometry is to compute square roots when that's not necessary: def cosine_similarity(a, b): dot_product = sum(x \* y for x, y in zip(a, b)) magnitude_a = sum(x \* x for x in a) \* 0.5 # <- no need for \*0.5 magnitude_b = sum(x \* x for x in b) \* 0.5 # <- no need for \*0.5 return dot_product / (magnitude_a \* magnitude_b) If you compare your cosines, you might as well compare their squares and avoid costly root computation. Similarly, in elliptic curve crypto certain expensive operations, such as inversion (x^-1 mod n) are delayed as much as possible down the computation pipeline or even avoided completely when you need to compare two points instead of computing their canonical values.
- dist-epoch 3y agoThis code is written to be easy to understand. Otherwise you would replace it with some low-level SIMD code.
- mikewarot 3y agoI'm amazed at how autoencoding actually works, despite the simplicity of the approach. I've dabbled with this stuff quite a bit, but I tend towards stuff I can run on my own computer, rather than trusting any service to remain stable for years or decades. Are there any good document embedding models I can run on my own hardware? Word2Vec is interesting, but I'd like to also try out cross-linking blog entries, etc.
- SushiHippie 3y agoI've found gpt4all to be quite good and runs locally. https://docs.gpt4all.io/gpt4all_python_embedding.html https://docs.gpt4all.io/gpt4all_python_embedding.html
- navbaker 3y agoThe HuggingFace embedding models are really good! See here for a good intro tutorial: https://huggingface.co/blog/getting-started-with-embeddings https://huggingface.co/blog/getting-started-with-embeddings
- ahoho 3y agosentence-transformers is excellent. It will be slow on CPU but, because it’s on top of pytorch, does have apple silicon/GPU support.
- simonw 3y agoI've been running these models on my own laptop without the GPU. They're generally a lot less resource demanding than full LLMs (though thanks to llama.cpp I've been able to run those on the CPU as well). More about the tooling I use to run embedding models locally here: https://simonwillison.net/2023/Sep/4/llm-embeddings/ https://simonwillison.net/2023/Sep/4/llm-embeddings/
- kordlessagain 3y agoInstall Instructor and dependencies: pip install InstructorEmbedding pip install torch pip install numpy pip install tqdm pip install sentence_transformers Use it: from InstructorEmbedding import INSTRUCTOR import numpy as np from scipy.spatial.distance import cosine xl = INSTRUCTOR('hkunlp/instructor-xl') embeddings = xl.encode(["I'm amazed at how autoencoding actually works, despite the simplicity of the approach.", "Are there any good document embedding models I can run on my own hardware? "]).tolist() cosine_distance = 1 - np.dot(embeddings[0], embeddings[1]) / (np.linalg.norm(embeddings[0]) * np.linalg.norm(embeddings[1])) print(cosine_distance) Output: 0.33516267980261427
- toasted-subs 3y agoThe mental model I have for embedding in languages has many points in multiple locations as if in an extremely high dimensional space. Kind of wish we could have visuals describing that.
- 8thcross 3y agocan someone explain on best means to embed large text? does the techniques matter (averaging vs. concat vectors)?
- eneuman 3y agoIf you want to see what Show HN posts, ProductHunt Startups, YC Companies, and Github repos are related to LLM embeddings, you can quickly find them on the LLM-Embeddings-Based Search Engine (MVP) I just launched: https://payperrun.com/%3E/search?displayParams={%22q%22:%22LLM-Embeddings%22,%22ds%22:%22showhn%22} https://payperrun.com/%3E/search?displayParams={%22q%22:%22L...
- thingification 3y agoNice. I expected clicking on the different "filter" buttons would update of the search results right away: I didn't expect I had to repeat the search (though I can see why you'd do that)
- eneuman 3y agoThanks for the feedback, I just fixed this!
- thingification 3y agoBehaves as I expected now! I went here looking for more info about payperrun https://payperrun.com/%3E/welcome https://payperrun.com/%3E/welcome and clicked on the "Spotlight" section and saw 4 popups blocked - I never see popups anywhere these days and have to admit that sends me away pretty quickly.
- eneuman 3y ago(My Show HN post here) https://news.ycombinator.com/item?id=38011802 https://news.ycombinator.com/item?id=38011802
- Andrew018 3y ago[dead]
- dboreham 3y agoIs "embedding" an old term? I'm wondering what's different vs https://en.wikipedia.org/wiki/Metric_space https://en.wikipedia.org/wiki/Metric_space
- wongarsu 3y agoWikipedia traces it back to this [1] 2002 paper, which talks about translated documents having an "embedding space" in each language. I guess you could interpret their paper as both English and French being embedded in some larger space, which would make it a valid use of the mathematical term. From there, people apparently just rolled with it and used the term more broadly, as tends to happen with language. 1: https://proceedings.neurips.cc/paper/2002/file/d5e2fbef30a4eb668a203060ec8e5eef-Paper.pdf https://proceedings.neurips.cc/paper/2002/file/d5e2fbef30a4e...
- blovescoffee 3y agoCheckout my comment next to yours. Embeddings in Geometry / in vector spaces are a very old topic. Also, your interpretation of their use of the word embedding is exactly correct.
- blovescoffee 3y agoCheckout https://en.wikipedia.org/wiki/Embedding#Topology_and_geometry https://en.wikipedia.org/wiki/Embedding#Topology_and_geometr... Crudely put, embedding is the verb, metric space is the noun.
- nehalecky 3y agoWe're using embeddings in key part of a new product feature. It's great to have references liked this to share with users to learn about them. Thanks for posting and sharing!
- afpx 3y agoCould you create a new language using embeddings? A perfect Esperanto
- lordofmoria 3y agoIn case anyone is interested, Heroku finally released pgvector support for Postgres yesterday: https://github.com/heroku/roadmap/issues/156 https://github.com/heroku/roadmap/issues/156 Pgvector is an extremely excellent way to experiment with embeddings in a lightweight way, without adding a bunch of extra infrastructure dependencies.
- simonw 3y agoOh awesome! My main blog is running on PostgreSQL on Heroku, can't wait to try this out. UPDATE: It looks like it's only available on the $50/month+ plans, my blog's database runs on a $9/month plan so I can't install extra extensions.
- greenie_beans 3y agothank you @simonw for all your writing about LLM's. also thank you for your blog in general, it's an amazing resource. also thank you for django. just, thank you!!!
- ofermend 3y agoFirst time I ran into embeddings was with word2vec and could not resist showing that, similar to the "king - man + woman ~ queen", it also the case that "yoda - good + evil ~ vader". It's also cool that the semantic meaning is the same in vector space, regardless of the language. Embeddings are also super important in retrieval-augmented-generation (RAG), and getting the best "embeddings model" is important to achieve the best RAG performance. At Vectara we recently launched our new Boomerang model that pushes the limit on performance on embedding models, and I hope will spur more innovation and further improvements in this space. https://vectara.com/introducing-boomerang-vectaras-new-and-improved-retrieval-model/ https://vectara.com/introducing-boomerang-vectaras-new-and-i...
- ozb 3y agoHas anyone actually used embeddings for anything other than Approximate Nearest Neighbor and clustering? Some speculative possibilities that come to mind: - projection, indexing and sorting on arbitrary axes (eg "hot minus cold", "happy minus sad", "scifi minus realism", "literary minus commercial") - SVM-style classification in Embeddings space - word2vec-style reasoning (woman-man+king=queen) - directly training embeddings (ie, not just taking a layer off an LLM); I know people use contrastive training methods, but I'd expect that other methods might be worth exploring, eg you could train embeddings together with neural nets representing functions, generate functional equations, and calculate MSE loss But really, I'm just surprised that it seems to be so focused on semantic search, to the exclusion of anything else... Surely there are other interesting applications?
- mcswell 3y agoCross-language embeddings, where you create an embedding space in each of two languages and then use a seed dictionary to align the spaces, has potential (actual?) applications in cross-lingual search and probably MT.
- mcswell 3y agoI forgot, we also used word2vec to build an embedding space based on PubMed abstracts. We found lots of variant spellings (including hyphenated, unhyphenated, and space-separated), acronyms and abbreviations for chemical and biochem names. We could probably have built a lexicon of technical terminology from that. Not sure how far we would have gotten with definitions (vectors don't quite work...), but it would be a start. Pretty sure others have done dictionary building using that mechanism too.
- godelski 3y agoI'm a bit confused by the comment because it would seem that these are all relatively common tasks. The first and third being identical. Good example is in vision you might want to semantically change the photo like adding a pair of glasses or doing one of those things you see on the Google commercials. That's done in a latent space I think this is clearest in the Normalizing Flow cases because you're just turning your space into a Gaussian (diffusion does this too, but through approximation methods and isn't invertible, though it is reversible). You project the image/sentence/data you want to manipulate, manipulate within gaussian space, then return back to target space. Or maybe my confusion is shared confusion because embedding is an overloaded word that means a lot of things? Maybe you're just thinking of the first block that converts discrete integer tokens into continuous floats? But we learn those embeddings so even though it becomes like a lookup table it's still a neural process. People do things like SVMs on this space alone. But I think it is like latent space, which is only a bit more abstract. At least embeddings need to be injective, well... mathematically...
- nuancebydefault 3y agoAt work I was making an hyperspectral image pixel classifier in Python. I just noticed how close its implementation is to the use of embeddings to find the relatedness of articles. ie. by calculation of dot product and cosine on their embeddings. How weird, the natural language world lends itself to classification with the same calculations as used in the images world.
- trash_cat 3y agoYou can indeed use Retrieval-Augmented Generation or question and answering. Is this not what "Uploading a pdf" or Advanced Data Analytics do in ChatGPT Plus under the hood? Another point of finetuning a model is to caputure the behaviour and underlying "reasoning" that comes with it. The token limit is another reason why RAG might not be enough if you want more than just Q&A with an LLM.