4 ms·
One of my biggest annoyances with the modern AI tooling hype is that you need to use a vector store for just working with embeddings. You don't. The reason vec
by minimaxir 3y ago
One of my biggest annoyances with the modern AI tooling hype is that you need to use a vector store for just working with embeddings. You don't.
The reason vector stores are important for production use-cases are mostly latency-related for larger sets of data (100k+ records), but if you're working on a toy project just learning how to use embeddings, you can compute cosine distance with a couple lines of numpy by doing a dot product of a normalized query vectors with a matrix of normalized records.
Best of all, it gives you a reason to use Python's @ operator, which with numpy matrices does a dot product.
- twelfthnight 3y agoEven in production my guess is most teams would be better off just rolling their own embedding model (huggingface) + caching (redis/rocksdb) + FAISS (nearest neighbor) and be good to go. I suppose there is some expertise needed, but working with a vector database vendor has major drawbacks too.
- hackernoteng 3y agoUsing Postgres with pgvector is trivial and cheap. Its also available on AWS RDS.
- jonplackett 3y agoAlso on supabase!
- danielbln 3y agoOr you just shove it into Postgres + pg_vector and just use the DBMS you already use anyway.
- christiangenco 3y agoYup. I was just playing around with this in Javascript yesterday and with ChatGPT's help it was surprisingly simple to go from text => embedding (via. `openai.embeddings.create`) and then to compare the embedding similarity with the cosine distance (which ChatGPT wrote for me): https://gist.github.com/christiangenco/3e23925885e3127f2c1775871b8f52f1 https://gist.github.com/christiangenco/3e23925885e3127f2c177... Seems like the next standard feature in every app is going to be natural language search powered by embeddings.
- minimaxir 3y agoFor posterity, OpenAI embeddings come pre-normalized so you can immediately dot-product. Most embeddings providers do normalization by default, and SentenceTransformers has a normalize_embeddings parameter which does that. (it's a wrapper around PyTorch's F.normalize)
- itronitron 3y agoDoes anyone know the provenance for when vectors started to be called embeddings?
- minimaxir 3y agoI think it was due to GloVe embeddings back then: I don't recall them ever being called GloVe vectors, although the "Ve" does stand for vector so it could have been RAS syndrome.
- itronitron 3y ago>> https://nlp.stanford.edu/projects/glove/ https://nlp.stanford.edu/projects/glove/ A quick scan of the project website yields zero uses of 'embedding' and 23 of 'vector'
- minimaxir 3y agoIt's how I remember it when I was working with them back in the day (word embeddings): I could be wrong.
- nmfisher 3y agoIn an NLP context, earliest I could find was ICML 2008: http://machinelearning.org/archive/icml2008/papers/391.pdf http://machinelearning.org/archive/icml2008/papers/391.pdf I'm sure there are earlier instances, though - the strict mathematical definition of embedding has surely been around for a lot longer. (interestingly, the word2vec papers don't use the term either, so I guess it didn't enter "common" usage until the mid-late 2010s)
- bryantwolf 3y agoAs an individual, I love the idea of pushing to simplify even further to understand these core concepts. For the ecosystem, I like that vector stores make these features accessible to environments outside of Python.
- simonw 3y agoIf you ask ChatGPT to give you a cosine similarity function that works against two arrays of floating numbers in any programming language you'll get the code that you need. Here's one in JavaScript (my prompt was "cosine similarity function for two javascript arrays of floating point numbers"): function cosineSimilarity(vecA, vecB) { if (vecA.length !== vecB.length) { throw "Vectors do not have the same dimensions"; } let dotProduct = 0.0; let normA = 0.0; let normB = 0.0; for (let i = 0; i < vecA.length; i++) { dotProduct += vecA[i] * vecB[i]; normA += vecA[i] ** 2; normB += vecB[i] ** 2; } if (normA === 0 || normB === 0) { throw "One of the vectors is zero, cannot compute similarity"; } return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB)); } Vector stores really aren't necessary if you're dealing with less than a few hundred thousand vectors - load them up in a bunch of in-memory arrays and run a function like that against them using brute-force.
- bryantwolf 3y agoI love it!
- sixbrx 3y agoi get triggered that the norm vars aren't ever really the norms but their squares
- hereonout2 3y ago100k records is still pretty small! It feels a bit like the hype that happended with "big data". People ended up creating spark clusters to query a few million records. Or using Hadoop for a dataset you could process with awk. Professionally I've only ever worked with dataset sizes in the region of low millions and have never needed specialist tooling to cope. I assume these tools do serve a purpose but perhaps one that only kicks in at a scale approaching billions.
- smahs 3y agoThis sentiment is pretty common I guess. Outside of a niche, the massive scale for which a vast majority of the data tech was designed doesn't exist and KISS wins outright. Though I guess that's evolution, we want to test the limits in pursuit of grandeur before mastering the utility (ex. pyramids).
- jonnycoder 3y agoKISS doesn't get me employed though. I narrowly missed being the chosen candidate for a State job which called for Apache Spark experience. I missed two questions relating to Spark and "what is a parquet file?" but otherwise did great on the remaining behavioral questions (the hiring manager gave me feedback after requesting it). Too bad they did not have a question about processing data using command lines tools.
- mritchie712 3y agoyeah, glad the hype around big data is dead. Not a lot of solid numbers in here, but this post covers it well[0]. We have duckdb embedded in our product[1] and it works perfectly well for billions of rows of a data without the hadoop overhead. 0 - https://motherduck.com/blog/big-data-is-dead/ https://motherduck.com/blog/big-data-is-dead/ 1 - https://www.definite.app/ https://www.definite.app/
- hot_gril 3y agoI've been in the "mid-sized" area a lot where Numpy etc cannot handle it, so I had to go to Postgres or more specialized tooling like Spark. But I always started with the simple thing and only moved up if it didn't suffice. Similarly, I read how Postgres won't scale for a backend application and I should use Citus, Spanner, or some NoSQL thing. But that day has not yet arrived.
- leobg 3y agohnswlib, usearch. Both handle tens of millions of vectors easily. The latter even without holding them in RAM.
- ertgbnm 3y agoWhen I'm messing around, I normally have everything in a Pandas DataFrame already so I just add embeddings as a column and calculate cosine similarity on the fly. Even with a hundred thousand rows, it's fast enough to calculate before I can even move my eyes down on the screen to read the output. I regret ever messing around with Pinecone for my tiny and infrequently used set ups.
- laurshelly 3y agoCould not agree more. For some reason Pandas seems to get phased out as developers advance.
- m1117 3y agoActually, I had a pretty good experience with Pinecone.