4 ms·
LLM Python/CLI tool adds support for embeddings
- linsomniac 3y agoSimon, you are a force of nature!
- simonw 3y agoThere's a lot of stuff in this release. Don't miss the new llm-cluster plugin, which can both calculate clusters from embeddings and use another LLM call to generate a name for each cluster: https://github.com/simonw/llm-cluster https://github.com/simonw/llm-cluster Example usage: Fetch all issues, embed them and store the embeddings and content in SQLite: paginate-json 'https://api.github.com/repos/simonw/llm/issues?state=all&filter=all' \ | jq '[.[] | {id: .id, title: .title}]' \ | llm embed-multi llm-issues - \ --database issues.db \ --model sentence-transformers/all-MiniLM-L6-v2 \ --store Group those in 10 clusters and generate a summary for each one using a call to GPT-4: llm cluster llm-issues --database issues.db 10 --summary --model gpt-4
- panarky 3y agoVery curious to follow your journey through embedding search. If I want 100 close matches that match a filter, is it better to filter first then find vector similarity within that, or find 1000 similar vectors and then filter that subset?
- simonw 3y agoI experimented with that a few months ago. Building a fresh FAISS index for a few thousand matches is really quick, so I think it's often better to filter first, build a scratch index and then use that for similarity: https://github.com/simonw/datasette-faiss/issues/3 https://github.com/simonw/datasette-faiss/issues/3 ... although on thinking about this more I realize that a better approach may well be to just filter down to ~1,000 and then run a brute-force score across all of them, rather than messing around with an index.
- rolisz 3y agoFor 1000 points, brute force is super quick. Actually, up to 100k (on my machine), brute force takes less than 1 second.
- deleted 3y ago[deleted]
- maleldil 3y ago> jq '[.[] | {id: .id, title: .title}]' Can be simplified to: jq 'map({id, title})'
- quickthrower2 3y agoI would change the title to: Python Library "llm" now provides tools for working with embeddings I initially was trying to parse that, thinking "is this an open AI thing?". Of course the answer is just a click away, but people might miss this if they are interested in Python coding and AI.
- dang 3y agoOK, we've put Python library up there.
- simonw 3y agoLooks like you missed my reply by seconds pointing out that it's not just a Python library, it's also a CLI tool: https://news.ycombinator.com/item?id=37385788 https://news.ycombinator.com/item?id=37385788
- quickthrower2 3y agoAah! Sorry about that both of you. I didn't think dang would see this and simon would update the title and sanity check it.
- dang 3y agoOk - but we need some qualifier. If the title just says "LLM" that's far too general. Any suggestions?
- thatcherthorn 3y agoThis is a fantastic library. I plan to use some of the search functionality with a system that tries to figure out how to manipulate/work with/add features to existing code.
- haxton 3y agoCurious to know what value you've seen out of these clusters. In my experience k means clustering was very lackluster. Having to define the number of clusters was a big pain point too. You almost certainly want a graph like structure (overlapping communities rather than clusters). But unsupervised clustering was almost entirely ineffective for every use case I had :/
- simonw 3y agoI only got the clustering working this morning, so aside from playing around with it a bit I've not had any results that have convinced me it's a tool I should throw at lots of different problems. I mainly like it as another example of the kind of things you can use embeddings for. My implementation is very naive - it's just this: sklearn.cluster.MiniBatchKMeans(n_clusters=n, n_init="auto") I imagine there are all kinds of improvements that could be made to this kind of thing. I'd love to understand if there's a good way to automatically pick an interesting number of clusters, as opposed to picking a number at the start. https://github.com/simonw/llm-cluster/blob/main/llm_cluster.py https://github.com/simonw/llm-cluster/blob/main/llm_cluster....
- haxton 3y agoElbow method is a good place to start for finding the number of clusters.
- simonw 3y agoThat's a useful hint, thanks. I fed it through GPT-4 and got some interesting leads: https://chat.openai.com/share/400f76ae-b53b-4d07-ac31-adcef22dc0ba https://chat.openai.com/share/400f76ae-b53b-4d07-ac31-adcef2... and https://chat.openai.com/share/48650db8-5a29-49c5-84b2-574f53beb911 https://chat.openai.com/share/48650db8-5a29-49c5-84b2-574f53...
- FreakLegion 3y agoThere are iterative methods for optimizing the number of clusters in k-means (silhouette and knee/elbow are common), but in practice I prefer density-based methods like HDBSCAN and OPTICS. There's a very basic visual comparison at https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html#sphx-glr-auto-examples-cluster-plot-cluster-comparison-py https://scikit-learn.org/stable/auto_examples/cluster/plot_c....
- dimal 3y agoJust have to say, I’m enjoying playing with llm so much. Being able to easily play with this technology at the command line feels like magic. Thanks for the awesome work!
- e12e 3y agoReally thrilled to see such a simple tool handle for complex LLM tools. I wonder how well it would work for building an "apropos" tool for searching man pages (and infotext)?
- syntaxing 3y agoThis looks absolutely awesome. Does this handle prompt/instruct format using the plugins? It's been the biggest pain point for me using llama.cpp directly.
- simonw 3y agoI'm still iterating on that. Plugins get complete control over the prompts, so they can handle the various weirdnesses of them. Here's some relevant code: https://github.com/simonw/llm-gpt4all/blob/0046e2bf5d0a9c369b804d7125a1ab50bd5878f1/llm_gpt4all.py#L87-L92 https://github.com/simonw/llm-gpt4all/blob/0046e2bf5d0a9c369... https://github.com/simonw/llm-mlc/blob/b05eec9ba008e700ecc422c69e28395fa8e146fd/llm_mlc.py#L290-L295 https://github.com/simonw/llm-mlc/blob/b05eec9ba008e700ecc42... https://github.com/simonw/llm-llama-cpp/blob/29ee8d239f5cfbf0b4bc1e55af39045708840b25/llm_llama_cpp.py#L197-L220 https://github.com/simonw/llm-llama-cpp/blob/29ee8d239f5cfbf... I'm not completely happy with this yet. Part of the problem is that different models on the same architecture may have completely different prompting styles. I expect I'll eventually evolve the plugins to allow them to be configured in an easier and more flexible way. Ideally I'd like you to be able to run new models on existing architectures using an existing plugin.
- kordlessagain 3y agoStill pondering on using embeddings for classification. Yes, we can group similars with embeddings through clustering, but how do you extract the label for the groups? What I've come up with is either a) ask an LLM for the common label from samples from the grouped set after indexing (what keyterm best describes the relationship between these documents), or b) determine the label (or keyword) while indexing (by having the LLM find the keyterms ahead of time), then use set overlap on the grouped set's keyterms after to determine a label for the group.
- Nydhal 3y agoYou seem to be looking for a topic model. BERTopic might help: https://maartengr.github.io/BERTopic/index.html#quick-start https://maartengr.github.io/BERTopic/index.html#quick-start
- kordlessagain 3y agoThis is great...thank you!
- stavros 3y agoWell this is timely, I was just yesterday needing a tool that would easily do embeddings for me so I could do RAG. LLM seems ideal, thanks Simon! One question, I saw a comment here that doing RAG efficiently entails some more trickery, like chunking the embeddings. In your experience, is stuff like that necessary, or do I pass the returned documents to GPT-4 and that's it for my RAG? For an example of what I'm doing now, I bought an ESP32-Box (think basically an OSS Amazon Echo) and want to ask it questions about my (Markdown) notes. What would be the easiest way to do that?
- simonw 3y agoI'm still trying to figure out the answer to that question myself. The absolute easiest approach right now is to use Claude, since it has a 100,000 token limit - so you can stuff a ton of documentation into it at once and start asking questions. Doing RAG with smaller models requires much more cleverness, which I'm only just starting to explore.
- stavros 3y agoThat's fair, thanks! Do you plan to integrate the cleverness into LLM, so we can benefit from it too? I'm not sure if LLM can be used as a library, currently, I've only been using it as a cli, but it would be great if I could use it in my programs without shelling out.
- simonw 3y agoYes, all of this stuff will end up in LLM - either in core or as a plugin for it. LLM works as a library already, but there's definitely room for improvement there: https://llm.datasette.io/en/stable/python-api.html https://llm.datasette.io/en/stable/python-api.html https://llm.datasette.io/en/stable/embeddings/python-api.html https://llm.datasette.io/en/stable/embeddings/python-api.htm...
- stavros 3y agoThis is great news, thanks!
- dankle 3y agoWhat is the benefit of this lib over hf transformers?
- simonw 3y agoLLM is to transformers like the Django ORM or SQLAlchemy is to the Python psycopg2 and MySQLdb client libraries. It's a higher layer abstraction that runs on top of multiple libraries like HF transformers, in some case calling plugins that use transformers directly.
- Tomte 3y ago> # Generate and store embeddings for every README.md in your home directory, recursively Do I have to run it against my own corpus? Are there "standard" embeddings that many people use that I could use with llm?
- simonw 3y agoI found this project recently which is really interesting: https://alex.macrocosm.so/download https://alex.macrocosm.so/download They've embedded all papers on Arxiv and made the results available for anyone to use. I believe they used the Instructor XL model - I need to build an LLM plugin for that.
- bionhoward 3y agoIs there a natural embedding for text? Why do we need to make up random extremely precise numbers and optimize them expensively and keep that thing hanging around?