5 ms·
I've been gleefully exploring the intersection of LLMs and CLI utilities for a few months now - they are such a great fit for each other! The unix philosophy of
by simonw 3y ago
I've been gleefully exploring the intersection of LLMs and CLI utilities for a few months now - they are such a great fit for each other! The unix philosophy of piping things together is a perfect fit for how LLMs work.
I've mostly been exploring this with my https://llm.datasette.io/ https://llm.datasette.io/ CLI tool, but I have a few other one-off tools as well: https://github.com/simonw/blip-caption https://github.com/simonw/blip-caption and https://github.com/simonw/ospeak https://github.com/simonw/ospeak
I'm puzzled that more people aren't loudly exploring this space (LLM+CLI) - it's really fun.
- sevagh 3y ago>I'm puzzled that more people aren't loudly exploring this space (LLM+CLI) - it's really fun. 70% of the front page of Hackernews and Twitter for the past 9 months is about everybody and their mother's new LLM CLI. It's the loudest exploration I've ever witnessed in my tech life so far. We need to be hearing far less about LLM CLIs, not more.
- minimaxir 3y agoGranted half of those are submissions about Simon's new projects.
- squigz 3y agoErr... LLM in general? Sure. Specifically CLI LLM stuff? Certainly not 70%...
- simonw 3y agoI've been reading Hacker News pretty closely and I haven't seen that. Plenty of posts about LLM tools - Ollama, llama.cpp etc - but very few that were specifically about using LLMs with Unix-style CLI piping etc. What did I miss?
- deleted 3y ago[deleted]
- jart 3y agoHas anyone written a shell script before that uses a local llm as a composable tool? I know there's plenty of stuff like https://github.com/ggerganov/llama.cpp/blob/master/examples/Miku.sh https://github.com/ggerganov/llama.cpp/blob/master/examples/... where the shell script is being used to supply all the llama.cpp arguments you need to get a chatbot ui. But I haven't seen anything yet that treats the LLM as though it were a traditional UNIX utility like sed, awk, cat, etc. I wouldn't be surprised if no one's done it, because I had to invent the --silent-prompt flag that let me do it. I also had to remove all the code from llava-cli that logged stuff to stdout. Anyway, here's the script I wrote: https://gist.github.com/jart/bd2f603aefe6ac8004e6b709223881c0 https://gist.github.com/jart/bd2f603aefe6ac8004e6b709223881c...
- verdverm 3y agoIt's really hard to put an LLM in the middle of unix pipes because the output is unreliable I wrote one a while back, mainly so I could use it with vim (without a plugin) and pipe my content in (also at the CLI), but haven't maintained it https://github.com/verdverm/chatgpt https://github.com/verdverm/chatgpt
- npongratz 3y agoJustine may have addressed unreliable output by using `--temp 0` [0]. I'd agree that while it may be deterministic, there are other definitions or axes of reliability that may still make it poorly suited for pipes. [0] > Notice how I'm using the --temp 0 flag again? That's so output is deterministic and reproducible. If you don't use that flag, then llamafile will use a randomness level of 0.8 so you're certain to receive unique answers each time. I personally don't like that, since I'd rather have clean reproducible insights into training knowledge.
- jart 3y ago`--temp 0` makes it deterministic. What can make output reliable is `--grammar` which the blog post discusses in detail. It's really cool. For example, the BNF expression `root ::= "yes" | "no"` forces the LLM to only give you a yes/no answer.
- sevagh 3y ago> LLM tools - Ollama, llama.cpp I was including these since they're LLM-related cli tools.
- alchemist1e9 3y agoThose are great links. I’ve been using: https://github.com/npiv/chatblade https://github.com/npiv/chatblade https://github.com/tbckr/sgpt https://github.com/tbckr/sgpt I totally agree with LLM+CLI are perfect fit. One pattern I used recently was httrack + w3m dump + sgpt images with gpt vision to generate a 278K token specific knowledge base with a custom perl hack for a RAG that preserved the outline of the knowledge. Which brings me to my question for you - have you seen anything unix philosophy aligned for processing inputs and doing RAG locally? EDIT. Turns out OP has done quite a bit toward what I’m asking. Written up here: https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ Something I’m currently a bit hung up on is finding a toolchain for chunking content on which to create embeddings. Ideally it would detect location context, like section “2.1 Failover” or “Chapter 8: The dream” or whatever from the original text, also handle 80 character wide source unwrapping, smart splitting so paragraphs are kept together, etc etc.
- levidos 3y agoI'm also a very heavy daily user of a similar tool, shell_gpt. Not having it would feel like a limb missing, very odd. https://github.com/TheR1D/shell_gpt https://github.com/TheR1D/shell_gpt
- simonw 3y agoThat's the same problem I haven't figured out yet: the best strategies for chunking. I'm hoping good, well proven patterns emerge soon so I can integrate them into my various tools.
- alchemist1e9 3y agoI’ve been looking at this https://freeling-user-manual.readthedocs.io/en/v4.2/modules/splitter/ https://freeling-user-manual.readthedocs.io/en/v4.2/modules/... at the freeling library in general, also spaCy and NLTK. The chunking algorithms being used in the likes of LangChain are remarkably bad surprisingly. There is also https://github.com/Unstructured-IO/unstructured https://github.com/Unstructured-IO/unstructured But I don’t like it, can’t explain why yet. My intuition is that 1st step is clean sentences and paragraphs and titles/labels/headers. Then probably an LLM can handle outlining and table of contents generation using a stripped down list of objects in the text. BRIO/BERT summarization could also have a role of some type. Those are my ideas so far.
- larve 3y agoI'm heavily using https://github.com/go-go-golems/geppetto https://github.com/go-go-golems/geppetto for my work, which has a CLI mode and TUI chat mode. It exposes prompt templates as command line verbs, which it can load from multiple "repositories". I maintain a set of prompts for each repository I am working in (alongside custom "prompto" https://github.com/go-go-golems/prompto https://github.com/go-go-golems/prompto scripts that generate dynamic prompting context, i made quite a few for thirdparty libraries for example: https://github.com/go-go-golems/promptos https://github.com/go-go-golems/promptos ). Here's some of the public prompts I use: https://github.com/go-go-golems/geppetto/tree/main/cmd/pinocchio/prompts https://github.com/go-go-golems/geppetto/tree/main/cmd/pinoc... I am currently working on a declarative agent framework.
- verdverm 3y ago> I'm puzzled that more people aren't loudly exploring this space (LLM+CLI) - it's really fun. I've been seeing less and less enthusiasm for CLI driven workflows. I think VS Code is the main driver for this and anecdotally the developers I serve want point & click over terminal & cli
- csdvrx 3y ago> anecdotally the developers I serve want point & click over terminal & cli I think it's due to a lack of familiarity, as the CLI should be more efficient > I've been seeing less and less enthusiasm for CLI driven workflows. Any CLI is 1 dimensional. Point and click is 2 dimensional. The CLI should be more efficient, as you can reduce the complexity: you may need extra flags to achieve the behavior you want, but you can then serialize that into a file (shell script) to guarantee the reproduction of the outcome you want. GUIs are harder, even without adding more dimensions like time (double click, scripts like AHK or AutoIT...) If you don't have comparative exposure (automatizing workflows in Windows vs doing the same in Linux), or if you don't have enough experience to achieve what you want, you might jump to the wrong conclusions - but this is a case of being limited by knowledge, not by the tools
- pacoverdi 3y ago> Point and click is 2 dimensional. I would have said point and click is 3-dimensional. Otherwise, how can you read the text through the edges of buttons before clicking?
- ElectricalUnion 3y agoBy this definition, a cli is at least 52-dimensional (26*2 latin letters), because you can often readline reverse-search-history back in time and into the exact text you typed before. This model of interaction is simply not possible in your average "WYSIWYG" Windows-Icons-Menus-Pointer GUI, not without somehow falling back to a sort CLI paradigm. Or you could use one of the most successful desktop/web models of the last 15 years: search engines, also with many million if not more dimensions, all available in a CLI. Or you could go for broke and use a 70B dimensions LLM prompt, integrated into your shell, and generate a custom new one-liner/script for you. I don't think a WYSIWYG Windows-Icons-Menus-Pointer GUI will ever have that many dimensions.