4 ms·

>I'm puzzled that more people aren't loudly exploring this space (LLM+CLI) - it's really fun. 70% of the front page of Hackernews and Twitter for the past 9 mo

by sevagh 3y ago

>I'm puzzled that more people aren't loudly exploring this space (LLM+CLI) - it's really fun. 70% of the front page of Hackernews and Twitter for the past 9 months is about everybody and their mother's new LLM CLI. It's the loudest exploration I've ever witnessed in my tech life so far. We need to be hearing far less about LLM CLIs, not more.

minimaxir 3y ago
Granted half of those are submissions about Simon's new projects.
squigz 3y ago
Err... LLM in general? Sure. Specifically CLI LLM stuff? Certainly not 70%...
simonw 3y ago
I've been reading Hacker News pretty closely and I haven't seen that. Plenty of posts about LLM tools - Ollama, llama.cpp etc - but very few that were specifically about using LLMs with Unix-style CLI piping etc. What did I miss?
- deleted 3y ago
  [deleted]
- jart 3y ago
  Has anyone written a shell script before that uses a local llm as a composable tool? I know there's plenty of stuff like https://github.com/ggerganov/llama.cpp/blob/master/examples/Miku.sh https://github.com/ggerganov/llama.cpp/blob/master/examples/... where the shell script is being used to supply all the llama.cpp arguments you need to get a chatbot ui. But I haven't seen anything yet that treats the LLM as though it were a traditional UNIX utility like sed, awk, cat, etc. I wouldn't be surprised if no one's done it, because I had to invent the --silent-prompt flag that let me do it. I also had to remove all the code from llava-cli that logged stuff to stdout. Anyway, here's the script I wrote: https://gist.github.com/jart/bd2f603aefe6ac8004e6b709223881c0 https://gist.github.com/jart/bd2f603aefe6ac8004e6b709223881c...
  - verdverm 3y ago
    It's really hard to put an LLM in the middle of unix pipes because the output is unreliable I wrote one a while back, mainly so I could use it with vim (without a plugin) and pipe my content in (also at the CLI), but haven't maintained it https://github.com/verdverm/chatgpt https://github.com/verdverm/chatgpt
    - npongratz 3y ago
      Justine may have addressed unreliable output by using `--temp 0` [0]. I'd agree that while it may be deterministic, there are other definitions or axes of reliability that may still make it poorly suited for pipes. [0] > Notice how I'm using the --temp 0 flag again? That's so output is deterministic and reproducible. If you don't use that flag, then llamafile will use a randomness level of 0.8 so you're certain to receive unique answers each time. I personally don't like that, since I'd rather have clean reproducible insights into training knowledge.
      - jart 3y ago
        `--temp 0` makes it deterministic. What can make output reliable is `--grammar` which the blog post discusses in detail. It's really cool. For example, the BNF expression `root ::= "yes" | "no"` forces the LLM to only give you a yes/no answer.
        verdverm 3y ago
        that only works up to a point. If you are trying to transform a text based cli output into a JSON object, even with a grammar, you can get variation in the output. A simple example is field or list ordering. Omission is the real problematic one
      - verdverm 3y ago
        > Justine may have addressed unreliable output by using `--temp 0` That only works if you have the same input. It also nerfs the model considerably https://ai.stackexchange.com/questions/32477/what-is-the-temperature-in-the-gpt-models https://ai.stackexchange.com/questions/32477/what-is-the-tem...
- sevagh 3y ago
  > LLM tools - Ollama, llama.cpp I was including these since they're LLM-related cli tools.

by sevagh 3y ago

minimaxir 3y ago

Granted half of those are submissions about Simon's new projects.

squigz 3y ago

Err... LLM in general? Sure. Specifically CLI LLM stuff? Certainly not 70%...

simonw 3y ago

I've been reading Hacker News pretty closely and I haven't seen that. Plenty of posts about LLM tools - Ollama, llama.cpp etc - but very few that were specifically about using LLMs with Unix-style CLI piping etc. What did I miss?

deleted 3y ago
[deleted]
jart 3y ago
Has anyone written a shell script before that uses a local llm as a composable tool? I know there's plenty of stuff like https://github.com/ggerganov/llama.cpp/blob/master/examples/Miku.sh https://github.com/ggerganov/llama.cpp/blob/master/examples/... where the shell script is being used to supply all the llama.cpp arguments you need to get a chatbot ui. But I haven't seen anything yet that treats the LLM as though it were a traditional UNIX utility like sed, awk, cat, etc. I wouldn't be surprised if no one's done it, because I had to invent the --silent-prompt flag that let me do it. I also had to remove all the code from llava-cli that logged stuff to stdout. Anyway, here's the script I wrote: https://gist.github.com/jart/bd2f603aefe6ac8004e6b709223881c0 https://gist.github.com/jart/bd2f603aefe6ac8004e6b709223881c...
- verdverm 3y ago
  It's really hard to put an LLM in the middle of unix pipes because the output is unreliable I wrote one a while back, mainly so I could use it with vim (without a plugin) and pipe my content in (also at the CLI), but haven't maintained it https://github.com/verdverm/chatgpt https://github.com/verdverm/chatgpt
  - npongratz 3y ago
    Justine may have addressed unreliable output by using `--temp 0` [0]. I'd agree that while it may be deterministic, there are other definitions or axes of reliability that may still make it poorly suited for pipes. [0] > Notice how I'm using the --temp 0 flag again? That's so output is deterministic and reproducible. If you don't use that flag, then llamafile will use a randomness level of 0.8 so you're certain to receive unique answers each time. I personally don't like that, since I'd rather have clean reproducible insights into training knowledge.
    - jart 3y ago
      `--temp 0` makes it deterministic. What can make output reliable is `--grammar` which the blog post discusses in detail. It's really cool. For example, the BNF expression `root ::= "yes" | "no"` forces the LLM to only give you a yes/no answer.
      - verdverm 3y ago
        that only works up to a point. If you are trying to transform a text based cli output into a JSON object, even with a grammar, you can get variation in the output. A simple example is field or list ordering. Omission is the real problematic one
    - verdverm 3y ago
      > Justine may have addressed unreliable output by using `--temp 0` That only works if you have the same input. It also nerfs the model considerably https://ai.stackexchange.com/questions/32477/what-is-the-temperature-in-the-gpt-models https://ai.stackexchange.com/questions/32477/what-is-the-tem...
sevagh 3y ago
> LLM tools - Ollama, llama.cpp I was including these since they're LLM-related cli tools.