4 ms·

Language models on the command line

141 points by rednafi 2y ago

bearjaws 2y ago
Aider continues to be the best way to interact with LLMs while coding, and its a command line tool. Copilot is pretty good, but the forced change > commit > QA process that Aider forces you through is really powerful.
behnamoh 2y ago
I wish llm were more stable, but unfortunately things just kept breaking out of the blue without me touching any settings of the program. I often had to reinstall the package. but finally I gave up and implemented my own.
- simonw 2y ago
  Which plugins are you using? Did you install via pipx or Homebrew or something else?
  - behnamoh 2y ago
    I used pipx and used AnthropicAI's plugin to use Claude. After 2 weeks of perfectly working, suddenly I got the "this model is invalid" error. PS: I appreciate the work put into llm tho, it's a neat program I used with my Automator scripts to bring LLMs to macOS before Apple Intelligence was announced. I just wish the stability was not a concern.
    - foobarqux 2y ago
      The plugins seem to need to be reinstalled after every upgrade
    - simonw 2y ago
      That's weird. Were you using llm-claude or llm-claude-3?
  - jimmcslim 2y ago
    I followed along with the blog post, but got unstuck with llm-cmd not working on Mac OS. Looks like this PR would fix it https://github.com/simonw/llm-cmd/pull/12 https://github.com/simonw/llm-cmd/pull/12
simonw 2y ago
This was a workshop I gave in my https://llm.datasette.io/ https://llm.datasette.io/ CLI tool. What other CLI tools are people using to work with LLMs in the terminal? There one comment here about https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider and Ollama is probably the most widely used CLI tool at the moment: https://github.com/ollama/ollama/blob/main/README.md#quickstart https://github.com/ollama/ollama/blob/main/README.md#quickst...
- throwup238 2y ago
  > What other CLI tools are people using to work with LLMs in the terminal? I use aichat: https://github.com/sigoden/aichat https://github.com/sigoden/aichat I especially like the terminal integration where I can type a natural language request at the terminal and press Alt+E to have it converted to a command to run.
  - kelvie 2y ago
    I also use aichat. For some reason (well, I know the reason, it's shell quoting) I don't like passing prompts in via command line, so having a simple text-based TUI like a readline prompt is nice.
  - mikeqq2024 2y ago
    There is similar ShellOracle: https://github.com/djcopley/ShellOracle https://github.com/djcopley/ShellOracle It is used as Bash widget, where you press CTL-F, describe your command, then press Enter, the generated command will be inserted into your shell prompt.
- bt1a 2y ago
  (aichat in tangential comment looks much better) Besides aider and ollama, I think shell_gpt https://github.com/TheR1D/shell_gpt https://github.com/TheR1D/shell_gpt is great for quick chats / lookups. Being able to quickly cat files to repl sessions saves a lot of time. I need to integrate distil whisper large v3, aider, and shell_gpt to tidy up a lot of my disjointed LLM use. As someone else mentioned, the commits created by aider allow me to "skip" some intermediate steps that would be required when working on coding tasks with other frameworks.
liamYC 2y ago
This is awesome, thanks for sharing. I find this kind of tool really useful, Aider in particular. I made my own cli tool for interacting with GPT. It’s really useful with the -c flag for generating code especially bash commands I've forgotten https://github.com/ljsimpkin/chat-gpt-cli-tool https://github.com/ljsimpkin/chat-gpt-cli-tool
bagels 2y ago
I wrote one to help with creating command line commands. It just hits openai api with a prompt asking for just a code block to run on bash + whatever is passed in, and then it prints the command out. I wrote it because I can never remember all the weird command args for all the tools. $ bashy find large files over 10 gb find / -type f -size +10G
- QuiDortDine 2y ago
  Do you have a followup command to execute the suggested command?
  - sdf4j 2y ago
    $ bashy evaluate the previous command output
  - bagels 2y ago
    No, I want to review or tweak them, just in case it's trying to do something bad. It's usually pretty good though.
    - godelski 2y ago
      If you use zsh and are willing to source your scripts you can do print -z $command And the command will appear on your cli as if you had written it. I don't think you can do this in bash. Interestingly, this is something that seems quite difficult to both google and ask GPT for help. Both get confused and are thinking different questions are being asked. Probably because there are similar more common questions but the subtleties of possible wordings makes it difficult to differentiate.
dvt 2y ago
Fantastic work here! I'm working on a local tool, affectionately called Descartes, which does something similar—but with a spotlight-like UX for the non-hackers out there. I do think that LLMs have the potential to fundamentally change the way we interact with our computers. There's a lot of edge cases (especially when combining it with the inaccurate science of screen readers) but it's pretty mind-blowing when it works. I'm working on a blog post, but here's my little proof of concept working on both Windows in a web browser[1] and MacOS in the Finder [2]. [1] https://vimeo.com/931907811 https://vimeo.com/931907811 [2] https://dvt.name/wp-content/uploads/2024/04/image-11.png https://dvt.name/wp-content/uploads/2024/04/image-11.png
- iJohnDoe 2y ago
  Really cool! Do you plan to release it?
qiakai 2y ago
> What other CLI tools are people using to work with LLMs in the terminal? I personally love using x-cmd. Small size (1.1MB), open source, interactive operation [1] https://www.x-cmd.com/ https://www.x-cmd.com/ [2] https://www.x-cmd.com/mod/openai https://www.x-cmd.com/mod/openai
jillesvangurp 2y ago
> We have implemented basic RAG—Retrieval Augmented Generation, where search results are used to answer a question—using a terminal script that scrapes search results from Google and pipes them into an LLM. I love this. Simple and effective. RAG is just search leveled up with LLMs. Such an obvious thing to do. We know how to do search and can use it to unlock vast amounts of knowledge. Instead of letting LLMs dream up facts by compressing all knowledge into them, a better use of them is letting them summarize and reason about the facts it finds. IMHO the art is actually going to be in letting them come up with the right query as well. Or queries. It could be a lot more exhaustive in its searches than we could be.
- RansomStark 2y ago
  While I agree with the sentiment in general, I've came to the conclusion that what I really want is the flexibility of the natural language interface that LLM's provide, and the return of the correct document. No 'reasoning', no summarizing, just better search [0]. The issue with the current generation of models is that they can't reason, they may do very well at pretending to reason, but they can't [1]. Reasoning requires the ability to identify and reuse patterns, and while there has been some advancement in this area [2] with getting models to learn the underlying pattern and rule, it doesn't generalize. This results in models that will happily tell you that a statement is both true and false, and be unable to identify the logical problem with that. Even creating summaries is difficult, and LLM's are more than happy to hallucinate even when summarizing documents, providing incorrect, or entirely made up facts [3]. The general workaround is multiple runs with the same work and averaging the response, but that's a lot of work, and energy. [0] https://win-vector.com/2024/05/21/i-want-flexible-queries-not-rag/ https://win-vector.com/2024/05/21/i-want-flexible-queries-no... [1] https://medium.com/@konstantine_45825/gpt-4-cant-reason-2eab795e2523 https://medium.com/@konstantine_45825/gpt-4-cant-reason-2eab... [2] https://arxiv.org/abs/2405.15071 https://arxiv.org/abs/2405.15071 [3] https://community.openai.com/t/gpt-4o-hallucinating-at-temp-0-unusable-in-production/746750 https://community.openai.com/t/gpt-4o-hallucinating-at-temp-...
  - mark_l_watson 2y ago
    I have an experiment that you can reproduce: use a search API (e.g., Brave or Duckduckgo) and for each result, ask a local LLM to rate it as useful or not useful, then I fetch the entire web pages of ‘useful’ results and ask for summaries made considering the original search query. I like to look at these summaries, and I take one more pass asking for the concatenated summaries to be summarized as a group, asking for concise and de-duplicated final summary. Anyway, I enjoy playing with this and because I am using my own little Python scripts, I can switch models and hack in it easily.
etx 2y ago
[flagged]
jgalt212 2y ago
Every time I see an LLM demo, I'm blown away. Every time I use one for myself, I feel like a fool. I say this because the scraper demo bit looks very neat, but I've been down this path before and I don't want to waste my time getting bad or deceptively incorrect results.
- ziml77 2y ago
  When you're looking at a demo, the results just have to look good enough for the demo to be impressive. When you're actually using it, the results actually need to be good.

141 points by rednafi 2y ago

Aider continues to be the best way to interact with LLMs while coding, and its a command line tool. Copilot is pretty good, but the forced change > commit > QA process that Aider forces you through is really powerful.

behnamoh 2y ago

I wish llm were more stable, but unfortunately things just kept breaking out of the blue without me touching any settings of the program. I often had to reinstall the package. but finally I gave up and implemented my own.

simonw 2y ago
Which plugins are you using? Did you install via pipx or Homebrew or something else?
- behnamoh 2y ago
  I used pipx and used AnthropicAI's plugin to use Claude. After 2 weeks of perfectly working, suddenly I got the "this model is invalid" error. PS: I appreciate the work put into llm tho, it's a neat program I used with my Automator scripts to bring LLMs to macOS before Apple Intelligence was announced. I just wish the stability was not a concern.
  - foobarqux 2y ago
    The plugins seem to need to be reinstalled after every upgrade
  - simonw 2y ago
    That's weird. Were you using llm-claude or llm-claude-3?
- jimmcslim 2y ago
  I followed along with the blog post, but got unstuck with llm-cmd not working on Mac OS. Looks like this PR would fix it https://github.com/simonw/llm-cmd/pull/12 https://github.com/simonw/llm-cmd/pull/12

simonw 2y ago

This was a workshop I gave in my https://llm.datasette.io/ https://llm.datasette.io/ CLI tool. What other CLI tools are people using to work with LLMs in the terminal? There one comment here about https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider and Ollama is probably the most widely used CLI tool at the moment: https://github.com/ollama/ollama/blob/main/README.md#quickstart https://github.com/ollama/ollama/blob/main/README.md#quickst...

throwup238 2y ago
> What other CLI tools are people using to work with LLMs in the terminal? I use aichat: https://github.com/sigoden/aichat https://github.com/sigoden/aichat I especially like the terminal integration where I can type a natural language request at the terminal and press Alt+E to have it converted to a command to run.
- kelvie 2y ago
  I also use aichat. For some reason (well, I know the reason, it's shell quoting) I don't like passing prompts in via command line, so having a simple text-based TUI like a readline prompt is nice.
- mikeqq2024 2y ago
  There is similar ShellOracle: https://github.com/djcopley/ShellOracle https://github.com/djcopley/ShellOracle It is used as Bash widget, where you press CTL-F, describe your command, then press Enter, the generated command will be inserted into your shell prompt.
bt1a 2y ago
(aichat in tangential comment looks much better) Besides aider and ollama, I think shell_gpt https://github.com/TheR1D/shell_gpt https://github.com/TheR1D/shell_gpt is great for quick chats / lookups. Being able to quickly cat files to repl sessions saves a lot of time. I need to integrate distil whisper large v3, aider, and shell_gpt to tidy up a lot of my disjointed LLM use. As someone else mentioned, the commits created by aider allow me to "skip" some intermediate steps that would be required when working on coding tasks with other frameworks.

liamYC 2y ago

This is awesome, thanks for sharing. I find this kind of tool really useful, Aider in particular. I made my own cli tool for interacting with GPT. It’s really useful with the -c flag for generating code especially bash commands I've forgotten https://github.com/ljsimpkin/chat-gpt-cli-tool https://github.com/ljsimpkin/chat-gpt-cli-tool

bagels 2y ago

I wrote one to help with creating command line commands. It just hits openai api with a prompt asking for just a code block to run on bash + whatever is passed in, and then it prints the command out. I wrote it because I can never remember all the weird command args for all the tools. $ bashy find large files over 10 gb find / -type f -size +10G

QuiDortDine 2y ago
Do you have a followup command to execute the suggested command?
- sdf4j 2y ago
  $ bashy evaluate the previous command output
- bagels 2y ago
  No, I want to review or tweak them, just in case it's trying to do something bad. It's usually pretty good though.
  - godelski 2y ago
    If you use zsh and are willing to source your scripts you can do print -z $command And the command will appear on your cli as if you had written it. I don't think you can do this in bash. Interestingly, this is something that seems quite difficult to both google and ask GPT for help. Both get confused and are thinking different questions are being asked. Probably because there are similar more common questions but the subtleties of possible wordings makes it difficult to differentiate.

dvt 2y ago

Fantastic work here! I'm working on a local tool, affectionately called Descartes, which does something similar—but with a spotlight-like UX for the non-hackers out there. I do think that LLMs have the potential to fundamentally change the way we interact with our computers. There's a lot of edge cases (especially when combining it with the inaccurate science of screen readers) but it's pretty mind-blowing when it works. I'm working on a blog post, but here's my little proof of concept working on both Windows in a web browser[1] and MacOS in the Finder [2]. [1] https://vimeo.com/931907811 https://vimeo.com/931907811 [2] https://dvt.name/wp-content/uploads/2024/04/image-11.png https://dvt.name/wp-content/uploads/2024/04/image-11.png

iJohnDoe 2y ago
Really cool! Do you plan to release it?

qiakai 2y ago

> What other CLI tools are people using to work with LLMs in the terminal? I personally love using x-cmd. Small size (1.1MB), open source, interactive operation [1] https://www.x-cmd.com/ https://www.x-cmd.com/ [2] https://www.x-cmd.com/mod/openai https://www.x-cmd.com/mod/openai

jillesvangurp 2y ago

> We have implemented basic RAG—Retrieval Augmented Generation, where search results are used to answer a question—using a terminal script that scrapes search results from Google and pipes them into an LLM. I love this. Simple and effective. RAG is just search leveled up with LLMs. Such an obvious thing to do. We know how to do search and can use it to unlock vast amounts of knowledge. Instead of letting LLMs dream up facts by compressing all knowledge into them, a better use of them is letting them summarize and reason about the facts it finds. IMHO the art is actually going to be in letting them come up with the right query as well. Or queries. It could be a lot more exhaustive in its searches than we could be.

RansomStark 2y ago
While I agree with the sentiment in general, I've came to the conclusion that what I really want is the flexibility of the natural language interface that LLM's provide, and the return of the correct document. No 'reasoning', no summarizing, just better search [0]. The issue with the current generation of models is that they can't reason, they may do very well at pretending to reason, but they can't [1]. Reasoning requires the ability to identify and reuse patterns, and while there has been some advancement in this area [2] with getting models to learn the underlying pattern and rule, it doesn't generalize. This results in models that will happily tell you that a statement is both true and false, and be unable to identify the logical problem with that. Even creating summaries is difficult, and LLM's are more than happy to hallucinate even when summarizing documents, providing incorrect, or entirely made up facts [3]. The general workaround is multiple runs with the same work and averaging the response, but that's a lot of work, and energy. [0] https://win-vector.com/2024/05/21/i-want-flexible-queries-not-rag/ https://win-vector.com/2024/05/21/i-want-flexible-queries-no... [1] https://medium.com/@konstantine_45825/gpt-4-cant-reason-2eab795e2523 https://medium.com/@konstantine_45825/gpt-4-cant-reason-2eab... [2] https://arxiv.org/abs/2405.15071 https://arxiv.org/abs/2405.15071 [3] https://community.openai.com/t/gpt-4o-hallucinating-at-temp-0-unusable-in-production/746750 https://community.openai.com/t/gpt-4o-hallucinating-at-temp-...
- mark_l_watson 2y ago
  I have an experiment that you can reproduce: use a search API (e.g., Brave or Duckduckgo) and for each result, ask a local LLM to rate it as useful or not useful, then I fetch the entire web pages of ‘useful’ results and ask for summaries made considering the original search query. I like to look at these summaries, and I take one more pass asking for the concatenated summaries to be summarized as a group, asking for concise and de-duplicated final summary. Anyway, I enjoy playing with this and because I am using my own little Python scripts, I can switch models and hack in it easily.

etx 2y ago

[flagged]

jgalt212 2y ago

Every time I see an LLM demo, I'm blown away. Every time I use one for myself, I feel like a fool. I say this because the scraper demo bit looks very neat, but I've been down this path before and I don't want to waste my time getting bad or deceptively incorrect results.

ziml77 2y ago
When you're looking at a demo, the results just have to look good enough for the demo to be impressive. When you're actually using it, the results actually need to be good.