3 ms·

Can the base URL be overridden so I can point it at eg Ollama or any other OpenAI compatible endpoint? I’d love to use this with local LLMs, for the speed and p

by dymk 11mo ago

Can the base URL be overridden so I can point it at eg Ollama or any other OpenAI compatible endpoint? I’d love to use this with local LLMs, for the speed and privacy boost.

jedbrooke 11mo ago
https://github.com/chr15m/runprompt/blob/main/runprompt#L9 https://github.com/chr15m/runprompt/blob/main/runprompt#L9 seems like it would be, just swap the openai url here or add a new one
chr15m 11mo ago
Good idea. Will figure out a way to do this.
- benatkin 11mo ago
  Perhaps instead of writing an llm abstraction layer, you could use a lightweight one, such as @simonw's llm.
  - chr15m 11mo ago
    I don't want to introduce a dependency. Simon's tool is great but I don't like the way it stores template state. I want my state in a file in my working folder.
    - threecheese 11mo ago
      Can you explain this decision a bit more? I’m using ‘llm’ and I find your project interesting.
      - chr15m 11mo ago
        llm stores data (prompts, responses, chats, fragments, aliases, attachment metadta) in a central sqlite database outside the working directory, and you have to use the tool to view and manipulate that data. I prefer a tool like this to default to storing things in a file or files in the project directory I'm working in, in a way that is legible e.g. plain text files. Contrast with e.g. git where everything goes into .git. Functions require you to specify them on the command line every time they're invoked. I would prefer a tool like this to default to reading the functions from a hierarchy where it reads e.g. .llm-functions in the current folder, then ~/.config/llm-functions or something like that. In general I found myself baffled when trying to figure out where and how to configure things. That's probably me being impatient but I have found other tools to have more straightforward setup and less indirection. Basically I like things to be less centralized, magic, and less controlled by the tool. Another thing, which is not the fault of llm at all, is I find Python based tools annoying to install. I have to remember the env where I set them up. Contrast with a golang application which is generally a single file I can put in ~/bin. That's the reason I don't want to introduce a dep to runprompt if I can avoid it. The final thing that I found frustrating was the name 'llm' which makes it difficult to conduct searches as it is the generic name for what the thing is. It is an amazing piece of engineering and I am a huge fan of simonw's work, but I don't use llm much for these reasons.
- khimaros 11mo ago
  simple solution: honor OPENAI_API_BASE env var
chr15m 11mo ago
I've implemented this now. You can set it with BASE_URL or OPENAI_BASE_URL which seems to vaguely be the standard. I also plan to use this with local LLMs. Thanks for the suggestion!

by dymk 11mo ago

Can the base URL be overridden so I can point it at eg Ollama or any other OpenAI compatible endpoint? I’d love to use this with local LLMs, for the speed and privacy boost.

jedbrooke 11mo ago

https://github.com/chr15m/runprompt/blob/main/runprompt#L9 https://github.com/chr15m/runprompt/blob/main/runprompt#L9 seems like it would be, just swap the openai url here or add a new one

chr15m 11mo ago

Good idea. Will figure out a way to do this.

benatkin 11mo ago
Perhaps instead of writing an llm abstraction layer, you could use a lightweight one, such as @simonw's llm.
- chr15m 11mo ago
  I don't want to introduce a dependency. Simon's tool is great but I don't like the way it stores template state. I want my state in a file in my working folder.
  - threecheese 11mo ago
    Can you explain this decision a bit more? I’m using ‘llm’ and I find your project interesting.
    - chr15m 11mo ago
      llm stores data (prompts, responses, chats, fragments, aliases, attachment metadta) in a central sqlite database outside the working directory, and you have to use the tool to view and manipulate that data. I prefer a tool like this to default to storing things in a file or files in the project directory I'm working in, in a way that is legible e.g. plain text files. Contrast with e.g. git where everything goes into .git. Functions require you to specify them on the command line every time they're invoked. I would prefer a tool like this to default to reading the functions from a hierarchy where it reads e.g. .llm-functions in the current folder, then ~/.config/llm-functions or something like that. In general I found myself baffled when trying to figure out where and how to configure things. That's probably me being impatient but I have found other tools to have more straightforward setup and less indirection. Basically I like things to be less centralized, magic, and less controlled by the tool. Another thing, which is not the fault of llm at all, is I find Python based tools annoying to install. I have to remember the env where I set them up. Contrast with a golang application which is generally a single file I can put in ~/bin. That's the reason I don't want to introduce a dep to runprompt if I can avoid it. The final thing that I found frustrating was the name 'llm' which makes it difficult to conduct searches as it is the generic name for what the thing is. It is an amazing piece of engineering and I am a huge fan of simonw's work, but I don't use llm much for these reasons.
khimaros 11mo ago
simple solution: honor OPENAI_API_BASE env var

chr15m 11mo ago

I've implemented this now. You can set it with BASE_URL or OPENAI_BASE_URL which seems to vaguely be the standard. I also plan to use this with local LLMs. Thanks for the suggestion!