3 ms·
I added a Python library API to my LLM CLI tool recently which offers a very lightweight way to call models: https://llm.datasette.io/en/stable/python-api.html
by simonw 3y ago
I added a Python library API to my LLM CLI tool recently which offers a very lightweight way to call models: https://llm.datasette.io/en/stable/python-api.html https://llm.datasette.io/en/stable/python-api.html
import llm
model = llm.get_model("gpt-3.5-turbo")
model.key = 'YOUR_API_KEY_HERE'
response = model.prompt(
"Five surprising names for a pet pelican"
)
print(response.text())
Or you can stream the responses like this:
response = model.prompt(
"Five diabolical names for a pet goat"
)
for chunk in response:
print(chunk, end="")
It works with other models too, installed via plugins - including models that can run directly on your machine:
pip install llm-gpt4all
Then:
model = llm.get_model("ggml-vicuna-7b-1")
print(model.prompt(
"What is the capital of France?"
).text())
It also handles conversations, where each prompt needs to include the previous context of the conversation:
c = model.conversation()
print(c.prompt("Capital of France?").text())
print(c.prompt("what language do they speak?").text())
I wrote more about the new plugin system for adding extra models here: https://simonwillison.net/2023/Jul/12/llm/ https://simonwillison.net/2023/Jul/12/llm/
- sitkack 3y agoWhy is the conversation tied to the model?
- simonw 3y agoGood question! It's because there's an aspect of conversations that differs between different models: the way the previous messages are injected into the context of the prompt. I went back and forth on a bunch of different designs, but eventually decided to try to make it so that each plugin that implemented a new model would only have to subclass Model and add new methods. You can see all of the design arguments I had with myself about this here, across the course of 129 pull requests comments: https://github.com/simonw/llm/pull/65 https://github.com/simonw/llm/pull/65
- brianjking 3y agoSuper pumped about this, Simon. Will be trying it this weekend. Any of the gpt4all models work, but need to duplicate the model files for now, right?
- simonw 3y agoYeah I haven't figured out how to have it reuse the models from the desktop GPT4All installation yet, issue here: https://github.com/simonw/llm-gpt4all/issues/5 https://github.com/simonw/llm-gpt4all/issues/5
- bspammer 3y agoThis looks wonderful, a similar breath of fresh air to using the requests library for the first time. Really impressed by the amount of documentation too. Is support for embedding and querying a corpus of custom text planned at all? 99% of what I wanted to use langchain for was building a chatbot that can answer questions about my own documents.
- simonw 3y agoUndecided yet, but I think there's a good chance embedding stuff will eventually show up as an LLM plugin.
- rcarmo 3y agoNice. Now all we need is a vector database atop SQLite.
- bread90 3y agoJust implemented pgvector in an existing psql db, works quite well
- simonw 3y agoI had a go at one of those a few months ago: https://datasette.io/plugins/datasette-faiss https://datasette.io/plugins/datasette-faiss Alex Garcia built a better one here as a SQLite Rust extension: https://github.com/asg017/sqlite-vss https://github.com/asg017/sqlite-vss