4 ms·
Locally running, like, llama.cpp? Or Python? Either way I guess. I would have thought this was impossible, I contribute to llama.cpp and there's an awful lot
by refulgentis 2y ago
Locally running, like, llama.cpp? Or Python?
Either way I guess.
I would have thought this was impossible, I contribute to llama.cpp and there's an awful lot of per-model ugliness to make things work, even just in terms of "get it the tool calls in the form it expects."
cries at the Phi-4 PR in the other window that I'm still working on, and discovering new things, 4 weeks later
- bearjaws 2y agoI am not sure about llama.cpp but it works with Ollama. That's not to say won't need to tweak things when you cut down to smaller models, there are always trade offs swapping models.
- dragonwriter 2y ago> Locally running, like, llama.cpp? Or Python? I’d guess it supports a small set of popular HTTP APIs (particularly, its very common for self-hosting LLM toolkits—not the per-model reference implementations or low-level console frontends—to present an OpenAI compatible API), so you could support a very wide range of local models, through a variety of toolkits, just by supporting the OpenAI API with configurable endpoint addresses.
- BoorishBears 2y agoAt some point someone tried a toy direct integration, but the only actually supported way is via a Python library that wraps llama.cpp in an OpenAI compatible API endpoint.