5 ms·

Even Google is making a llama.cpp wrapper. Technically a wrapper of a wrapper, though llama-cpp-python is quite excellent... TBH I would just recommend using t

Even Google is making a llama.cpp wrapper. Technically a wrapper of a wrapper, though llama-cpp-python is quite excellent... TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).

andy99 3y ago
Oh it's a wrapper? I just skimmed and thought they reinvented llama.cpp for some reason, which would have been mildly interesting.
- williamstein 3y ago
  (edit: this is completely irrelevant remark. sorry) It's (1) written in Python, and (2) ollama appears nowhere in their Docker repo [1], and (3) it gets the models only from huggingface instead of the ollama repos. Thus maybe it is NOT a wrapper of ollama? [1] https://github.com/googlecloudplatform/localllm https://github.com/googlecloudplatform/localllm
  - andy99 3y ago
    https://github.com/GoogleCloudPlatform/localllm/blob/main/llm-tool/setup.py#L30 https://github.com/GoogleCloudPlatform/localllm/blob/main/ll... I see llama.cpp in there, but I didn't go deeper than that. Ollama is not the same thing.
thund 3y ago
does Functionary work with generic models like llama.cpp? had the impression you could only use Functionary GGUFs
- gbickford 3y ago
  Llama.cpp is an inference engine. The author of llama.cpp designed gguf. Funcionary is a model that does function calling. You can download functionary weights in the gguf format and then run it using llama.cpp on low-end machines using CPU or GPU or a mix of both.
  - thund 3y ago
    ok so the answer is no, thanks
    - behnamoh 3y ago
      The GP educated you and you still said the answer was "no"? It's clearly "yes".
      - TeMPOraL 3y ago
        I read the original question as "is the 'Functionary' thing something that works with any LLM you can run in llama.cpp, not just only with the special 'Functionary' finetuned models?", to which the answer now is clearly no. Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.
- brucethemoose2 3y ago
  This question is confusing to me. Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean. Hence, it can be turned into a gguf like any other llama finetune. It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.
  - thund 3y ago
    from the parent: > TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary). I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models? The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself” :-)
discardedrefuse 3y ago
Slightly off topic, here is the best local llama.cpp wrapper I've run into: https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram for it). I was running 7B models with ease on an old FX8350 and now 13B models on a 5600X (32GB RAM on both machines). This wrapper spins up a local web server that runs a simple web frontend to use immediately with no code, but also exposes an OpenAI compatible API for dev work and alt frontends (like SillyTavern).
- 3abiton 3y ago
  What's the speed? I heard of llamafile before.
  - discardedrefuse 3y ago
    I wasn't really timing it, but it felt fine, especially since it prints out while generating. The 13B model on the 5600X is about 10 seconds. Tho once the conversation gets too long (about 10 replies) it will go sideways. I gotta play with the default settings. IIRC the 7B on the Fx8350 was closer to 25 seconds per response.
m00x 3y ago
I wonder what need they're fulfilling here. It's not really more useful than using llamacpp-python. This is not something old Google would even release.
- bigiain 3y ago
  > I wonder what need they're fulfilling here. The need to fill more seats on the AI hype train. (Someone's probably got "release AI software as Google open source" as an OKR)

andy99 3y ago

Oh it's a wrapper? I just skimmed and thought they reinvented llama.cpp for some reason, which would have been mildly interesting.

williamstein 3y ago
(edit: this is completely irrelevant remark. sorry) It's (1) written in Python, and (2) ollama appears nowhere in their Docker repo [1], and (3) it gets the models only from huggingface instead of the ollama repos. Thus maybe it is NOT a wrapper of ollama? [1] https://github.com/googlecloudplatform/localllm https://github.com/googlecloudplatform/localllm
- andy99 3y ago
  https://github.com/GoogleCloudPlatform/localllm/blob/main/llm-tool/setup.py#L30 https://github.com/GoogleCloudPlatform/localllm/blob/main/ll... I see llama.cpp in there, but I didn't go deeper than that. Ollama is not the same thing.

thund 3y ago

does Functionary work with generic models like llama.cpp? had the impression you could only use Functionary GGUFs

gbickford 3y ago
Llama.cpp is an inference engine. The author of llama.cpp designed gguf. Funcionary is a model that does function calling. You can download functionary weights in the gguf format and then run it using llama.cpp on low-end machines using CPU or GPU or a mix of both.
- thund 3y ago
  ok so the answer is no, thanks
  - behnamoh 3y ago
    The GP educated you and you still said the answer was "no"? It's clearly "yes".
    - TeMPOraL 3y ago
      I read the original question as "is the 'Functionary' thing something that works with any LLM you can run in llama.cpp, not just only with the special 'Functionary' finetuned models?", to which the answer now is clearly no. Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.
brucethemoose2 3y ago
This question is confusing to me. Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean. Hence, it can be turned into a gguf like any other llama finetune. It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.
- thund 3y ago
  from the parent: > TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary). I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models? The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself” :-)

discardedrefuse 3y ago

Slightly off topic, here is the best local llama.cpp wrapper I've run into: https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram for it). I was running 7B models with ease on an old FX8350 and now 13B models on a 5600X (32GB RAM on both machines). This wrapper spins up a local web server that runs a simple web frontend to use immediately with no code, but also exposes an OpenAI compatible API for dev work and alt frontends (like SillyTavern).

3abiton 3y ago
What's the speed? I heard of llamafile before.
- discardedrefuse 3y ago
  I wasn't really timing it, but it felt fine, especially since it prints out while generating. The 13B model on the 5600X is about 10 seconds. Tho once the conversation gets too long (about 10 replies) it will go sideways. I gotta play with the default settings. IIRC the 7B on the Fx8350 was closer to 25 seconds per response.

m00x 3y ago

I wonder what need they're fulfilling here. It's not really more useful than using llamacpp-python. This is not something old Google would even release.

bigiain 3y ago
> I wonder what need they're fulfilling here. The need to fill more seats on the AI hype train. (Someone's probably got "release AI software as Google open source" as an OKR)