3 ms·

does Functionary work with generic models like llama.cpp? had the impression you could only use Functionary GGUFs

by thund 3y ago

gbickford 3y ago
Llama.cpp is an inference engine. The author of llama.cpp designed gguf. Funcionary is a model that does function calling. You can download functionary weights in the gguf format and then run it using llama.cpp on low-end machines using CPU or GPU or a mix of both.
- thund 3y ago
  ok so the answer is no, thanks
  - behnamoh 3y ago
    The GP educated you and you still said the answer was "no"? It's clearly "yes".
    - TeMPOraL 3y ago
      I read the original question as "is the 'Functionary' thing something that works with any LLM you can run in llama.cpp, not just only with the special 'Functionary' finetuned models?", to which the answer now is clearly no. Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.
brucethemoose2 3y ago
This question is confusing to me. Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean. Hence, it can be turned into a gguf like any other llama finetune. It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.
- thund 3y ago
  from the parent: > TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary). I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models? The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself” :-)

does Functionary work with generic models like llama.cpp? had the impression you could only use Functionary GGUFs

by thund 3y ago

does Functionary work with generic models like llama.cpp? had the impression you could only use Functionary GGUFs

Llama.cpp is an inference engine. The author of llama.cpp designed gguf. Funcionary is a model that does function calling. You can download functionary weights in the gguf format and then run it using llama.cpp on low-end machines using CPU or GPU or a mix of both.

thund 3y ago
ok so the answer is no, thanks
- behnamoh 3y ago
  The GP educated you and you still said the answer was "no"? It's clearly "yes".
  - TeMPOraL 3y ago
    I read the original question as "is the 'Functionary' thing something that works with any LLM you can run in llama.cpp, not just only with the special 'Functionary' finetuned models?", to which the answer now is clearly no. Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.

brucethemoose2 3y ago

This question is confusing to me. Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean. Hence, it can be turned into a gguf like any other llama finetune. It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.

thund 3y ago
from the parent: > TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary). I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models? The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself” :-)