4 ms·

Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b ope

by keyle 3y ago

Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.

nilsherzig 3y ago
Thanks :). It's just a lot of prompting and string parsing. There are models like "Hermes-2-Pro-Mistral" (the one from the video) which are trained to work with function signatures and outputting structured text. But at the end it's just strings in > strings out, haha. But its fun (and sometimes frustrating) to use LLMs for flow control (conditions, loops...) inside your programs.
- madacol 3y ago
  Have you considered using grammar sampling?
- keyle 3y ago
  Wow, I didn't know about "Hermes 2 Pro - Mistral 7B", cheers!
  - nilsherzig 3y ago
    It's my go to "structured text model" atm. Try "starling-ml-beta" (7b) for some very impressive chat capabilities. I honestly think that it outperforms GPT3 half the time.
    - peter_l_downs 3y ago
      Sorry to repeat the same question I just asked the other commenter in this thread, but could you link the model page and recommend a specific level of quantization for the models you've referenced? I'd love to play with these models and see what you're talking about.
      - BOOSTERHIDROGEN 3y ago
        It's from nous research https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B Q5 is minimum.
        peter_l_downs 3y ago
        Thank you — from that page, at the bottom, I was able to find this link to what I think are the quantized versions https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/tree/main https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-... If you have the time, could you explain what you mean by "Q5 is minimum"? Did you determine that by trying the different models and finding this one is best, or did someone else do that evaluation, or is that just generally accepted knowledge? Sorry, I find this whole ecosystem quite confusing still, but I'm very new and that's not your problem.
        BOOSTERHIDROGEN 3y ago
        It's the best balance if you have limited compute performance.
        peter_l_downs 3y ago
        Thank you
        d-z-m 3y ago
        Talking GGUF, Usually the higher you can afford to go wrt. quantization(e.g. Q5 is better than Q4, etc), the better. A Q6_K has minimal performance loss from the Q8, so in most cases if you can fit a Q6_K it's recommended to just use that. TheBloke's READMEs[0] usually have a good table summarizing each quantization level. If you're RAM constrained, you'll also have to make trade-offs about the context length. e.g. you could have 8 GB RAM and a Q5 quant with shorter context, vs Q3 with longer, etc. [0]:https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF
- davidcollantes 3y ago
  Got a link for that one? I have found a few with Hermes-2-Mistral in the name.
  - deleted 3y ago
    [deleted]
viksit 3y ago
curious what hardware you use? and is any of this runnable on an m1 laptop?
- keyle 3y ago
  Absolutely, 7B will run comfortably on 16GB of RAM and most consumer level hardware. Some of the 40B run on 32GB, but it depends on the model I found (GGUF, crossing fingers help). I ran this originally on a M1 with 32GB, I run this on an Air M2 with 16GB (and mac mini M2 32GB), no problem. I use llama.cpp with a SwiftUI interface (my own), all native, no scripts python/js/web. 7b is obviously less capable but the instant response makes it worth exploring. It's very useful as a Google search replacement that is instantly more valuable, for general questions, than dealing with the hellscape of blog spam ruling Google atm. Note, for my complex code queries at $dayjob where time is of the essence, I still use GPT4 plus, which is still unmatched imho, without running special hardware at least.
  - regularfry 3y ago
    I've been occasionally using a 7b Q4 quant on llama.cpp on an 8GB M1. It's usable, if not amazing.
- nilsherzig 3y ago
  Depends on your m1 specs, but should definitely be able to run a 7b model (at least with some quantization).
peter_l_downs 3y ago
I'm just starting to get into downloading and testing models using llama.cpp and I'm curious which model you're actually using, since they seem to come in varying levels of quantization. Is this [0] the model page for the one you're using, or should I be looking somewhere else? What is the actual file name of the model you're using? [0] https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GG...
windexh8er 3y ago
Have you looked into tools like CrewAI [0]? [0] https://www.crewai.io/ https://www.crewai.io/

by keyle 3y ago

Impressive, I don't think I've seen a local model call upon specialised modules yet (although I can't keep up with everything going on). I too use local 7b open-hermes and it's really good.

nilsherzig 3y ago

Thanks :). It's just a lot of prompting and string parsing. There are models like "Hermes-2-Pro-Mistral" (the one from the video) which are trained to work with function signatures and outputting structured text. But at the end it's just strings in > strings out, haha. But its fun (and sometimes frustrating) to use LLMs for flow control (conditions, loops...) inside your programs.

madacol 3y ago
Have you considered using grammar sampling?
keyle 3y ago
Wow, I didn't know about "Hermes 2 Pro - Mistral 7B", cheers!
- nilsherzig 3y ago
  It's my go to "structured text model" atm. Try "starling-ml-beta" (7b) for some very impressive chat capabilities. I honestly think that it outperforms GPT3 half the time.
  - peter_l_downs 3y ago
    Sorry to repeat the same question I just asked the other commenter in this thread, but could you link the model page and recommend a specific level of quantization for the models you've referenced? I'd love to play with these models and see what you're talking about.
    - BOOSTERHIDROGEN 3y ago
      It's from nous research https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B Q5 is minimum.
      - peter_l_downs 3y ago
        Thank you — from that page, at the bottom, I was able to find this link to what I think are the quantized versions https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/tree/main https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-... If you have the time, could you explain what you mean by "Q5 is minimum"? Did you determine that by trying the different models and finding this one is best, or did someone else do that evaluation, or is that just generally accepted knowledge? Sorry, I find this whole ecosystem quite confusing still, but I'm very new and that's not your problem.
        BOOSTERHIDROGEN 3y ago
        It's the best balance if you have limited compute performance.
        peter_l_downs 3y ago
        Thank you
        d-z-m 3y ago
        Talking GGUF, Usually the higher you can afford to go wrt. quantization(e.g. Q5 is better than Q4, etc), the better. A Q6_K has minimal performance loss from the Q8, so in most cases if you can fit a Q6_K it's recommended to just use that. TheBloke's READMEs[0] usually have a good table summarizing each quantization level. If you're RAM constrained, you'll also have to make trade-offs about the context length. e.g. you could have 8 GB RAM and a Q5 quant with shorter context, vs Q3 with longer, etc. [0]:https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF
davidcollantes 3y ago
Got a link for that one? I have found a few with Hermes-2-Mistral in the name.
- deleted 3y ago
  [deleted]

viksit 3y ago

curious what hardware you use? and is any of this runnable on an m1 laptop?

keyle 3y ago
Absolutely, 7B will run comfortably on 16GB of RAM and most consumer level hardware. Some of the 40B run on 32GB, but it depends on the model I found (GGUF, crossing fingers help). I ran this originally on a M1 with 32GB, I run this on an Air M2 with 16GB (and mac mini M2 32GB), no problem. I use llama.cpp with a SwiftUI interface (my own), all native, no scripts python/js/web. 7b is obviously less capable but the instant response makes it worth exploring. It's very useful as a Google search replacement that is instantly more valuable, for general questions, than dealing with the hellscape of blog spam ruling Google atm. Note, for my complex code queries at $dayjob where time is of the essence, I still use GPT4 plus, which is still unmatched imho, without running special hardware at least.
- regularfry 3y ago
  I've been occasionally using a 7b Q4 quant on llama.cpp on an 8GB M1. It's usable, if not amazing.
nilsherzig 3y ago
Depends on your m1 specs, but should definitely be able to run a 7b model (at least with some quantization).

peter_l_downs 3y ago

I'm just starting to get into downloading and testing models using llama.cpp and I'm curious which model you're actually using, since they seem to come in varying levels of quantization. Is this [0] the model page for the one you're using, or should I be looking somewhere else? What is the actual file name of the model you're using? [0] https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GG...

windexh8er 3y ago

Have you looked into tools like CrewAI [0]? [0] https://www.crewai.io/ https://www.crewai.io/