5 ms·

I run in Linux system it via cli using Ollama, very easy to setup https://ollama.ai/library/mistral https://ollama.ai/library/mistral curl https://ollama.ai/i

I run in Linux system it via cli using Ollama, very easy to setup https://ollama.ai/library/mistral https://ollama.ai/library/mistral curl https://ollama.ai/install.sh https://ollama.ai/install.sh | sh ollama run mistral:text

traverseda 3y ago
Huh, I've fought with a few of these things on my laptop, no nvidia GPU, limited ram, etc. This actually worked as advertised.
- 3abiton 3y ago
  What's the performance like (quality and speed wise)?
  - all2 3y ago
    Mistral 7b is serviceable for short contexts. If you have a longer conversation, token generation can start to lag a lot.
  - tarruda 3y ago
    Yesterday I tried mixtral 7bx8 running on the CPU. With an Intel 11th gen chip and 64gb DDR4 at 3200mhz, I got around 2-4 tokens/second in a small context, this gets progressively slower as the context grows. You would get a much better experience with apple silicon and lots of RAM
    - yobanate 3y ago
      Can confirm. My M3 Max gets about 22t/s, putting the bottleneck BKAC.
      - 3abiton 3y ago
        That's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?
        bugglebeetle 3y ago
        Unified memory and optimizations in llama.cpp (which Ollama wraps).
        ithkuil 3y ago
        Is that using the GPU?
        bugglebeetle 3y ago
        It can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.
bugglebeetle 3y ago
Yeah, I was surprised Ollama was not mentioned as it’s by far the easiest to get started with. If it only had real grammar support, I’d never have to use another library again (it has a JSON mode that generally works, however).
- Tomte 3y ago
  What is grammar support? I've seen that mentioned several times now. Does it allow to restrict the output to a given template, or am I totally wrong there?
  - biwills 3y ago
    Yes. Here's a quick rundown of grammar in llama.cpp (link on the docs of faraday.dev which runs llama.cpp under the hood) https://docs.faraday.dev/character-creation/grammars https://docs.faraday.dev/character-creation/grammars
vunderba 3y ago
I want to like Ollama, but I wish it didn't obfuscate the actual directives (full prompt) that it sends to the underlying model. Ollama uses a custom templatizing script in its Modelfiles to translate user input into the format that a specific model expects ([INST], etc), but it can be difficult to tell if it's working as expected because it won't show up in the logs at all. Other than that it's a great project - very easy to get started and has a solid API implementation. I've got it running on both a Win 10 + WSL2 docker and on a Mac M1.
- coffeeri 3y ago
  You can bypass the templating in raw mode, by setting the request parameter `raw` to true. https://github.com/jmorganca/ollama/blob/main/docs%2Fapi.md https://github.com/jmorganca/ollama/blob/main/docs%2Fapi.md
  - vunderba 3y ago
    yeah I guess I could compare the output at 0.0 temperature with same prompt using Modelfile and then after using raw mode with my best guess of how the modelfile is creating the raw data it's passing to Ollama. I'd push a PR to the repo itself but I have zero experience with Go...
giblaz 3y ago
Ollama is great. I discovered it today while looking for a way to serve LLMs locally for my terminal command generator tool (cmdh: https://github.com/pgibler/cmdh https://github.com/pgibler/cmdh) and was able to get it up and running and implement support for it very easily.

traverseda 3y ago

Huh, I've fought with a few of these things on my laptop, no nvidia GPU, limited ram, etc. This actually worked as advertised.

3abiton 3y ago
What's the performance like (quality and speed wise)?
- all2 3y ago
  Mistral 7b is serviceable for short contexts. If you have a longer conversation, token generation can start to lag a lot.
- tarruda 3y ago
  Yesterday I tried mixtral 7bx8 running on the CPU. With an Intel 11th gen chip and 64gb DDR4 at 3200mhz, I got around 2-4 tokens/second in a small context, this gets progressively slower as the context grows. You would get a much better experience with apple silicon and lots of RAM
  - yobanate 3y ago
    Can confirm. My M3 Max gets about 22t/s, putting the bottleneck BKAC.
    - 3abiton 3y ago
      That's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?
      - bugglebeetle 3y ago
        Unified memory and optimizations in llama.cpp (which Ollama wraps).
        ithkuil 3y ago
        Is that using the GPU?
        bugglebeetle 3y ago
        It can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.

bugglebeetle 3y ago

Yeah, I was surprised Ollama was not mentioned as it’s by far the easiest to get started with. If it only had real grammar support, I’d never have to use another library again (it has a JSON mode that generally works, however).

Tomte 3y ago
What is grammar support? I've seen that mentioned several times now. Does it allow to restrict the output to a given template, or am I totally wrong there?
- biwills 3y ago
  Yes. Here's a quick rundown of grammar in llama.cpp (link on the docs of faraday.dev which runs llama.cpp under the hood) https://docs.faraday.dev/character-creation/grammars https://docs.faraday.dev/character-creation/grammars

vunderba 3y ago

I want to like Ollama, but I wish it didn't obfuscate the actual directives (full prompt) that it sends to the underlying model. Ollama uses a custom templatizing script in its Modelfiles to translate user input into the format that a specific model expects ([INST], etc), but it can be difficult to tell if it's working as expected because it won't show up in the logs at all. Other than that it's a great project - very easy to get started and has a solid API implementation. I've got it running on both a Win 10 + WSL2 docker and on a Mac M1.

coffeeri 3y ago
You can bypass the templating in raw mode, by setting the request parameter `raw` to true. https://github.com/jmorganca/ollama/blob/main/docs%2Fapi.md https://github.com/jmorganca/ollama/blob/main/docs%2Fapi.md
- vunderba 3y ago
  yeah I guess I could compare the output at 0.0 temperature with same prompt using Modelfile and then after using raw mode with my best guess of how the modelfile is creating the raw data it's passing to Ollama. I'd push a PR to the repo itself but I have zero experience with Go...

giblaz 3y ago

Ollama is great. I discovered it today while looking for a way to serve LLMs locally for my terminal command generator tool (cmdh: https://github.com/pgibler/cmdh https://github.com/pgibler/cmdh) and was able to get it up and running and implement support for it very easily.