3 ms·

I'm running it on a pure cpu 2020 model ryzen server with 2x128 GB RAM with llama.cpp, it seems as intelligent as gpt4. I optimized a little bit by forcing it t

I'm running it on a pure cpu 2020 model ryzen server with 2x128 GB RAM with llama.cpp, it seems as intelligent as gpt4. I optimized a little bit by forcing it to run on a single RAM stick and tuning llama.cpp build parameters, going from 3-5 tok/s to a more acceptable 5-8 tok/s. It can call tools and reason adequately enough to use them when appropriate.

ramgine 8mo ago
Does keeping it on one stick make it more performant? I have a epyc server with 1tb of 64 gig sticks and a 3060, looking to maximize what models I can run on it
- nurettin 8mo ago
  In my case it definitely helps, but keeping 64k context window would probably require more than 64 GB.