3 ms·
There are certainly challenges. When setting up a new model, I get AI to walk me through the commands using llama-benchmark that determine the best parameters f
by drittich 2mo ago
There are certainly challenges. When setting up a new model, I get AI to walk me through the commands using llama-benchmark that determine the best parameters for my particular configuration and needs. Once you've got that it's pretty easy to port those parameters to llama-server. It takes me about an hour to run through this process. It would be great if there was a registry of hardware, models, configuration parameters, and resulting tokens per second. Maybe one day we'll get there.
- kevin42 2mo agoWhat kind of parameters do you end up changing, and how much difference does it make. Perhaps I am missing something and get more tok/sec, but I usually just do a git pull, then rebuild the latest whenever I get a new model. In the past, I had to play with chat templates for some models to work with agents for tool calling. But I've never had to do anything other than specify the model, and tweaking the context size in some cases.
- drittich 2mo agoThings like the number of GPU layers, whether to use flash attention or not, enabling MTP, playing with context size.
- purplemoonx 2mo ago[dead]