5 ms·
The first number I look at these days is the file size via Ollama, which for this model is 14GB https://ollama.com/library/devstral/tags https://ollama.com/libr
by simonw 1y ago
The first number I look at these days is the file size via Ollama, which for this model is 14GB https://ollama.com/library/devstral/tags https://ollama.com/library/devstral/tags
I find that on my M2 Mac that number is a rough approximation to how much memory the model needs (usually plus about 10%) - which matters because I want to know how much RAM I will have left for running other applications.
Anything below 20GB tends not to interfere with the other stuff I'm running too much. This model looks promising!
- lis 1y agoYes, I agree. I've just ran the model locally and it's making a good impression. I've tested it with some ruby/rspec gotchas, which it handled nicely. I'll give it a try with aider to test the large context as well.
- ericb 1y agoIn ollama, how do you set up the larger context, and figure out what settings to use? I've yet to find a good guide. I'm also not quite sure how I should figure out what those settings should be for each model. There's context length, but then, how does that relate to input length and output length? Should I just make the numbers match? 32k is 32k? Any pointers?
- lis 1y agoFor aider and ollama, see: https://aider.chat/docs/llms/ollama.html https://aider.chat/docs/llms/ollama.html Just for ollama, see: https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size https://github.com/ollama/ollama/blob/main/docs/faq.md#how-c... I’m using llama.cpp though, so I can’t confirm these methods.
- nico 1y agoAre you using it with aider? If so, how has your experience been?
- zackify 1y agoOllama breaks for me. If I manually set the context higher. The next api call from clone resets it back. And ollama keeps taking it out of memory every 4 minutes. LM studio with MLX on Mac is performing perfectly and I can keep it in my ram indefinitely. Ollama keep alive is broken as a new rest api call resets it after. I’m surprised it’s this glitched with longer running calls and custom context length.
- nico 1y agoAny agentic dev software you could recommend that runs well with local models? I’ve been using Cursor and I’m kind of disappointed. I get better results just going back and forth between the editor and ChatGPT I tried localforge and aider, but they are kinda slow with local models
- ynniv 1y agohttps://github.com/block/goose https://github.com/block/goose
- jabroni_salad 1y agoDo you have any other interface for the model? what kind of tokens/sec are you getting? Try hooking aider up to gemini and see how the speed is. I have noticed that people in the localllama scene do not like to talk about their TPS.
- nico 1y agoThe models feel pretty snappy when interacting with them directly via ollama, not sure about the TPS However I've also ran into 2 things: 1) most models don't support tools, sometimes it's hard to find a version of the model that correctly uses tools, 2) even with good TPS, since the agents are usually doing chain-of-thought and running multiple chained prompts, the experience feels slow - this is even true with Cursor using their models/apis
- segmondy 1y agoPeople have all sorts of hardware, TPS is meaningless without the full spec of the hardware, and GPU is not the only thing, CPU, ram speed, memory channel, PCIe speed, inference software, partial CPU offload? RPC? even OS, all of these things add up. So if someone tells you TPS for a given model, it's meaningless unless you understand their entire setup.
- zackify 1y agoI used devstral today with cline and open hands. Worked great in both. About 1 minute initial prompt processing time on an m4 max Using LM studio because the ollama api breaks if you set the context to 128k.
- rahimnathwani 1y agoAlmost all models listed in the ollama model library have a version that's under 20GB. But whether that's a 4-bit quantization (as in this case) or more/fewer bits varies. AFAICT they usually set the default tag to sa version around 15GB.
- davedx 1y agoI couldn’t run it on my 16gb MBP (I tried, it just froze up, probably lots of swapping), they say it needs 32gb
- deleted 1y ago[deleted]
- ics 1y agoI was able to run it on my M2 Air with 24GB. Startup was very slow but less than 10 minutes. After that responses were reasonably quick. Edit: I should point out that I had many other things open at the time. Mail, Safari, Messages, and more. I imagine startup would be quicker otherwise but it does mean you can run with less than 32GB.