3 ms·

I don’t use local LLMs for CoPilot-like functionality, but I have toyed with the concept. There are a few things to keep in mind: no programmer that I know is

I don’t use local LLMs for CoPilot-like functionality, but I have toyed with the concept. There are a few things to keep in mind: no programmer that I know is sitting there typing code for hours at a time without stopping. There’s a lot more to being a developer than just typing, whether it is debugging, thinking, JIRA, Slack, or whatever else. These CoPilot-like tools will only activate after you type something, then stop for a defined timeout period. While you’re typing, they do nothing. After they generate, they do nothing. I would honestly be surprised if the GPU active time was more than 10% averaged over an hour. When actively working on a large LLM, the RTX 3090 is drawing close to 400W in my desktop. At a 10% duty cycle (active time), that would be 40W on average, which would be 320Wh over the course of a full 8-hour day of crazy productivity. My electric rate is about 15¢/kWh, so that would be about 5¢ per day. It is absolutely not running at a 100% duty cycle, and it’s absurd to even do the math for that, but we can multiply by 10 and say that if you’re somehow a mythical “10x developer” then it would be 50¢/day in electricity here. I think 5¢/day to 10¢/day is closer to reality. Either way, the cost is marginal at the scale of a software developer’s salary.

sorenjan 3y ago
That sounds perfectly reasonable. I'm more worried about noise and heat than the cost though, but I guess that's not too bad either then. What's the latency like? When I've used generative image models the programs unload the model after they're done, so it takes a while to generate the next image. Is the model sitting in VRAM when it's idle?
- coder543 3y ago
  Fan noise isn’t very much, and you can always limit the max clockspeeds on a GPU (and/or undervolt it) to be quieter and more efficient at a cost of a small amount of performance. The RTX 3090 still seems to be faster than the M3 Max for LLMs that fit on the 3090, so giving up a little performance for near-silent operation wouldn’t be a big loss. Ollama caches the last used model in memory for a few minutes, then unloads it if it hasn’t been used in that time to free up VRAM. I think they’re working on making this period configurable. Latency is very good in my experience, but I haven’t used the local code completion stuff much, just a few quick experiments on personal projects, so my experience with that aspect is limited. If I ever have a job that encourages me to use my own LLM server, I would certainly consider using it more for that.

sorenjan 3y ago

That sounds perfectly reasonable. I'm more worried about noise and heat than the cost though, but I guess that's not too bad either then. What's the latency like? When I've used generative image models the programs unload the model after they're done, so it takes a while to generate the next image. Is the model sitting in VRAM when it's idle?

coder543 3y ago
Fan noise isn’t very much, and you can always limit the max clockspeeds on a GPU (and/or undervolt it) to be quieter and more efficient at a cost of a small amount of performance. The RTX 3090 still seems to be faster than the M3 Max for LLMs that fit on the 3090, so giving up a little performance for near-silent operation wouldn’t be a big loss. Ollama caches the last used model in memory for a few minutes, then unloads it if it hasn’t been used in that time to free up VRAM. I think they’re working on making this period configurable. Latency is very good in my experience, but I haven’t used the local code completion stuff much, just a few quick experiments on personal projects, so my experience with that aspect is limited. If I ever have a job that encourages me to use my own LLM server, I would certainly consider using it more for that.