Fan noise isn’t very much, and you can always limit the max clockspeeds on a GPU (and/or undervolt it) to be quieter and more efficient at a cost of a small amo

Fan noise isn’t very much, and you can always limit the max clockspeeds on a GPU (and/or undervolt it) to be quieter and more efficient at a cost of a small amount of performance. The RTX 3090 still seems to be faster than the M3 Max for LLMs that fit on the 3090, so giving up a little performance for near-silent operation wouldn’t be a big loss. Ollama caches the last used model in memory for a few minutes, then unloads it if it hasn’t been used in that time to free up VRAM. I think they’re working on making this period configurable. Latency is very good in my experience, but I haven’t used the local code completion stuff much, just a few quick experiments on personal projects, so my experience with that aspect is limited. If I ever have a job that encourages me to use my own LLM server, I would certainly consider using it more for that.