4 ms·
I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried
by insanitybit 3y ago
I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and:
a) They're much slower on my 6GB laptop GPU
b) The seem to not be as good, functionally
b) I can't make use of larger models
I haven't done more than just some experimentation but I can see how this would make sense to put on a server.
- baq 3y agoMacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device. I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.
- TillE 3y agoAMD is probably best positioned to make a laptop/desktop SoC with the same kind of unified memory. Though I wouldn't be surprised if Intel's little experiment in budget dedicated GPUs is also a byproduct of similar R&D. Lisa Su was interviewed at Code recently, and the AI discussion focused on competing against NVIDIA in big datacenters, but I hope they're also thinking about client-side stuff like that.
- Saline9515 3y agoAt 600$+/y subscription fees, it makes sense to own a beefy NUC with 32Gb and a speedy SSD to run the model locally.
- jononor 3y agoI would hope developers value their time at least 100USD/hour. So unless the local option is less than 5 hours per year of maintenance, the subscription makes sense.
- Saline9515 3y agoOne's time is free. Many developpers make less than 3k$ /m net, so it still makes sense.