4 ms·

I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beef

I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beefy laptops. If ever there were a case to run these models client side, a software developer’s laptop is probably the ideal place. What are the specs that are needed to run inference on these models?

civilized 3y ago
There's a huge range of CPU and memory requirements to run these models. The lower end of the range can be run locally but performance is noticeably worse. I'm guessing this will change as more people are looking at efficiency now. Specialized hardware like Apple's Neural Engine on the MacBook Pro may also help.
KaoruAoiShiho 3y ago
A lot higher than the beefy laptop... Even a 4090 can't run it well.
deleted 3y ago
[deleted]
insanitybit 3y ago
I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and: a) They're much slower on my 6GB laptop GPU b) The seem to not be as good, functionally b) I can't make use of larger models I haven't done more than just some experimentation but I can see how this would make sense to put on a server.
- baq 3y ago
  MacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device. I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.
  - TillE 3y ago
    AMD is probably best positioned to make a laptop/desktop SoC with the same kind of unified memory. Though I wouldn't be surprised if Intel's little experiment in budget dedicated GPUs is also a byproduct of similar R&D. Lisa Su was interviewed at Code recently, and the AI discussion focused on competing against NVIDIA in big datacenters, but I hope they're also thinking about client-side stuff like that.
- Saline9515 3y ago
  At 600$+/y subscription fees, it makes sense to own a beefy NUC with 32Gb and a speedy SSD to run the model locally.
  - jononor 3y ago
    I would hope developers value their time at least 100USD/hour. So unless the local option is less than 5 hours per year of maintenance, the subscription makes sense.
    - Saline9515 3y ago
      One's time is free. Many developpers make less than 3k$ /m net, so it still makes sense.
pjc50 3y ago
> I feel like the centralisation of server resources for something like copilot doesn’t really make sense. For whom? For those selling them it absolutely does, because that's how they get to charge a huge ongoing markup on hardware and electricity costs. That's why everything is a service now.
lathiat 3y ago
The secondary problem to the high resource requirements, is that I think once you ship the model people can just basically pirate it right? I’d be surprised to see the models go the ground breaking commercial AIs shipping but then I’m surprised at how much we’ve gotten open source anyway.

civilized 3y ago

There's a huge range of CPU and memory requirements to run these models. The lower end of the range can be run locally but performance is noticeably worse. I'm guessing this will change as more people are looking at efficiency now. Specialized hardware like Apple's Neural Engine on the MacBook Pro may also help.

KaoruAoiShiho 3y ago

A lot higher than the beefy laptop... Even a 4090 can't run it well.

deleted 3y ago

[deleted]

insanitybit 3y ago

I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and: a) They're much slower on my 6GB laptop GPU b) The seem to not be as good, functionally b) I can't make use of larger models I haven't done more than just some experimentation but I can see how this would make sense to put on a server.

baq 3y ago
MacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device. I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.
- TillE 3y ago
  AMD is probably best positioned to make a laptop/desktop SoC with the same kind of unified memory. Though I wouldn't be surprised if Intel's little experiment in budget dedicated GPUs is also a byproduct of similar R&D. Lisa Su was interviewed at Code recently, and the AI discussion focused on competing against NVIDIA in big datacenters, but I hope they're also thinking about client-side stuff like that.
Saline9515 3y ago
At 600$+/y subscription fees, it makes sense to own a beefy NUC with 32Gb and a speedy SSD to run the model locally.
- jononor 3y ago
  I would hope developers value their time at least 100USD/hour. So unless the local option is less than 5 hours per year of maintenance, the subscription makes sense.
  - Saline9515 3y ago
    One's time is free. Many developpers make less than 3k$ /m net, so it still makes sense.

pjc50 3y ago

> I feel like the centralisation of server resources for something like copilot doesn’t really make sense. For whom? For those selling them it absolutely does, because that's how they get to charge a huge ongoing markup on hardware and electricity costs. That's why everything is a service now.

lathiat 3y ago

The secondary problem to the high resource requirements, is that I think once you ship the model people can just basically pirate it right? I’d be surprised to see the models go the ground breaking commercial AIs shipping but then I’m surprised at how much we’ve gotten open source anyway.