3 ms·
> Writing code with AI? I could swap to a local model. Costs nothing. Costs nothing? I'd have to buy a desktop computer with a good GPU to run that model. How
by pmontra 2y ago
> Writing code with AI? I could swap to a local model. Costs nothing.
Costs nothing? I'd have to buy a desktop computer with a good GPU to run that model. How much would that cost?
- nateglims 2y agoYou need to own that or buy someone else's compute to run any LLM. The marginal cost of the open local models is free.
- Arainach 2y agoYou can get a LOT of compute credits, at the public rates the cloud providers charge (not OpenAI's discount) for the price of an NVIDIA 4xxx with 24GB or more of RAM. Enough to do what most people need AI to do for a long time.
- tabtab 2y agoI think the point is that anyone can enter the AI cloud market to compete with the fat cats if they get too pricy. NVIDIA's de-facto standards are the closest thing to a market moat. If AI ends up highly tied to NVIDIA's standards, then NVIDIA could become the Microsoft of AI. It's hard to know how tool chains will evolve to answer that though, it's new territory. (It could be all moot if the bots take over and eat us humans. I taste like chicken.)
- falcor84 2y agoYou can get a second hand MacBook Air 16GB with Apple silicon from a couple of years ago for about $1,000 and get a decent performance from Llama 3.1
- linotype 2y agoRight? Or a second hand desktop for $300 and put a $300 3060 in it and get even better performance. So many options.
- taneq 2y agoTo run a 7b model? Not much. To run the 405b model mentioned, at interactive speeds, at reasonable quantisation? A lot.