4 ms·

In five year's time - by 2030, I foresee that lots of inference would be happening on local machines with models being downloaded on demand. Think docker regist

by wg0 3y ago

In five year's time - by 2030, I foresee that lots of inference would be happening on local machines with models being downloaded on demand. Think docker registry of AI models which is pretty much Hugging Face already there. This all would be due to optimisations within model inference code and techniques, hardware and packaging of software like the above. Don't see billion dollar valuations for lots of AI startups out there to materialise into anything.

openquery 3y ago
> I foresee that lots of inference would be happening on local machines with models being downloaded on demand Why? It's much more efficient to have centralized special purpose hardware to run enormous models and then ship the comparatively small result over the internet. By analogy, you don't have a search engine running on your phone right?
- vachina 3y ago
  A more appropriate analogy would be driving your own car vs. taking the bus.
  - bufferoverflow 3y ago
    No, a more appropriate analogy would be driving your own billion-dollar super-yacht vs driving your own car. Will not happen any time soon. Consumer hardware can't even run GPT-4 locally, and won't be able for a looong time. Each GPT-4 instance runs on 8 A100. The cost of such system is ~$81K. Not even in the ballpark of what most consumers can afford.
- Sammi 3y ago
  You currently can't have a search engine running locally on your phone. Google search is possible the single largest c++ program every built. And nevermind the storage needs... But in a few years we might be able to have LLMs running on our phones that work just as well if not better. Of couse as you mention the LLMs running on large servers might still be much more powerfull, but the local ones might be powerfull enough.
  - deleted 3y ago
    [deleted]
- dns_snek 3y ago
  > Why? Privacy, security, latency, offline availability, access to local data and services running on the device, just to name a few.
  - ilc 3y ago
    Big Tech + Countries: Those all sound like great reasons to centralize all access to AIs!
ThinkBeat 3y ago
"640kb will be enough for everyone." (Gates) I think that the models will evolve and grow as more powerful compute/hardware comes out. You may be able to run scaled down n versions of what state of the art now, but by then the giant models will have grown in size and in required compute. The 6 year old models will be retro computingish. Somewhat like how you can play 6 year old games on a new powerful PC but by then the new huge games will no longer play well on your Old mach
- lobocinza 3y ago
  There will be demand and supply for both cases.
bufferoverflow 3y ago
Unfortunately training is insanely expensive. StabilityAI is struggling with staying alive. And Anthropic wants to spend $100 BILLION building a supercomputer just for training. That's why I think these private companies will have the best AIs for many decades.

by wg0 3y ago

openquery 3y ago

> I foresee that lots of inference would be happening on local machines with models being downloaded on demand Why? It's much more efficient to have centralized special purpose hardware to run enormous models and then ship the comparatively small result over the internet. By analogy, you don't have a search engine running on your phone right?

vachina 3y ago
A more appropriate analogy would be driving your own car vs. taking the bus.
- bufferoverflow 3y ago
  No, a more appropriate analogy would be driving your own billion-dollar super-yacht vs driving your own car. Will not happen any time soon. Consumer hardware can't even run GPT-4 locally, and won't be able for a looong time. Each GPT-4 instance runs on 8 A100. The cost of such system is ~$81K. Not even in the ballpark of what most consumers can afford.
Sammi 3y ago
You currently can't have a search engine running locally on your phone. Google search is possible the single largest c++ program every built. And nevermind the storage needs... But in a few years we might be able to have LLMs running on our phones that work just as well if not better. Of couse as you mention the LLMs running on large servers might still be much more powerfull, but the local ones might be powerfull enough.
- deleted 3y ago
  [deleted]
dns_snek 3y ago
> Why? Privacy, security, latency, offline availability, access to local data and services running on the device, just to name a few.
- ilc 3y ago
  Big Tech + Countries: Those all sound like great reasons to centralize all access to AIs!

ThinkBeat 3y ago

"640kb will be enough for everyone." (Gates) I think that the models will evolve and grow as more powerful compute/hardware comes out. You may be able to run scaled down n versions of what state of the art now, but by then the giant models will have grown in size and in required compute. The 6 year old models will be retro computingish. Somewhat like how you can play 6 year old games on a new powerful PC but by then the new huge games will no longer play well on your Old mach

lobocinza 3y ago
There will be demand and supply for both cases.

bufferoverflow 3y ago

Unfortunately training is insanely expensive. StabilityAI is struggling with staying alive. And Anthropic wants to spend $100 BILLION building a supercomputer just for training. That's why I think these private companies will have the best AIs for many decades.