14 ms·

I've been playing around a lot with llama.cpp recently, and it's making me re-think my predictions for the future... Given how big these models are (and the st

I've been playing around a lot with llama.cpp recently, and it's making me re-think my predictions for the future... Given how big these models are (and the steep cost for GPUs to load them), I had been thinking that most people would interact with them via some hosted API (like what openAI is offering) or via some product like Bard or Copilot which offload inference to some big cloud datacenter. But given how well some of these models perform on the CPU when quantized down to 4, 6, or 8 bits, I'm starting to think that there will be quite a few interesting applications for fully local inference on relatively modest hardware

minimaxir 3y ago
The model running well is only one constraint for consumer adoption. The other ones, namely RAM and making the CPU super warm and rapidly draining battery, are another which is much harder to solve.
- falcor84 3y ago
  I'm somewhat bullish on the Edge Computing approach to this. For example, how about a co-op neighborhood LLM rack?
  - alberth 3y ago
    How is “Edge Computing” any different than what the GP was stating about current cloud hosted (eg OpenAI)?
    - falcor84 3y ago
      There are different possible approaches here, but for me, the main benefit would be of having control over the model(s) used. Other than that, low latency would be crucial for effective voice interaction.
  - sho_hn 3y ago
    I currently use a TrueNAS as my home server, and I'm sure whatever I replace it with will add HW for local inference and act as my Home Brain for various automations and assistants.
  - runjake 3y ago
    > how about a co-op neighborhood LLM rack? It'll work about as well as those "wire your neighborhood for Internet as a collective" movements of the 1990s. In fact, you'll need those to deal with latency issues, if my experience with consumer ISP quality is any indication.
    - dharmab 3y ago
      Funny enough, I know two separate groups of people doing that. One in an apartment in SF running a shared Wi-Fi network with VLANs so they don't step on each other's frequencies, and another group sharing WISP infrastructure in semi-rural Utah where they can't convince ISPs to lay fiber. (Although I think the second group now uses that as a backup for Starlink.)
      - runjake 3y ago
        Yes, UBNT nanobeam 5ac AC wireless[1] is pretty dirt cheap. Can get a pair for about $199 for a wireless P2P link that can work over several kilometers. I have deployed a number of these myself, along with more high bandwidth airFiber stuff. 1. https://store.ui.com/us/en/pro/category/all-wireless/products/nanobeam-5ac https://store.ui.com/us/en/pro/category/all-wireless/product...
  - ozr 3y ago
    Why would you do that, and when else has it worked?
  - zztop44 3y ago
    I would much rather send my personal information to Microsoft and the NSA than people in my neighborhood
nly 3y ago
There won't, because there's no money in it
- joegibbs 3y ago
  There is for whichever company wants to make use of it - say for example LLM-powered NPCs in singleplayer video games (which I'm working on) - pretty much economically unviable with the current model if you want to run everything on the cloud. You're selling a game once rather than selling a subscription, so you're locked in to paying for cloud-based LLM services indefinitely, unless you want to shut the game down after X years, which really annoys people. Also, you're incentivised to be stingy with the LLM-based features because the more you use it the more it costs you, but by running it locally you can offload that cost to the consumer.
  - moneywoes 3y ago
    would love to learn more and help out on NPCs llms. What sort of performance hit are you seeing?
    - joegibbs 3y ago
      Sure, here's my writeup about it: https://jgibbs.dev/blogs/local-llm-npcs-in-unreal-engine https://jgibbs.dev/blogs/local-llm-npcs-in-unreal-engine. There's barely any performance hit, only a slight hitch when it starts generating a response.
  - larodi 3y ago
    There’s indeed a lot of GPU time not utilised even by modern games. I can definitely imagine new games utilising llms much more even and with local inference option.
- andy99 3y ago
  At some point soon, LLMs matching current SOTA will run locally in a browser. If applications for LLMs emerge, having one available will be table stakes.
api 3y ago
I think in 3-5 years we will have way more competition in the GPU space and maybe even affordable ASICs on the market. Alternatively we may see CPUs gain AI oriented vector acceleration. Either way I expect it to be cheaper in the near future to run large models locally. That being said your less technical users will probably still use them remotely, and the very largest models will also probably be hosted just because of the capital cost. It doesn’t make sense to spring for a super high end rig to run a massive model locally unless you are using it very very heavily or are a serious homelab enthusiast willing to shell out some bucks. Of course there are wildcards here like breakthroughs in model efficiency, compression, or distributed execution and training. Any of that could change the game. I get the sense that we really don’t know how far we are from optimal efficiency right now. There may be far more efficient architectures or ways of compressing these things.
- girvo 3y ago
  > maybe even affordable ASICs on the market Part of me envisions a Coral-like device, but the kicker/difficulty/cost problem is always going to be fast, big memory. I think that puts a bit of a price floor on some of it. Can't wait to be wrong though!
moneywoes 3y ago
what sort of applications are you thinking? live translations?
colechristensen 3y ago
There seems to be a lot of potential for efficiency gains in models where a much smaller model can achieve the same results. On the other end computing hardware always gets faster and GPU size/performance has been growing exponentially for a good long while. Local inference is quite inevitable.

minimaxir 3y ago

The model running well is only one constraint for consumer adoption. The other ones, namely RAM and making the CPU super warm and rapidly draining battery, are another which is much harder to solve.

falcor84 3y ago
I'm somewhat bullish on the Edge Computing approach to this. For example, how about a co-op neighborhood LLM rack?
- alberth 3y ago
  How is “Edge Computing” any different than what the GP was stating about current cloud hosted (eg OpenAI)?
  - falcor84 3y ago
    There are different possible approaches here, but for me, the main benefit would be of having control over the model(s) used. Other than that, low latency would be crucial for effective voice interaction.
- sho_hn 3y ago
  I currently use a TrueNAS as my home server, and I'm sure whatever I replace it with will add HW for local inference and act as my Home Brain for various automations and assistants.
- runjake 3y ago
  > how about a co-op neighborhood LLM rack? It'll work about as well as those "wire your neighborhood for Internet as a collective" movements of the 1990s. In fact, you'll need those to deal with latency issues, if my experience with consumer ISP quality is any indication.
  - dharmab 3y ago
    Funny enough, I know two separate groups of people doing that. One in an apartment in SF running a shared Wi-Fi network with VLANs so they don't step on each other's frequencies, and another group sharing WISP infrastructure in semi-rural Utah where they can't convince ISPs to lay fiber. (Although I think the second group now uses that as a backup for Starlink.)
    - runjake 3y ago
      Yes, UBNT nanobeam 5ac AC wireless[1] is pretty dirt cheap. Can get a pair for about $199 for a wireless P2P link that can work over several kilometers. I have deployed a number of these myself, along with more high bandwidth airFiber stuff. 1. https://store.ui.com/us/en/pro/category/all-wireless/products/nanobeam-5ac https://store.ui.com/us/en/pro/category/all-wireless/product...
- ozr 3y ago
  Why would you do that, and when else has it worked?
- zztop44 3y ago
  I would much rather send my personal information to Microsoft and the NSA than people in my neighborhood

nly 3y ago

There won't, because there's no money in it

joegibbs 3y ago
There is for whichever company wants to make use of it - say for example LLM-powered NPCs in singleplayer video games (which I'm working on) - pretty much economically unviable with the current model if you want to run everything on the cloud. You're selling a game once rather than selling a subscription, so you're locked in to paying for cloud-based LLM services indefinitely, unless you want to shut the game down after X years, which really annoys people. Also, you're incentivised to be stingy with the LLM-based features because the more you use it the more it costs you, but by running it locally you can offload that cost to the consumer.
- moneywoes 3y ago
  would love to learn more and help out on NPCs llms. What sort of performance hit are you seeing?
  - joegibbs 3y ago
    Sure, here's my writeup about it: https://jgibbs.dev/blogs/local-llm-npcs-in-unreal-engine https://jgibbs.dev/blogs/local-llm-npcs-in-unreal-engine. There's barely any performance hit, only a slight hitch when it starts generating a response.
- larodi 3y ago
  There’s indeed a lot of GPU time not utilised even by modern games. I can definitely imagine new games utilising llms much more even and with local inference option.
andy99 3y ago
At some point soon, LLMs matching current SOTA will run locally in a browser. If applications for LLMs emerge, having one available will be table stakes.

api 3y ago

I think in 3-5 years we will have way more competition in the GPU space and maybe even affordable ASICs on the market. Alternatively we may see CPUs gain AI oriented vector acceleration. Either way I expect it to be cheaper in the near future to run large models locally. That being said your less technical users will probably still use them remotely, and the very largest models will also probably be hosted just because of the capital cost. It doesn’t make sense to spring for a super high end rig to run a massive model locally unless you are using it very very heavily or are a serious homelab enthusiast willing to shell out some bucks. Of course there are wildcards here like breakthroughs in model efficiency, compression, or distributed execution and training. Any of that could change the game. I get the sense that we really don’t know how far we are from optimal efficiency right now. There may be far more efficient architectures or ways of compressing these things.

girvo 3y ago
> maybe even affordable ASICs on the market Part of me envisions a Coral-like device, but the kicker/difficulty/cost problem is always going to be fast, big memory. I think that puts a bit of a price floor on some of it. Can't wait to be wrong though!

moneywoes 3y ago

what sort of applications are you thinking? live translations?

colechristensen 3y ago

There seems to be a lot of potential for efficiency gains in models where a much smaller model can achieve the same results. On the other end computing hardware always gets faster and GPU size/performance has been growing exponentially for a good long while. Local inference is quite inevitable.