13 ms·

It's great they got LLMs running on resource constrained devices but are they any good? Or I should ask, with the limited resources they get, what good are they

by emrah 4y ago

It's great they got LLMs running on resource constrained devices but are they any good? Or I should ask, with the limited resources they get, what good are they for?

BulgarianIdiot 4y ago
They’re crude but will be getting better quickly.
- cgearhart 4y ago
  Even with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly running LLMs. But I joked recently that as long as they prove useful then consumers would be willing to swap their iPhone for the old car battery with a phone handle attached.
  - BulgarianIdiot 4y ago
    The first step is for it to be viable for smaller models on desktops. The rest will follow, as hardware catches up. Hybrid analog-hybrid NN hardware is on the horizon, maybe in 3-4 years. This would allow GPT-4 level performance on an iPhone with plausible battery life. The current hardware of course can't pull anything like this yet. But iPhone supports on-device facial recognition, object recognition, dictation and translation, so small steps...
    - boppo1 4y ago
      Who is making said hardware?
      - BulgarianIdiot 4y ago
        There are few efforts under way. https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://news.mit.edu/2022/analog-deep-learning-ai-computing-0728 https://news.mit.edu/2022/analog-deep-learning-ai-computing-...
tudorw 4y ago
I guess if the hardware is cheap for some uses speed is not so important, you can just walk away and let it grind.
kbrkbr 4y ago
From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.
- deleted 4y ago
  [deleted]
- synergy20 4y ago
  are we talking about training or inference for local LLM here? it's hard to do any meaningful training on the edge unless we all carry a heavy gaming pc, even that, the training quality will be subpar?
  - qumpis 4y ago
    Inference, even fine-tuning a few layers would be difficult since one needs to use non-quantized model, I'd imagine
    - ctoth 4y ago
      Checkout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.
      - homarp 4y ago
        DyLoRA https://news.ycombinator.com/item?id=35514228 https://news.ycombinator.com/item?id=35514228 is what you meant
        whimsicalism 4y ago
        Disagree that this is a major breakthrough :) but it is probably a marginal improvement.
turmeric_root 4y ago
I like using them for memeing

by emrah 4y ago

It's great they got LLMs running on resource constrained devices but are they any good? Or I should ask, with the limited resources they get, what good are they for?

BulgarianIdiot 4y ago

They’re crude but will be getting better quickly.

cgearhart 4y ago
Even with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly running LLMs. But I joked recently that as long as they prove useful then consumers would be willing to swap their iPhone for the old car battery with a phone handle attached.
- BulgarianIdiot 4y ago
  The first step is for it to be viable for smaller models on desktops. The rest will follow, as hardware catches up. Hybrid analog-hybrid NN hardware is on the horizon, maybe in 3-4 years. This would allow GPT-4 level performance on an iPhone with plausible battery life. The current hardware of course can't pull anything like this yet. But iPhone supports on-device facial recognition, object recognition, dictation and translation, so small steps...
  - boppo1 4y ago
    Who is making said hardware?
    - BulgarianIdiot 4y ago
      There are few efforts under way. https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://news.mit.edu/2022/analog-deep-learning-ai-computing-0728 https://news.mit.edu/2022/analog-deep-learning-ai-computing-...

tudorw 4y ago

I guess if the hardware is cheap for some uses speed is not so important, you can just walk away and let it grind.

kbrkbr 4y ago

From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.

deleted 4y ago
[deleted]
synergy20 4y ago
are we talking about training or inference for local LLM here? it's hard to do any meaningful training on the edge unless we all carry a heavy gaming pc, even that, the training quality will be subpar?
- qumpis 4y ago
  Inference, even fine-tuning a few layers would be difficult since one needs to use non-quantized model, I'd imagine
  - ctoth 4y ago
    Checkout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.
    - homarp 4y ago
      DyLoRA https://news.ycombinator.com/item?id=35514228 https://news.ycombinator.com/item?id=35514228 is what you meant
      - whimsicalism 4y ago
        Disagree that this is a major breakthrough :) but it is probably a marginal improvement.

turmeric_root 4y ago

I like using them for memeing