3 ms·

That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of mod

by sva_ 4y ago

That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of models that are so large that it doesn't seem feasible to run them locally. But I think there is reason to believe that these models carry a lot of redundancy. Redundancy that could lead to order of magnitude less memory/compute needed. Or perhaps hardware will catch up before.

nwoli 4y ago
The trick here will be using large models as data generators to distill some sub task into a web computable model. (I’ve done it a few times for vision rather than text and it’s amazing how potent it is.)
- ShamelessC 4y ago
  Right! In a lot of cases, just having the synthetic responses plus human filtering for your sub task is enough for less essential tasks. I’m thinking of “procedural” content useful for less sensitive things like games.
- thebruce87m 4y ago
  Can you describe the vision bit? I have a general idea but would like to know the details, e.g. which models you used.
simonw 4y ago
It's possible to run a full GPT-3 style language model on any device with 4GB of RAM now, so running models on consumer devices is getting more and more feasible by the day. https://simonwillison.net/2023/Mar/11/llama/ https://simonwillison.net/2023/Mar/11/llama/
- whimsicalism 4y ago
  Its possible to run a RLHF tuned Llama 7b model. Whether this is "full GPT-3 style" is up for debate.
  - freedomben 4y ago
    I'm mostly a layman with ML stuff, so I might be doing something wrong, but I've not been impressed with Llama even at higher levels. I've run the 35B model in my home lab and it gave some pretty nonsensical responses. The 13B did better though, so could very well be user error.
    - refulgentis 4y ago
      There’s this gold rush going on, you’re right, any B without RLHF is meh. The things getting published as “on device LLM” focus on bitcrushing the lowest B model with minimal RLHF and then pronouncing we have on device LLMs. We’ll definitely get there but signal >>> noise currently. First person to admit this and write their blog post with A / B tests vs. a Markov chain deserves the gold.
    - simonw 4y ago
      Have you tried Alpaca yet? It's a massive improvement on base LLaMA.
Ameo 4y ago
> Or perhaps hardware will catch up before. I feel like that's been the pretty consistent lesson in computing over the past decades. New technologies start out as expensive, exotic, and specialized and become cheap and commonplace over time. The more business value the technology provides, the faster it will happen as well I think. The models will certainly get better (faster to train, less data needed, smaller param counts, etc.) too, though, just like compilers and software have evolved hugely alongside hardware.
yieldcrv 4y ago
they'll meet in the middle. that's what's already happening, and there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need.
- dragonwriter 4y ago
  > there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need. There already are, e.g., Google Edge TPU, Apple Neural Engine, etc.
  - Tostino 4y ago
    They don't help with the memory requirements of these LLMs though.
  - yieldcrv 4y ago
    are any of the LLM or image AI like Stable Diffusion fine tuning methods leveraging Apple Neural Engine? the best I've seen is a renderer leveraging "metal"
illiarian 4y ago
> I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. Running ML on the device has been one of Apple's value propositions for a long time. They are currently silent on everything that's unfolding, but I expect them to at least mention something and WWDC (and trying to run that something on the device)
- refulgentis 4y ago
  They’re there, released Core ML Stable Diffusion a couple months ago.
  - bredren 4y ago
    I am not a Swift dev but it seemed like the speed of this release was very fast by Apple standards. Can anyone in the know confirm that?
- bredren 4y ago
  If I understand correctly, there was an all-company invited annual AI day which was silent on recent developments. But then ~two weeks later there was what seemed like an on-background / press leak about the XDG group that specifically mentioned AI as a current discipline. (Gurman / Bloomberg) It seems to me that the release of Core ML stable diffusion (mentions itt) is something if a comment in of itself. At least in the read between the lines / hiding in plain sight style of Apple. The company is unveiling a new and presumably next major computing platform at a quality level only they could possibly deliver. So the relative quiet / lack of comment may be in deference to the gravity of that work. That said, these changes are too big to ignore—-we should at least hear language that acknowledges the major developments in AI of late at WWDC and some idea for how Apple is thinking about them.

by sva_ 4y ago

nwoli 4y ago

The trick here will be using large models as data generators to distill some sub task into a web computable model. (I’ve done it a few times for vision rather than text and it’s amazing how potent it is.)

ShamelessC 4y ago
Right! In a lot of cases, just having the synthetic responses plus human filtering for your sub task is enough for less essential tasks. I’m thinking of “procedural” content useful for less sensitive things like games.
thebruce87m 4y ago
Can you describe the vision bit? I have a general idea but would like to know the details, e.g. which models you used.

simonw 4y ago

It's possible to run a full GPT-3 style language model on any device with 4GB of RAM now, so running models on consumer devices is getting more and more feasible by the day. https://simonwillison.net/2023/Mar/11/llama/ https://simonwillison.net/2023/Mar/11/llama/

whimsicalism 4y ago
Its possible to run a RLHF tuned Llama 7b model. Whether this is "full GPT-3 style" is up for debate.
- freedomben 4y ago
  I'm mostly a layman with ML stuff, so I might be doing something wrong, but I've not been impressed with Llama even at higher levels. I've run the 35B model in my home lab and it gave some pretty nonsensical responses. The 13B did better though, so could very well be user error.
  - refulgentis 4y ago
    There’s this gold rush going on, you’re right, any B without RLHF is meh. The things getting published as “on device LLM” focus on bitcrushing the lowest B model with minimal RLHF and then pronouncing we have on device LLMs. We’ll definitely get there but signal >>> noise currently. First person to admit this and write their blog post with A / B tests vs. a Markov chain deserves the gold.
  - simonw 4y ago
    Have you tried Alpaca yet? It's a massive improvement on base LLaMA.

Ameo 4y ago

> Or perhaps hardware will catch up before. I feel like that's been the pretty consistent lesson in computing over the past decades. New technologies start out as expensive, exotic, and specialized and become cheap and commonplace over time. The more business value the technology provides, the faster it will happen as well I think. The models will certainly get better (faster to train, less data needed, smaller param counts, etc.) too, though, just like compilers and software have evolved hugely alongside hardware.

yieldcrv 4y ago

they'll meet in the middle. that's what's already happening, and there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need.

dragonwriter 4y ago
> there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need. There already are, e.g., Google Edge TPU, Apple Neural Engine, etc.
- Tostino 4y ago
  They don't help with the memory requirements of these LLMs though.
- yieldcrv 4y ago
  are any of the LLM or image AI like Stable Diffusion fine tuning methods leveraging Apple Neural Engine? the best I've seen is a renderer leveraging "metal"

illiarian 4y ago

> I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. Running ML on the device has been one of Apple's value propositions for a long time. They are currently silent on everything that's unfolding, but I expect them to at least mention something and WWDC (and trying to run that something on the device)

refulgentis 4y ago
They’re there, released Core ML Stable Diffusion a couple months ago.
- bredren 4y ago
  I am not a Swift dev but it seemed like the speed of this release was very fast by Apple standards. Can anyone in the know confirm that?
bredren 4y ago
If I understand correctly, there was an all-company invited annual AI day which was silent on recent developments. But then ~two weeks later there was what seemed like an on-background / press leak about the XDG group that specifically mentioned AI as a current discipline. (Gurman / Bloomberg) It seems to me that the release of Core ML stable diffusion (mentions itt) is something if a comment in of itself. At least in the read between the lines / hiding in plain sight style of Apple. The company is unveiling a new and presumably next major computing platform at a quality level only they could possibly deliver. So the relative quiet / lack of comment may be in deference to the gravity of that work. That said, these changes are too big to ignore—-we should at least hear language that acknowledges the major developments in AI of late at WWDC and some idea for how Apple is thinking about them.