9 ms·

If Apple would wake up to what's happening with llama.cpp etc then I don't see such a market in paying for remote access to big models via API, though it's curr

If Apple would wake up to what's happening with llama.cpp etc then I don't see such a market in paying for remote access to big models via API, though it's currently the only game in town. Currently a Macbook has a Neural Engine that is sitting idle 99% of the time and only suitable for running limited models (poorly documented, opaque rules about what ops can be accelerated, a black box compiler [1] and an apparent 3GB model size limit [2]) OTOH you can buy a Macbook with 64GB 'unified' memory and a Neural Engine today If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE) And then imagine it does that while leaving the CPU+GPU mostly free to run apps/games ... the whole experience of using a computer could change radically in that case. I find it hard not to think this is coming within 5 years (although equally, I can imagine this is not on Apple's roadmap at all currently) [1] https://github.com/hollance/neural-engine https://github.com/hollance/neural-engine [2] https://github.com/smpanaro/more-ane-transformers/blob/main/src/experiments/NOTES.md https://github.com/smpanaro/more-ane-transformers/blob/main/...

anentropic 4y ago
Update: https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.html https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.h... The iPhone 14 runs Whisper model faster than an M1 Max, because it has a newer Neural Engine I look forward to the M3 Macbook launch eagerly, while expecting mild disappointment
vineyardmike 4y ago
Apple’s move to make stable diffusion run well on the iPhone makes me think they’re watching this space, just waiting for the right open model for them to commit to.
qumpis 4y ago
I wonder how good the neural engine with the unified memory is compared to say intel cpu with 32gb ram. Could anyone give some insight?
- anentropic 4y ago
  There seems to be a limit to the size of model you can load before CoreML decides it has to run on CPU instead (see the second link in my previous comment) If it could use the full 'unified' memory that would be a big step towards getting these models running on it I'm unsure how the performance compares to a beefy Intel CPU, but there's some numbers here [1] for running a variant of the small distilbert-base model on the Neural Engine... it's ~10x faster than running on the M1 CPU [1] https://github.com/anentropic/experiments-coreml-ane-distilbert https://github.com/anentropic/experiments-coreml-ane-distilb...
mike_hearn 4y ago
If I were Apple I'd be thinking about the following issues with that strategy: 1. That RAM isn't empty, it's being used by apps and the OS. Fill up 64GB of RAM with an LLM and there's nothing left for anything else. 2. 64GB probably isn't enough for competitive LLMs anyway. 3. Inferencing is extremely energy intensive, but the MacBook / Apple Silicon brand is partly about long battery life. 4. Weights are expensive to produce and valuable IP, but hard to protect on the client unless you do a lot of work with encrypted memory. 5. Even if a high end MacBook can do local inferencing, the iPhone won't and it's the iPhone that matters. 6. You might want to fine tune models based on your personal data and history, but training is different to inference and best done in the cloud overnight (probably?). 7. Apple already has all that stuff worked out for Siri, which is a cloud service, not a local service, even though it'd be easier to run locally than an LLM. And lots more issues with doing it all locally, fun though that is to play with for developers. I hope I'm wrong, it'd be cool to have LLMs be fully local, but it's hard to see situations where the local approach beats out the cloud approach. One possibility is simply cost: if your device does it, you pay for the hardware, if a cloud does it, you have to pay for that hardware again via subscription.
- LargoLasskhyfv 4y ago
  1. Quadruple it. 2. see above Should be cheap, or why else are Samsung, Micron and Kioxia whining about losses? Maybe go for something like Optane memory while doing so.
  - mike_hearn 4y ago
    Optane is sadly no longer being manufactured.
    - LargoLasskhyfv 4y ago
      I know. That's why I wrote something like ;-)
- anentropic 4y ago
  But right now what incentive have I to buy a new laptop? I got this 16GB M1 MBA two years ago and it's literally everything I need, always feels fast, silent etc 1. the idea would be that now there is a reason to buy loads more RAM, whereas currently the market for 64GB is pretty niche 2. 64GB is a big laptop today, in a few years time that will be small. And LLaMA 65B int4 quantized should fit comfortably 4. LLMs will be a commodity. There will be a free one 6. LLMs seem to avoid the need for finetuning by virtue of their size - what we see now with the largest models is you just do prompt engineering. Making use of personal data is a case of Langchain + vectorstores (or however the future of that approach pans out)
samstave 4y ago
Can we re-invent SETI with such LLMs/new GPU folding/whatever hardware and re-pipe the seti data through a Big Ass Neural whatever you want to call it and see if we have any new datapoints to look into? What about other older 'questions' we can point an AI lens at?
- khimaros 4y ago
  https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals might be up your alley.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
ThrowawayR2 4y ago
> "If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE)" Very doubtful unless the user wants to carry around another kilogram worth of batteries to power it. The hefty processing required by these models doesn't come for free (energy wise) and Moore's Law is dead as a nail.
- gregw134 4y ago
  My latest mac seems to have about a kilogram of extra battery already compared to the previous model.
- anentropic 4y ago
  Most of the time I have my laptop plugged in and sit at a desk... But anyway, there are two trends: - processors do more with less power - LLMs get larger, but also smaller and more efficient (via quantizing, pruning) Once upon a time it was prohibitively expensive to decode compressed video on the fly, later CPUs (both Intel [1] and Apple [2]) added dedicated decoding hardware. Now watching hours of YouTube or Netflix are part of standard battery life benchmarks [1] https://www.intel.com/content/www/us/en/developer/articles/technical/encode-and-decode-capabilities-for-7th-generation-intel-core-processors-and-newer.html https://www.intel.com/content/www/us/en/developer/articles/t... [2] https://www.servethehome.com/apple-ignites-the-industry-with-the-m1-pro-and-m1-max/apple-m1-pro-media-engine/ https://www.servethehome.com/apple-ignites-the-industry-with...
deleted 4y ago
[deleted]
fudged71 4y ago
Siri was launched with a server-based approach. It wouldn't be surprising if Apple's near-term LLM strategy would to put a small LLM on local chips/MacOS and a large model running in the cloud. The local model would only do basic fast operations while the cloud could provide the heavyweight intensive analysis/generation.
boringuser2 4y ago
You need state of the art consumer tech to run a model comparable to GPT-3 locally at a glacial pace. Or, you can use a superior GPT 3.5 for free.
rldjbpin 4y ago
I can see how the apple silicon memory system can help with LLMs, but a couple points of reality check: - such amounts of memory is locked behind very expensive sku which even most of mac userbase will not use ( <5% in the new purchases to be very conservative ). - not too long ago apple would restrict the amount of ram in their system for their own reasoning (source: https://9to5mac.com/2016/10/28/apple-macbook-pro-16gb-ram-limitation-battery-efficiency/ https://9to5mac.com/2016/10/28/apple-macbook-pro-16gb-ram-li...) - just like mid 2010s gpus with 6-8 gb vram but with little to benefit from it, i don't see the ml accelerators/gpu in current models being capable enough to make the most of the memory available to it.
- anentropic 4y ago
  That's today... think of the future My first computer had 512KB RAM and 20MB was an expensive hard drive. 64GB Macbooks are currently an expensive 'Pro' novelty, they will be the vanilla of tomorrow > i don't see the ml accelerators/gpu in current models being capable enough to make the most of the memory available to it that's exactly my point (and apparently today's Neural Engine can't even take advantage of all the unified memory available) until LLaMA there was no reason to have more than this, they probably imagined it would just run a bit of face-detection and speech-to-text on the side but if they got serious and beefed it up it could be the next wave of computing IMHO

anentropic 4y ago

Update: https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.html https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.h... The iPhone 14 runs Whisper model faster than an M1 Max, because it has a newer Neural Engine I look forward to the M3 Macbook launch eagerly, while expecting mild disappointment

vineyardmike 4y ago

Apple’s move to make stable diffusion run well on the iPhone makes me think they’re watching this space, just waiting for the right open model for them to commit to.

qumpis 4y ago

I wonder how good the neural engine with the unified memory is compared to say intel cpu with 32gb ram. Could anyone give some insight?

anentropic 4y ago
There seems to be a limit to the size of model you can load before CoreML decides it has to run on CPU instead (see the second link in my previous comment) If it could use the full 'unified' memory that would be a big step towards getting these models running on it I'm unsure how the performance compares to a beefy Intel CPU, but there's some numbers here [1] for running a variant of the small distilbert-base model on the Neural Engine... it's ~10x faster than running on the M1 CPU [1] https://github.com/anentropic/experiments-coreml-ane-distilbert https://github.com/anentropic/experiments-coreml-ane-distilb...

mike_hearn 4y ago

If I were Apple I'd be thinking about the following issues with that strategy: 1. That RAM isn't empty, it's being used by apps and the OS. Fill up 64GB of RAM with an LLM and there's nothing left for anything else. 2. 64GB probably isn't enough for competitive LLMs anyway. 3. Inferencing is extremely energy intensive, but the MacBook / Apple Silicon brand is partly about long battery life. 4. Weights are expensive to produce and valuable IP, but hard to protect on the client unless you do a lot of work with encrypted memory. 5. Even if a high end MacBook can do local inferencing, the iPhone won't and it's the iPhone that matters. 6. You might want to fine tune models based on your personal data and history, but training is different to inference and best done in the cloud overnight (probably?). 7. Apple already has all that stuff worked out for Siri, which is a cloud service, not a local service, even though it'd be easier to run locally than an LLM. And lots more issues with doing it all locally, fun though that is to play with for developers. I hope I'm wrong, it'd be cool to have LLMs be fully local, but it's hard to see situations where the local approach beats out the cloud approach. One possibility is simply cost: if your device does it, you pay for the hardware, if a cloud does it, you have to pay for that hardware again via subscription.

LargoLasskhyfv 4y ago
1. Quadruple it. 2. see above Should be cheap, or why else are Samsung, Micron and Kioxia whining about losses? Maybe go for something like Optane memory while doing so.
- mike_hearn 4y ago
  Optane is sadly no longer being manufactured.
  - LargoLasskhyfv 4y ago
    I know. That's why I wrote something like ;-)
anentropic 4y ago
But right now what incentive have I to buy a new laptop? I got this 16GB M1 MBA two years ago and it's literally everything I need, always feels fast, silent etc 1. the idea would be that now there is a reason to buy loads more RAM, whereas currently the market for 64GB is pretty niche 2. 64GB is a big laptop today, in a few years time that will be small. And LLaMA 65B int4 quantized should fit comfortably 4. LLMs will be a commodity. There will be a free one 6. LLMs seem to avoid the need for finetuning by virtue of their size - what we see now with the largest models is you just do prompt engineering. Making use of personal data is a case of Langchain + vectorstores (or however the future of that approach pans out)

samstave 4y ago

Can we re-invent SETI with such LLMs/new GPU folding/whatever hardware and re-pipe the seti data through a Big Ass Neural whatever you want to call it and see if we have any new datapoints to look into? What about other older 'questions' we can point an AI lens at?

khimaros 4y ago
https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals might be up your alley.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

ThrowawayR2 4y ago

> "If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE)" Very doubtful unless the user wants to carry around another kilogram worth of batteries to power it. The hefty processing required by these models doesn't come for free (energy wise) and Moore's Law is dead as a nail.

gregw134 4y ago
My latest mac seems to have about a kilogram of extra battery already compared to the previous model.
anentropic 4y ago
Most of the time I have my laptop plugged in and sit at a desk... But anyway, there are two trends: - processors do more with less power - LLMs get larger, but also smaller and more efficient (via quantizing, pruning) Once upon a time it was prohibitively expensive to decode compressed video on the fly, later CPUs (both Intel [1] and Apple [2]) added dedicated decoding hardware. Now watching hours of YouTube or Netflix are part of standard battery life benchmarks [1] https://www.intel.com/content/www/us/en/developer/articles/technical/encode-and-decode-capabilities-for-7th-generation-intel-core-processors-and-newer.html https://www.intel.com/content/www/us/en/developer/articles/t... [2] https://www.servethehome.com/apple-ignites-the-industry-with-the-m1-pro-and-m1-max/apple-m1-pro-media-engine/ https://www.servethehome.com/apple-ignites-the-industry-with...

deleted 4y ago

[deleted]

fudged71 4y ago

Siri was launched with a server-based approach. It wouldn't be surprising if Apple's near-term LLM strategy would to put a small LLM on local chips/MacOS and a large model running in the cloud. The local model would only do basic fast operations while the cloud could provide the heavyweight intensive analysis/generation.

boringuser2 4y ago

You need state of the art consumer tech to run a model comparable to GPT-3 locally at a glacial pace. Or, you can use a superior GPT 3.5 for free.

rldjbpin 4y ago

I can see how the apple silicon memory system can help with LLMs, but a couple points of reality check: - such amounts of memory is locked behind very expensive sku which even most of mac userbase will not use ( <5% in the new purchases to be very conservative ). - not too long ago apple would restrict the amount of ram in their system for their own reasoning (source: https://9to5mac.com/2016/10/28/apple-macbook-pro-16gb-ram-limitation-battery-efficiency/ https://9to5mac.com/2016/10/28/apple-macbook-pro-16gb-ram-li...) - just like mid 2010s gpus with 6-8 gb vram but with little to benefit from it, i don't see the ml accelerators/gpu in current models being capable enough to make the most of the memory available to it.

anentropic 4y ago
That's today... think of the future My first computer had 512KB RAM and 20MB was an expensive hard drive. 64GB Macbooks are currently an expensive 'Pro' novelty, they will be the vanilla of tomorrow > i don't see the ml accelerators/gpu in current models being capable enough to make the most of the memory available to it that's exactly my point (and apparently today's Neural Engine can't even take advantage of all the unified memory available) until LLaMA there was no reason to have more than this, they probably imagined it would just run a bit of face-detection and speech-to-text on the side but if they got serious and beefed it up it could be the next wave of computing IMHO