6 ms·

In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles. Manufactures such as Apple will be driv

In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles. Manufactures such as Apple will be driven by consumer desire for local LLMs, that are privet and fine tuned with personal data, to move in that direction. Apple are particularly well placed to do this with the unified memory architecture of the M series processors. Local LLMs are the future, not in the cloud, I really wouldn't be supprised if Apple drop something along these lines on Monday. I will be looking out for a "oh, just one more thing" after the headset demos.

sliken 3y ago
Dunno, the vast majority of phones, tablets, laptops, and desktops have had 64 or 128 bit wide memory interfaces for decades. Apple has a high profit market and a vertical market, unlike the competition. Apple skipped GPUs and did it their own, and the GPU is the primary justification for the 100, 200, 400, or 800GB/sec available with today's apple silicon. Don't think Intel, AMD, or phone/laptop/desktop ARMs (from anyone but apple) have any near term plans for great memory bandwidth. Sad. Obviously AMD can do it, they ship a much nicer memory interface on the PS5 and XboxX, but don't ship anything over 128 bit until you move up to $$$$ workstations with the threadripper (256 bit), threadripper pro (512 bit) or Epyc/Genoa (768 bit) wide memory. Even the Epyc/Genoa socket @ 460GB/sec doesn't match the M2 ultra (800GB/sec), despite the bare CPU being more expensive than most of the Mac studios sold.
gtirloni 3y ago
> In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles I'm surprised they have been holding back on improving their GPUs if this is true. Was there any technical achievement in the recent years to enable this leap?
smoldesu 3y ago
Apple is kinda hamstrung with the M1. Their base model is 8gb, only ~5 of which is addressable by the system if you have nothing else open. They need to bump up the minimum spec if they want local LLM support to feel seamless, especially on iPhone where the memory constraints are even more dire. Honestly, Apple doesn't strike me as any better or worse off than their competitors. Maybe AMD, given how late to the party they are. Intel has AVX fallback acceleration through Haswell though, and libraries like OpenVINO offer extra acceleration paths for modern Intel CPUs. ARM has ARMnn for any multicore ARMv8 system, and of course Nvidia has CUDA support for multiple arches and OS. Projects like the ONNX Runtime are unifying all of these inferencing interfaces, giving developers multiple deployment options without resorting to walled garden implementations. WebGPU may obsolete them altogether. Local LLMs are awesome, but Apple has some work to do if they want to lead the pack.
- KeplerBoy 3y ago
  AMD also has AVX support on all recent CPUs.
  - sitkack 3y ago
    Unless you have a CPU with vCache, AMD doesn't have 800GB/s of memory bandwidth.
- wmf 3y ago
  The Mac Studio can support 128 GB of VRAM and the MacBook Pro supports 96 GB. (This may change tomorrow.) Technically Intel and AMD also have unified memory but I wonder if it actually works.
rcarmo 3y ago
Their hardware development cycles are anything between 2-7 years, and even if the M3 was literally twice the performance of the M2, they would be unlikely to boost baseline RAM and storage on _any_ device to do that kind of thing this year. Over time, sure. Maybe even on the top-tier Macs as a demo. But I wouldn’t expect anything better than LLaMa. I would, however, expect them to come to their senses regarding how inadequate Siri is.
- senttoschool 3y ago
  I fully expect the Apple Silicon team to throw out the roadmap 2 years into the future and start over because of AI breakthroughs this year. Or concurrently, build a new team to develop the SoC of the future for AI. I fully expect the Neural Engine inside Apple SoCs to take up 80% of the transistors in the future, up from ~10% today.
emilsedgh 3y ago
> driven by consumer desire for local LLMs Citation needed. I don't think general public has any desire/care for that. Like they don't care about a locally run Siri.
- bobbylarrybobby 3y ago
  Individuals may not, but businesses absolutely care that their data isn't accessible to third parties
  - rcme 3y ago
    Really? Most businesses don’t seem to have any trouble uploading data to AWS.
    - viraptor 3y ago
      There's a massive difference between companies providing you services where they interact with your data only and AWS which provides you infrastructure and doesn't care much what you do with it. There's not a single case I'm aware of where someone complained about Amazon accessing their private data in AWS and using it in any way.
      - rcme 3y ago
        Walmart refuses to use AWS, for what it is worth. And no, I don't think there is a functional difference between what AWS does and what is described above.
        viraptor 3y ago
        That makes sense for Walmart. They don't want to sponsor the competition and the risk of leaking secrets may be unacceptable, even if you trust AWS operations to behave properly. I think Walmart is a complete outlier in this topic and they wouldn't use AWS even if homomorphic encryption was realistically usable for all their software. On the other hand, Netflix did (does?) use AWS.
itake 3y ago
> by consumer desire for local LLMs Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month, but the other big limiter for local llms is storage and network costs. > Local LLMs are the future I think the issue with local-llms will be how can businesses talk to them? How do I use my local LLM with HN, Twitter, Shopify. Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order?
- slowmovintarget 3y ago
  "My god! It's full of API Keys."
- r00fus 3y ago
  > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? Clearly this isn’t an XOR proposition. Use web query to get relevant data (ie to generate dynamic prompt) then use local (ie private data) to calculate. Repeat ad nauseam. Example: please get me the same vegetables I ordered last week from Whole Foods.
- solarkraft 3y ago
  > Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month The incentive may actually be the other way around. If it has to be run in the cloud, it has to be a service. And selling a service is more attractive to companies (even Apple nowadays) than selling a piece of hardware.
- selalipop 3y ago
  Consumers care about where the LLM is run because of the UX: People hate how current assistants randomly break because you have a spotty signal, or take forever to respond because it's making a round trip across the internet to actually reply, etc. They don't need to know how you solved that, but local LLMs are the way to solve it. > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? LLMs are like an implementation detail: Shopify wouldn't have a single "Shopify" model, but they might have models for specific things they need to do to generate value for themselves. Likewise you might have some local model that acts specifically like a personal assistant and is able to initiate a refund for you. OpenAI's plugin spec already works for advertising capabilities to LLMs you don't own: https://server.shop.app/.well-known/ai-plugin.json https://server.shop.app/.well-known/ai-plugin.json
MuffinFlavored 3y ago
I don't see a ton of demand for LLMs (especially offline/local ones) unless they fact check themselves. I don't see how there's a need for nonsense generators if you can't trust the output. I don't think Siri would've gotten popular if 50% of the time you asked her a question, she made up a fake/wrong response.
- viraptor 3y ago
  > I don't see how there's a need for nonsense generators if you can't trust the output. There's lots of use cases where you know the answer and can verify it yourself. Knowledge lookup is not the only use. For example "rephrase this text", or "give me ideas for X" just save you time and if the answer is wrong you can try again.
- kccqzy 3y ago
  Fact checking is important but a lot of use cases don't need fact checking; imagine LLM as an assistant to write fiction or poetry, or imagine even some kind of a rubber-duck debugging partner or brainstorming partner where the user supplies all pertinent facts. I don't see LLMs replacing traditional Search, but they still have plenty of other use cases.
  - rini17 3y ago
    Even writing fiction need some grip on reality. Supplying all pertinent facts is usually too tedious for users.
    - deleted 3y ago
      [deleted]
  - MuffinFlavored 3y ago
    > or brainstorming partner where the user supplies all pertinent facts. and then it recommends a 3rd party library to solve your problem that doesn't exist
NicoJuicy 3y ago
> I really wouldn't be supprised if Apple drop something along these lines on Monday Apple is really slow on catching up on things, while they are good in introducing "new" things.
fomine3 3y ago
"Unified memory" itself is not special. Many products with AMD/Intel/Qualcomm chip implements LPDDR5 and use it for both CPU and GPU. What's truly different is that only Apple can sell higher performance iGPU (with higher bandwidth memory bits). I wish it changes by this AI boom
- brandall10 3y ago
  I'm sure the comment was in regards to the amount available for training/inference. The fact you can buy an M2 Max w/ 96 GB available to the GPU is remarkable, when the far more powerful 4090 desktop card for instance only has 24 GB.
  - timschmidt 3y ago
    Ryzen 7940 supports up to 256Gb in a laptop form factor, with AMD's AI cores adopted from Xilinx (which seem to have some quite interesting features) as well as M2 beating CPU and GPU performance. I'm sure the race will continue, with wins for both sides and all of us. Unified memory is certainly a much less expensive way to enable large memories for GPUs and accelerators. It seems that all parties are introducing AI acceleration of some kind into CPU silicon within the next generation, which is exciting for the prospect of needing nothing more than a recent CPU to run inference on a large model.
    - senttoschool 3y ago
      >Ryzen 7940 supports up to 256Gb in a laptop form factor This is CPU-only RAM. It's way too low bandwidth to be used in high performance AI inference. The point with unified memory is that it's high bandwidth and available to the CPU, NPU, and GPU at the same time. It should beat the base M2. It's the highest-end AMD laptop chip compared to Apple's lowest end. However, I have doubts that the Ryzen 7940 has a faster GPU than the base M2. And of course, the M2 is significantly better in perf/watt.
      - timschmidt 3y ago

sliken 3y ago

Dunno, the vast majority of phones, tablets, laptops, and desktops have had 64 or 128 bit wide memory interfaces for decades. Apple has a high profit market and a vertical market, unlike the competition. Apple skipped GPUs and did it their own, and the GPU is the primary justification for the 100, 200, 400, or 800GB/sec available with today's apple silicon. Don't think Intel, AMD, or phone/laptop/desktop ARMs (from anyone but apple) have any near term plans for great memory bandwidth. Sad. Obviously AMD can do it, they ship a much nicer memory interface on the PS5 and XboxX, but don't ship anything over 128 bit until you move up to $$$$ workstations with the threadripper (256 bit), threadripper pro (512 bit) or Epyc/Genoa (768 bit) wide memory. Even the Epyc/Genoa socket @ 460GB/sec doesn't match the M2 ultra (800GB/sec), despite the bare CPU being more expensive than most of the Mac studios sold.

gtirloni 3y ago

> In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles I'm surprised they have been holding back on improving their GPUs if this is true. Was there any technical achievement in the recent years to enable this leap?

smoldesu 3y ago

Apple is kinda hamstrung with the M1. Their base model is 8gb, only ~5 of which is addressable by the system if you have nothing else open. They need to bump up the minimum spec if they want local LLM support to feel seamless, especially on iPhone where the memory constraints are even more dire. Honestly, Apple doesn't strike me as any better or worse off than their competitors. Maybe AMD, given how late to the party they are. Intel has AVX fallback acceleration through Haswell though, and libraries like OpenVINO offer extra acceleration paths for modern Intel CPUs. ARM has ARMnn for any multicore ARMv8 system, and of course Nvidia has CUDA support for multiple arches and OS. Projects like the ONNX Runtime are unifying all of these inferencing interfaces, giving developers multiple deployment options without resorting to walled garden implementations. WebGPU may obsolete them altogether. Local LLMs are awesome, but Apple has some work to do if they want to lead the pack.

KeplerBoy 3y ago
AMD also has AVX support on all recent CPUs.
- sitkack 3y ago
  Unless you have a CPU with vCache, AMD doesn't have 800GB/s of memory bandwidth.
wmf 3y ago
The Mac Studio can support 128 GB of VRAM and the MacBook Pro supports 96 GB. (This may change tomorrow.) Technically Intel and AMD also have unified memory but I wonder if it actually works.

rcarmo 3y ago

Their hardware development cycles are anything between 2-7 years, and even if the M3 was literally twice the performance of the M2, they would be unlikely to boost baseline RAM and storage on _any_ device to do that kind of thing this year. Over time, sure. Maybe even on the top-tier Macs as a demo. But I wouldn’t expect anything better than LLaMa. I would, however, expect them to come to their senses regarding how inadequate Siri is.

senttoschool 3y ago
I fully expect the Apple Silicon team to throw out the roadmap 2 years into the future and start over because of AI breakthroughs this year. Or concurrently, build a new team to develop the SoC of the future for AI. I fully expect the Neural Engine inside Apple SoCs to take up 80% of the transistors in the future, up from ~10% today.

emilsedgh 3y ago

> driven by consumer desire for local LLMs Citation needed. I don't think general public has any desire/care for that. Like they don't care about a locally run Siri.

bobbylarrybobby 3y ago
Individuals may not, but businesses absolutely care that their data isn't accessible to third parties
- rcme 3y ago
  Really? Most businesses don’t seem to have any trouble uploading data to AWS.
  - viraptor 3y ago
    There's a massive difference between companies providing you services where they interact with your data only and AWS which provides you infrastructure and doesn't care much what you do with it. There's not a single case I'm aware of where someone complained about Amazon accessing their private data in AWS and using it in any way.
    - rcme 3y ago
      Walmart refuses to use AWS, for what it is worth. And no, I don't think there is a functional difference between what AWS does and what is described above.
      - viraptor 3y ago
        That makes sense for Walmart. They don't want to sponsor the competition and the risk of leaking secrets may be unacceptable, even if you trust AWS operations to behave properly. I think Walmart is a complete outlier in this topic and they wouldn't use AWS even if homomorphic encryption was realistically usable for all their software. On the other hand, Netflix did (does?) use AWS.

itake 3y ago

> by consumer desire for local LLMs Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month, but the other big limiter for local llms is storage and network costs. > Local LLMs are the future I think the issue with local-llms will be how can businesses talk to them? How do I use my local LLM with HN, Twitter, Shopify. Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order?

slowmovintarget 3y ago
"My god! It's full of API Keys."
r00fus 3y ago
> Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? Clearly this isn’t an XOR proposition. Use web query to get relevant data (ie to generate dynamic prompt) then use local (ie private data) to calculate. Repeat ad nauseam. Example: please get me the same vegetables I ordered last week from Whole Foods.
solarkraft 3y ago
> Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month The incentive may actually be the other way around. If it has to be run in the cloud, it has to be a service. And selling a service is more attractive to companies (even Apple nowadays) than selling a piece of hardware.
selalipop 3y ago
Consumers care about where the LLM is run because of the UX: People hate how current assistants randomly break because you have a spotty signal, or take forever to respond because it's making a round trip across the internet to actually reply, etc. They don't need to know how you solved that, but local LLMs are the way to solve it. > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? LLMs are like an implementation detail: Shopify wouldn't have a single "Shopify" model, but they might have models for specific things they need to do to generate value for themselves. Likewise you might have some local model that acts specifically like a personal assistant and is able to initiate a refund for you. OpenAI's plugin spec already works for advertising capabilities to LLMs you don't own: https://server.shop.app/.well-known/ai-plugin.json https://server.shop.app/.well-known/ai-plugin.json

MuffinFlavored 3y ago

I don't see a ton of demand for LLMs (especially offline/local ones) unless they fact check themselves. I don't see how there's a need for nonsense generators if you can't trust the output. I don't think Siri would've gotten popular if 50% of the time you asked her a question, she made up a fake/wrong response.

viraptor 3y ago
> I don't see how there's a need for nonsense generators if you can't trust the output. There's lots of use cases where you know the answer and can verify it yourself. Knowledge lookup is not the only use. For example "rephrase this text", or "give me ideas for X" just save you time and if the answer is wrong you can try again.
kccqzy 3y ago
Fact checking is important but a lot of use cases don't need fact checking; imagine LLM as an assistant to write fiction or poetry, or imagine even some kind of a rubber-duck debugging partner or brainstorming partner where the user supplies all pertinent facts. I don't see LLMs replacing traditional Search, but they still have plenty of other use cases.
- rini17 3y ago
  Even writing fiction need some grip on reality. Supplying all pertinent facts is usually too tedious for users.
  - deleted 3y ago
    [deleted]
- MuffinFlavored 3y ago
  > or brainstorming partner where the user supplies all pertinent facts. and then it recommends a 3rd party library to solve your problem that doesn't exist

NicoJuicy 3y ago

> I really wouldn't be supprised if Apple drop something along these lines on Monday Apple is really slow on catching up on things, while they are good in introducing "new" things.

fomine3 3y ago

"Unified memory" itself is not special. Many products with AMD/Intel/Qualcomm chip implements LPDDR5 and use it for both CPU and GPU. What's truly different is that only Apple can sell higher performance iGPU (with higher bandwidth memory bits). I wish it changes by this AI boom

brandall10 3y ago
I'm sure the comment was in regards to the amount available for training/inference. The fact you can buy an M2 Max w/ 96 GB available to the GPU is remarkable, when the far more powerful 4090 desktop card for instance only has 24 GB.
- timschmidt 3y ago
  Ryzen 7940 supports up to 256Gb in a laptop form factor, with AMD's AI cores adopted from Xilinx (which seem to have some quite interesting features) as well as M2 beating CPU and GPU performance. I'm sure the race will continue, with wins for both sides and all of us. Unified memory is certainly a much less expensive way to enable large memories for GPUs and accelerators. It seems that all parties are introducing AI acceleration of some kind into CPU silicon within the next generation, which is exciting for the prospect of needing nothing more than a recent CPU to run inference on a large model.
  - senttoschool 3y ago
    >Ryzen 7940 supports up to 256Gb in a laptop form factor This is CPU-only RAM. It's way too low bandwidth to be used in high performance AI inference. The point with unified memory is that it's high bandwidth and available to the CPU, NPU, and GPU at the same time. It should beat the base M2. It's the highest-end AMD laptop chip compared to Apple's lowest end. However, I have doubts that the Ryzen 7940 has a faster GPU than the base M2. And of course, the M2 is significantly better in perf/watt.
    - timschmidt 3y ago