9 ms·

Llama.cpp can do 40 tok/s on M2 Max, 0% CPU usage, using all 38 GPU cores

123 points by samwillis 3y ago

samwillis 3y ago
In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles. Manufactures such as Apple will be driven by consumer desire for local LLMs, that are privet and fine tuned with personal data, to move in that direction. Apple are particularly well placed to do this with the unified memory architecture of the M series processors. Local LLMs are the future, not in the cloud, I really wouldn't be supprised if Apple drop something along these lines on Monday. I will be looking out for a "oh, just one more thing" after the headset demos.
- sliken 3y ago
  Dunno, the vast majority of phones, tablets, laptops, and desktops have had 64 or 128 bit wide memory interfaces for decades. Apple has a high profit market and a vertical market, unlike the competition. Apple skipped GPUs and did it their own, and the GPU is the primary justification for the 100, 200, 400, or 800GB/sec available with today's apple silicon. Don't think Intel, AMD, or phone/laptop/desktop ARMs (from anyone but apple) have any near term plans for great memory bandwidth. Sad. Obviously AMD can do it, they ship a much nicer memory interface on the PS5 and XboxX, but don't ship anything over 128 bit until you move up to $$$$ workstations with the threadripper (256 bit), threadripper pro (512 bit) or Epyc/Genoa (768 bit) wide memory. Even the Epyc/Genoa socket @ 460GB/sec doesn't match the M2 ultra (800GB/sec), despite the bare CPU being more expensive than most of the Mac studios sold.
- gtirloni 3y ago
  > In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles I'm surprised they have been holding back on improving their GPUs if this is true. Was there any technical achievement in the recent years to enable this leap?
- smoldesu 3y ago
  Apple is kinda hamstrung with the M1. Their base model is 8gb, only ~5 of which is addressable by the system if you have nothing else open. They need to bump up the minimum spec if they want local LLM support to feel seamless, especially on iPhone where the memory constraints are even more dire. Honestly, Apple doesn't strike me as any better or worse off than their competitors. Maybe AMD, given how late to the party they are. Intel has AVX fallback acceleration through Haswell though, and libraries like OpenVINO offer extra acceleration paths for modern Intel CPUs. ARM has ARMnn for any multicore ARMv8 system, and of course Nvidia has CUDA support for multiple arches and OS. Projects like the ONNX Runtime are unifying all of these inferencing interfaces, giving developers multiple deployment options without resorting to walled garden implementations. WebGPU may obsolete them altogether. Local LLMs are awesome, but Apple has some work to do if they want to lead the pack.
  - KeplerBoy 3y ago
    AMD also has AVX support on all recent CPUs.
    - sitkack 3y ago
      Unless you have a CPU with vCache, AMD doesn't have 800GB/s of memory bandwidth.
  - wmf 3y ago
    The Mac Studio can support 128 GB of VRAM and the MacBook Pro supports 96 GB. (This may change tomorrow.) Technically Intel and AMD also have unified memory but I wonder if it actually works.
- rcarmo 3y ago
  Their hardware development cycles are anything between 2-7 years, and even if the M3 was literally twice the performance of the M2, they would be unlikely to boost baseline RAM and storage on _any_ device to do that kind of thing this year. Over time, sure. Maybe even on the top-tier Macs as a demo. But I wouldn’t expect anything better than LLaMa. I would, however, expect them to come to their senses regarding how inadequate Siri is.
  - senttoschool 3y ago
    I fully expect the Apple Silicon team to throw out the roadmap 2 years into the future and start over because of AI breakthroughs this year. Or concurrently, build a new team to develop the SoC of the future for AI. I fully expect the Neural Engine inside Apple SoCs to take up 80% of the transistors in the future, up from ~10% today.
- emilsedgh 3y ago
  > driven by consumer desire for local LLMs Citation needed. I don't think general public has any desire/care for that. Like they don't care about a locally run Siri.
  - bobbylarrybobby 3y ago
    Individuals may not, but businesses absolutely care that their data isn't accessible to third parties
    - rcme 3y ago
      Really? Most businesses don’t seem to have any trouble uploading data to AWS.
      - viraptor 3y ago
        There's a massive difference between companies providing you services where they interact with your data only and AWS which provides you infrastructure and doesn't care much what you do with it. There's not a single case I'm aware of where someone complained about Amazon accessing their private data in AWS and using it in any way.
        rcme 3y ago
        Walmart refuses to use AWS, for what it is worth. And no, I don't think there is a functional difference between what AWS does and what is described above.
        viraptor 3y ago
        That makes sense for Walmart. They don't want to sponsor the competition and the risk of leaking secrets may be unacceptable, even if you trust AWS operations to behave properly. I think Walmart is a complete outlier in this topic and they wouldn't use AWS even if homomorphic encryption was realistically usable for all their software. On the other hand, Netflix did (does?) use AWS.
- itake 3y ago
  > by consumer desire for local LLMs Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month, but the other big limiter for local llms is storage and network costs. > Local LLMs are the future I think the issue with local-llms will be how can businesses talk to them? How do I use my local LLM with HN, Twitter, Shopify. Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order?
  - slowmovintarget 3y ago
    "My god! It's full of API Keys."
  - r00fus 3y ago
    > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? Clearly this isn’t an XOR proposition. Use web query to get relevant data (ie to generate dynamic prompt) then use local (ie private data) to calculate. Repeat ad nauseam. Example: please get me the same vegetables I ordered last week from Whole Foods.
  - solarkraft 3y ago
    > Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month The incentive may actually be the other way around. If it has to be run in the cloud, it has to be a service. And selling a service is more attractive to companies (even Apple nowadays) than selling a piece of hardware.
  - selalipop 3y ago
    Consumers care about where the LLM is run because of the UX: People hate how current assistants randomly break because you have a spotty signal, or take forever to respond because it's making a round trip across the internet to actually reply, etc. They don't need to know how you solved that, but local LLMs are the way to solve it. > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? LLMs are like an implementation detail: Shopify wouldn't have a single "Shopify" model, but they might have models for specific things they need to do to generate value for themselves. Likewise you might have some local model that acts specifically like a personal assistant and is able to initiate a refund for you. OpenAI's plugin spec already works for advertising capabilities to LLMs you don't own: https://server.shop.app/.well-known/ai-plugin.json https://server.shop.app/.well-known/ai-plugin.json
- MuffinFlavored 3y ago
  I don't see a ton of demand for LLMs (especially offline/local ones) unless they fact check themselves. I don't see how there's a need for nonsense generators if you can't trust the output. I don't think Siri would've gotten popular if 50% of the time you asked her a question, she made up a fake/wrong response.
  - viraptor 3y ago
    > I don't see how there's a need for nonsense generators if you can't trust the output. There's lots of use cases where you know the answer and can verify it yourself. Knowledge lookup is not the only use. For example "rephrase this text", or "give me ideas for X" just save you time and if the answer is wrong you can try again.
  - kccqzy 3y ago
    Fact checking is important but a lot of use cases don't need fact checking; imagine LLM as an assistant to write fiction or poetry, or imagine even some kind of a rubber-duck debugging partner or brainstorming partner where the user supplies all pertinent facts. I don't see LLMs replacing traditional Search, but they still have plenty of other use cases.
    - rini17 3y ago
      Even writing fiction need some grip on reality. Supplying all pertinent facts is usually too tedious for users.
      - deleted 3y ago
        [deleted]
    - MuffinFlavored 3y ago
      > or brainstorming partner where the user supplies all pertinent facts. and then it recommends a 3rd party library to solve your problem that doesn't exist
- NicoJuicy 3y ago
  > I really wouldn't be supprised if Apple drop something along these lines on Monday Apple is really slow on catching up on things, while they are good in introducing "new" things.
- fomine3 3y ago
  "Unified memory" itself is not special. Many products with AMD/Intel/Qualcomm chip implements LPDDR5 and use it for both CPU and GPU. What's truly different is that only Apple can sell higher performance iGPU (with higher bandwidth memory bits). I wish it changes by this AI boom
  - brandall10 3y ago
    I'm sure the comment was in regards to the amount available for training/inference. The fact you can buy an M2 Max w/ 96 GB available to the GPU is remarkable, when the far more powerful 4090 desktop card for instance only has 24 GB.
    - timschmidt 3y ago
      Ryzen 7940 supports up to 256Gb in a laptop form factor, with AMD's AI cores adopted from Xilinx (which seem to have some quite interesting features) as well as M2 beating CPU and GPU performance. I'm sure the race will continue, with wins for both sides and all of us. Unified memory is certainly a much less expensive way to enable large memories for GPUs and accelerators. It seems that all parties are introducing AI acceleration of some kind into CPU silicon within the next generation, which is exciting for the prospect of needing nothing more than a recent CPU to run inference on a large model.
      - senttoschool 3y ago
        >Ryzen 7940 supports up to 256Gb in a laptop form factor This is CPU-only RAM. It's way too low bandwidth to be used in high performance AI inference. The point with unified memory is that it's high bandwidth and available to the CPU, NPU, and GPU at the same time. It should beat the base M2. It's the highest-end AMD laptop chip compared to Apple's lowest end. However, I have doubts that the Ryzen 7940 has a faster GPU than the base M2. And of course, the M2 is significantly better in perf/watt.
        timschmidt 3y ago
electric_mayhem 3y ago
For those of us with Twitter.com blocked in local dns, what size model is it getting 40tok/s on?
- samwillis 3y ago
  From the thread: > Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. > Getting 24 tok/s with the 13B model > And 5 tok/s with 65B
  - geek_at 3y ago
    > And 5 tok/s with 65B Which is still incredible. I recently tested llama.cpp on a dual xeon server with 24 cores and 70gb RAM and on the 65B model it took like 15 seconds for one token. GPU seems the way to go here
    - samwillis 3y ago
      Exactly, 5 tok/s is just fast enough for real-time speech.
- jazzyjackson 3y ago
  7B here's the linked github fwiw https://github.com/ggerganov/llama.cpp/pull/1642 https://github.com/ggerganov/llama.cpp/pull/1642
  - sitkack 3y ago
    The tweet is literally just this.
  - kristianp 3y ago
    That's a better link than the twitter posts.
itg 3y ago
Anyone have comparisons to how many tokens/s a laptop with a mobile Nvidia gpu gets? Considering the cheapest M2 Max laptop in the US costs almost $3k, you can easily get a laptop with a 3090/4090 Nvidia mobile GPU.
- smoldesu 3y ago
  I like to use the OpenCL benchmarks as a rough point of comparison: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks M2 Max lands just around the 3060 mobile performance profile in this instance, it would be curious to see how the tokens/s reflect that.
  - senttoschool 3y ago
    It's a little bit more nuanced than that. First, OpenCL is basically deprecated at this point and only Metal is up to date. Second, M2 Max might have a lot more RAM available than a 4090 because of its unified memory architecture. It can have up to 96GB vs 24GB for the 4090. So we need to look at different RAM configurations.
    - smoldesu 3y ago
      > OpenCL is basically deprecated at this point By Apple. > unified memory architecture Not really a bottleneck when layering over PCI exists. You're only constrained by PCI bandwidth and memory speeds, neither of which are really slow enough to meaningfully impact AI inferencing performance. Honestly, disk speeds are the #1 AI bottleneck I've seen on older systems. > It can have up to 96GB vs 24GB for the 4090 Good, at the M2 Max's price point I could almost afford 4x 3090s anyways. I'd only need one to beat it in inferencing performance though.
- satysin 3y ago
  Yes I would like another laptop to compare this against as by itself it is a bit meaningless to me. Also total power draw would be interesting to know as well. However I suspect the unified memory architecture of the M2 is a big benefit here and this M2 Max system is either a 64 or 96GB model? Even the very highest end Nvidia dGPU powered laptops won't have anywhere near that amount of GPU memory available and I don't know what the performance impact is with using system memory vs GPU memory on such systems?
rvz 3y ago
Gerganov is probably going to find himself hired away by Apple Inc. at this point.
- rektide 3y ago
  Hopefully he can go somewhere where he can keep doing good public work, and not have to disappear into the void.
- behnamoh 3y ago
  Either that, or he will hire Apple at some point :)
akomtu 3y ago
Imo, the future is millions of micro-GPUs on a chip, with simplified instruction set. Such mGPU clusters will be called neuromorphic chips. Btw, it's time to measure performance of LLMs in watts.
- kurtoid 3y ago
  Sounds like what George Hotz is trying to do with tiny corp
  - airgapstopgap 3y ago
    No longer trying, I suppose.
faeriechangling 3y ago
I find Apple's performance with LLM's pretty exciting. At the very least, it promises that we don't be stuck on the vram pricing and capacities offered by AMD and Nvidia to run local LLMs. We should see some big improvements in the near future as lpddr6 is around the corner and Apple has a big focus on GPU.
senttoschool 3y ago
It doesn't say how much RAM the M2 Max has? How much does RAM matter? As we all know, M2 Max can have up to 96GB of unified memory, which is 4x more than an Nvidia 4090.
- deleted 3y ago
  [deleted]

123 points by samwillis 3y ago

In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles. Manufactures such as Apple will be driven by consumer desire for local LLMs, that are privet and fine tuned with personal data, to move in that direction. Apple are particularly well placed to do this with the unified memory architecture of the M series processors. Local LLMs are the future, not in the cloud, I really wouldn't be supprised if Apple drop something along these lines on Monday. I will be looking out for a "oh, just one more thing" after the headset demos.

sliken 3y ago
Dunno, the vast majority of phones, tablets, laptops, and desktops have had 64 or 128 bit wide memory interfaces for decades. Apple has a high profit market and a vertical market, unlike the competition. Apple skipped GPUs and did it their own, and the GPU is the primary justification for the 100, 200, 400, or 800GB/sec available with today's apple silicon. Don't think Intel, AMD, or phone/laptop/desktop ARMs (from anyone but apple) have any near term plans for great memory bandwidth. Sad. Obviously AMD can do it, they ship a much nicer memory interface on the PS5 and XboxX, but don't ship anything over 128 bit until you move up to $$$$ workstations with the threadripper (256 bit), threadripper pro (512 bit) or Epyc/Genoa (768 bit) wide memory. Even the Epyc/Genoa socket @ 460GB/sec doesn't match the M2 ultra (800GB/sec), despite the bare CPU being more expensive than most of the Mac studios sold.
gtirloni 3y ago
> In only a few years we are going to have close to this level of hardware on all laptops/desktops and maybe even mobiles I'm surprised they have been holding back on improving their GPUs if this is true. Was there any technical achievement in the recent years to enable this leap?
smoldesu 3y ago
Apple is kinda hamstrung with the M1. Their base model is 8gb, only ~5 of which is addressable by the system if you have nothing else open. They need to bump up the minimum spec if they want local LLM support to feel seamless, especially on iPhone where the memory constraints are even more dire. Honestly, Apple doesn't strike me as any better or worse off than their competitors. Maybe AMD, given how late to the party they are. Intel has AVX fallback acceleration through Haswell though, and libraries like OpenVINO offer extra acceleration paths for modern Intel CPUs. ARM has ARMnn for any multicore ARMv8 system, and of course Nvidia has CUDA support for multiple arches and OS. Projects like the ONNX Runtime are unifying all of these inferencing interfaces, giving developers multiple deployment options without resorting to walled garden implementations. WebGPU may obsolete them altogether. Local LLMs are awesome, but Apple has some work to do if they want to lead the pack.
- KeplerBoy 3y ago
  AMD also has AVX support on all recent CPUs.
  - sitkack 3y ago
    Unless you have a CPU with vCache, AMD doesn't have 800GB/s of memory bandwidth.
- wmf 3y ago
  The Mac Studio can support 128 GB of VRAM and the MacBook Pro supports 96 GB. (This may change tomorrow.) Technically Intel and AMD also have unified memory but I wonder if it actually works.
rcarmo 3y ago
Their hardware development cycles are anything between 2-7 years, and even if the M3 was literally twice the performance of the M2, they would be unlikely to boost baseline RAM and storage on _any_ device to do that kind of thing this year. Over time, sure. Maybe even on the top-tier Macs as a demo. But I wouldn’t expect anything better than LLaMa. I would, however, expect them to come to their senses regarding how inadequate Siri is.
- senttoschool 3y ago
  I fully expect the Apple Silicon team to throw out the roadmap 2 years into the future and start over because of AI breakthroughs this year. Or concurrently, build a new team to develop the SoC of the future for AI. I fully expect the Neural Engine inside Apple SoCs to take up 80% of the transistors in the future, up from ~10% today.
emilsedgh 3y ago
> driven by consumer desire for local LLMs Citation needed. I don't think general public has any desire/care for that. Like they don't care about a locally run Siri.
- bobbylarrybobby 3y ago
  Individuals may not, but businesses absolutely care that their data isn't accessible to third parties
  - rcme 3y ago
    Really? Most businesses don’t seem to have any trouble uploading data to AWS.
    - viraptor 3y ago
      There's a massive difference between companies providing you services where they interact with your data only and AWS which provides you infrastructure and doesn't care much what you do with it. There's not a single case I'm aware of where someone complained about Amazon accessing their private data in AWS and using it in any way.
      - rcme 3y ago
        Walmart refuses to use AWS, for what it is worth. And no, I don't think there is a functional difference between what AWS does and what is described above.
        viraptor 3y ago
        That makes sense for Walmart. They don't want to sponsor the competition and the risk of leaking secrets may be unacceptable, even if you trust AWS operations to behave properly. I think Walmart is a complete outlier in this topic and they wouldn't use AWS even if homomorphic encryption was realistically usable for all their software. On the other hand, Netflix did (does?) use AWS.
itake 3y ago
> by consumer desire for local LLMs Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month, but the other big limiter for local llms is storage and network costs. > Local LLMs are the future I think the issue with local-llms will be how can businesses talk to them? How do I use my local LLM with HN, Twitter, Shopify. Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order?
- slowmovintarget 3y ago
  "My god! It's full of API Keys."
- r00fus 3y ago
  > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? Clearly this isn’t an XOR proposition. Use web query to get relevant data (ie to generate dynamic prompt) then use local (ie private data) to calculate. Repeat ad nauseam. Example: please get me the same vegetables I ordered last week from Whole Foods.
- solarkraft 3y ago
  > Consumers don't care where the LLMs are run, but if companies can off-board the computing costs onto the consumers, then they will save millions per month The incentive may actually be the other way around. If it has to be run in the cloud, it has to be a service. And selling a service is more attractive to companies (even Apple nowadays) than selling a piece of hardware.
- selalipop 3y ago
  Consumers care about where the LLM is run because of the UX: People hate how current assistants randomly break because you have a spotty signal, or take forever to respond because it's making a round trip across the internet to actually reply, etc. They don't need to know how you solved that, but local LLMs are the way to solve it. > Can my local LLM help me request a refund from a Shopify order or would Shopify have to fine-tune their own model? Then what? I download a 300GB file just so I can refund an order? LLMs are like an implementation detail: Shopify wouldn't have a single "Shopify" model, but they might have models for specific things they need to do to generate value for themselves. Likewise you might have some local model that acts specifically like a personal assistant and is able to initiate a refund for you. OpenAI's plugin spec already works for advertising capabilities to LLMs you don't own: https://server.shop.app/.well-known/ai-plugin.json https://server.shop.app/.well-known/ai-plugin.json
MuffinFlavored 3y ago
I don't see a ton of demand for LLMs (especially offline/local ones) unless they fact check themselves. I don't see how there's a need for nonsense generators if you can't trust the output. I don't think Siri would've gotten popular if 50% of the time you asked her a question, she made up a fake/wrong response.
- viraptor 3y ago
  > I don't see how there's a need for nonsense generators if you can't trust the output. There's lots of use cases where you know the answer and can verify it yourself. Knowledge lookup is not the only use. For example "rephrase this text", or "give me ideas for X" just save you time and if the answer is wrong you can try again.
- kccqzy 3y ago
  Fact checking is important but a lot of use cases don't need fact checking; imagine LLM as an assistant to write fiction or poetry, or imagine even some kind of a rubber-duck debugging partner or brainstorming partner where the user supplies all pertinent facts. I don't see LLMs replacing traditional Search, but they still have plenty of other use cases.
  - rini17 3y ago
    Even writing fiction need some grip on reality. Supplying all pertinent facts is usually too tedious for users.
    - deleted 3y ago
      [deleted]
  - MuffinFlavored 3y ago
    > or brainstorming partner where the user supplies all pertinent facts. and then it recommends a 3rd party library to solve your problem that doesn't exist
NicoJuicy 3y ago
> I really wouldn't be supprised if Apple drop something along these lines on Monday Apple is really slow on catching up on things, while they are good in introducing "new" things.
fomine3 3y ago
"Unified memory" itself is not special. Many products with AMD/Intel/Qualcomm chip implements LPDDR5 and use it for both CPU and GPU. What's truly different is that only Apple can sell higher performance iGPU (with higher bandwidth memory bits). I wish it changes by this AI boom
- brandall10 3y ago
  I'm sure the comment was in regards to the amount available for training/inference. The fact you can buy an M2 Max w/ 96 GB available to the GPU is remarkable, when the far more powerful 4090 desktop card for instance only has 24 GB.
  - timschmidt 3y ago
    Ryzen 7940 supports up to 256Gb in a laptop form factor, with AMD's AI cores adopted from Xilinx (which seem to have some quite interesting features) as well as M2 beating CPU and GPU performance. I'm sure the race will continue, with wins for both sides and all of us. Unified memory is certainly a much less expensive way to enable large memories for GPUs and accelerators. It seems that all parties are introducing AI acceleration of some kind into CPU silicon within the next generation, which is exciting for the prospect of needing nothing more than a recent CPU to run inference on a large model.
    - senttoschool 3y ago
      >Ryzen 7940 supports up to 256Gb in a laptop form factor This is CPU-only RAM. It's way too low bandwidth to be used in high performance AI inference. The point with unified memory is that it's high bandwidth and available to the CPU, NPU, and GPU at the same time. It should beat the base M2. It's the highest-end AMD laptop chip compared to Apple's lowest end. However, I have doubts that the Ryzen 7940 has a faster GPU than the base M2. And of course, the M2 is significantly better in perf/watt.
      - timschmidt 3y ago

electric_mayhem 3y ago

For those of us with Twitter.com blocked in local dns, what size model is it getting 40tok/s on?

samwillis 3y ago
From the thread: > Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores. > Getting 24 tok/s with the 13B model > And 5 tok/s with 65B
- geek_at 3y ago
  > And 5 tok/s with 65B Which is still incredible. I recently tested llama.cpp on a dual xeon server with 24 cores and 70gb RAM and on the 65B model it took like 15 seconds for one token. GPU seems the way to go here
  - samwillis 3y ago
    Exactly, 5 tok/s is just fast enough for real-time speech.
jazzyjackson 3y ago
7B here's the linked github fwiw https://github.com/ggerganov/llama.cpp/pull/1642 https://github.com/ggerganov/llama.cpp/pull/1642
- sitkack 3y ago
  The tweet is literally just this.
- kristianp 3y ago
  That's a better link than the twitter posts.

itg 3y ago

Anyone have comparisons to how many tokens/s a laptop with a mobile Nvidia gpu gets? Considering the cheapest M2 Max laptop in the US costs almost $3k, you can easily get a laptop with a 3090/4090 Nvidia mobile GPU.

smoldesu 3y ago
I like to use the OpenCL benchmarks as a rough point of comparison: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks M2 Max lands just around the 3060 mobile performance profile in this instance, it would be curious to see how the tokens/s reflect that.
- senttoschool 3y ago
  It's a little bit more nuanced than that. First, OpenCL is basically deprecated at this point and only Metal is up to date. Second, M2 Max might have a lot more RAM available than a 4090 because of its unified memory architecture. It can have up to 96GB vs 24GB for the 4090. So we need to look at different RAM configurations.
  - smoldesu 3y ago
    > OpenCL is basically deprecated at this point By Apple. > unified memory architecture Not really a bottleneck when layering over PCI exists. You're only constrained by PCI bandwidth and memory speeds, neither of which are really slow enough to meaningfully impact AI inferencing performance. Honestly, disk speeds are the #1 AI bottleneck I've seen on older systems. > It can have up to 96GB vs 24GB for the 4090 Good, at the M2 Max's price point I could almost afford 4x 3090s anyways. I'd only need one to beat it in inferencing performance though.
satysin 3y ago
Yes I would like another laptop to compare this against as by itself it is a bit meaningless to me. Also total power draw would be interesting to know as well. However I suspect the unified memory architecture of the M2 is a big benefit here and this M2 Max system is either a 64 or 96GB model? Even the very highest end Nvidia dGPU powered laptops won't have anywhere near that amount of GPU memory available and I don't know what the performance impact is with using system memory vs GPU memory on such systems?

rvz 3y ago

Gerganov is probably going to find himself hired away by Apple Inc. at this point.

rektide 3y ago
Hopefully he can go somewhere where he can keep doing good public work, and not have to disappear into the void.
behnamoh 3y ago
Either that, or he will hire Apple at some point :)

akomtu 3y ago

Imo, the future is millions of micro-GPUs on a chip, with simplified instruction set. Such mGPU clusters will be called neuromorphic chips. Btw, it's time to measure performance of LLMs in watts.

kurtoid 3y ago
Sounds like what George Hotz is trying to do with tiny corp
- airgapstopgap 3y ago
  No longer trying, I suppose.

faeriechangling 3y ago

I find Apple's performance with LLM's pretty exciting. At the very least, it promises that we don't be stuck on the vram pricing and capacities offered by AMD and Nvidia to run local LLMs. We should see some big improvements in the near future as lpddr6 is around the corner and Apple has a big focus on GPU.

senttoschool 3y ago

It doesn't say how much RAM the M2 Max has? How much does RAM matter? As we all know, M2 Max can have up to 96GB of unified memory, which is 4x more than an Nvidia 4090.

deleted 3y ago
[deleted]