5 ms·

It looks like the local-LLM enthusiasts will want to use Apple computers. This is funny to me for some reason, like if the most cost-effective work boot of 2025

by ftxbro 3y ago

It looks like the local-LLM enthusiasts will want to use Apple computers. This is funny to me for some reason, like if the most cost-effective work boot of 2025 is produced by Gucci.

PragmaticPulp 3y ago
Local LLMs do best on big GPUs, which aren’t an option on Macs. What Apple has done with the M-series silicon is really impressive, but it’s not as fast as big, dedicated GPU silicon. > like if the most cost-effective work boot of 2025 is produced by Gucci. Apple M-silicon laptops are high quality, but they’re not the same as overpriced luxury goods. The price of entry level M1/M2 Macs is extremely reasonable, IMO.
- daniel-thompson 3y ago
  > Local LLMs do best on big GPUs, which aren’t an option on Macs. For processing speed, Ms are fast but I agree not anywhere near a top Nvdia chip. For memory size, the memory on an M chip can be used as graphics memory, so a person could get an M2 today with 128GB of graphics memory for ~ $5k. Not bad considering an Nvidia chip approaching that memory size is several times that much.
  - deleted 3y ago
    [deleted]
- samwillis 3y ago
  > it’s not as fast as big, dedicated GPU silicon It's not yet. Apples combined memory architecture enables them to significantly cut the cost of bundling more memory for various GPU, Neural or other domain specific cores. I believe they will catch up or even overtake Nvidia as the leading AI platform. Desktop AI class GPUs are hell expensive. If Apple can get something 50% as performant, but in an iPhone Pro or MacBook Air, thats going to change so very much.
  - gpderetta 3y ago
    but AI class GPUs are expensive because NVidia can pretty much ask whatever it wants as it has no competition. If Apple were to become competitive in the area, things can change.
  - xipix 3y ago
    It's so funny to see Apple's "combined memory architecture" praised. When Intel did this a decade+ ago with its integrated GPUs, it was roundly slated.
    - comex 3y ago
      Well, the difference is that compared to a baseline where GPUs typically get faster RAM than CPUs, Apple is providing fast RAM to both, whereas Intel provided slow RAM to both. Of course, that’s hardly Intel’s fault given that their integrated GPUs are aimed at the low end and are usually paired with external socketed RAM.
- Maursault 3y ago
  > Local LLMs do best on big GPUs, which aren’t an option on Macs. What Apple has done with the M-series silicon is really impressive, but it’s not as fast as big, dedicated GPU silicon. It'd be pretty silly for Apple to cram in big, dedicated GPU when 99.9% of their customers don't care and don't want to pay for it, especially considering that anyone that does want big, dedicated GPU can outboard as much GPU as they want.[1] And many seem to think that the onboard GPU along with Neural Engine should be adequate for local LLM. [1] https://support.apple.com/en-us/HT208544 https://support.apple.com/en-us/HT208544
  - thoughtsimple 3y ago
    Only on Intel Macs which are going away soon (only 2019 Mac Pro is remaining outside of refurbished.) Apple Silicon doesn't currently have any provision for external graphics cards.
    - Maursault 3y ago
      There are two issues. 1) Apple Silicon won't support PCIe. 2) Apple doesn't want it to. #2 means Apple is taking nVidia and AMD head on in the GPU space. Apple wants to control everything, and allowing these competitors on their platform is giving away too much. Because Apple Silicon scales better than competitors' hw, the desire for third party GPU is probably going to evaporate within a few generations of Apple Silicon. I mean, we'll see, but that is my best guess, because it seems like that was an intentional decision rather than oversight.
      - bhj 3y ago
        AS supports Thunderbolt just fine. Isn't the lack of eGPU support due to lack of ARM drivers for them?
        Maursault 3y ago
        To suggest it is merely lack of drivers is an oversimplification. There is a chip errata that prevents PCIe GPUs from working properly on Apple Silicon: the architectures are not compatible. Apple Silicon GPU drivers are deeply integrated into the system. Due to this integration, only graphics cards that use the same GPU architecture as Apple Silicon could be supported, and there just aren't any, and I don't see how there could be unless Apple developed one and released it.
- dacryn 3y ago
  they share their main memory with the GPU. There is no cheaper way to have 50gb of memory allocated to your GPU compute pipeline (buy a 64gb macbook pro or studio). So yes big, only the fast aspect remains a reason to buy expensive dedicated gpu's
- diffeomorphism 3y ago
  > The price of entry level M1/M2 Macs is extremely reasonable, IMO. Caveats: In the US, with low ram and low storage. Without these caveats you look at about 2000€ or so (m2 macbook air with 16gb of ram and non-halved storage performance in Germany). That can still be a decent deal for what you get, but "extremely reasonable" -- not quite.
nordsieck 3y ago
It is a little funny... but the combination of pretty good integrated graphics and (potentially) large amounts of on package ram mean that macs are actually a pretty inexpensive way to get access to a GPU with lots of graphics memory.
- braymundo 3y ago
  I wish gaming on Mac became more popular due to the same reason.
  - heliophobicdude 3y ago
    What are some broken-down reasons why gaming on Mac didn't take off. I'm sure it's a few different reasons all together
    - braymundo 3y ago
      I don't know, but if I had to guess it would that Apple was never really interested in it or in supporting it. I fondly remember some companies (like Blizzard) who would always support Macs, until they stopped (looking at you, Diablo 4).
    - dagmx 3y ago
      A lot of people will likely blame APIs etc, but that’s just an easy scapegoat for people outside the game dev industry. The reality is likely just market share, hardware and the makeup of that market share. Macs used to actually be a big gaming platform once upon a time. Myst was a Mac exclusive (built with HyperCard) and Bungie used to make games for macs first before getting bought by Microsoft. But windows got the upper edge: 1. Much larger slice of the market 2. More of the market is made up of gaming enthusiasts than Macs which are usually either targeting light use, education or professional. 3. You can make affordable gaming PCs with windows. You can’t really do it with any kind of Mac because there’s no product for that market between the Mac Mini and iMac (which were targeted at more casual users) up to the Mac Pro and iMac Pro (which were targeted at professional users) It’s easier to target the biggest piece of the pie and the one that will buy your products. That’s why the “API” reason never makes sense, because iOS is THE dominant gaming platform despite having the same APIs as a mac. Game engines often need to support multiple graphics APIs anyway. So Mac gaming did take off, but it fumbled and never recovered. Part of that is that Apple themselves fumbled as a company, on the verge of bankruptcy before the return of Jobs. What made them successful post his return is also what killed gaming on their platforms: they targeted a different audience.
Gare 3y ago
Apple was cost-effective most of the time if you wanted high-quality tools.
foldr 3y ago
If you care about the combination of portability, battery life, performance, and display+audio quality, then Apple’s laptops have been price competitive for quite a while now. People who think they are overpriced usually don’t care about at least one of these things.
KaoruAoiShiho 3y ago
I'm not aware of any local-LLM framework that works with Apple GPUs?
- Maursault 3y ago
  Does this qualify?[1] [1] https://www.infoq.com/news/2023/05/mlc-llm-mobile-laptops/ https://www.infoq.com/news/2023/05/mlc-llm-mobile-laptops/
raxxorraxor 3y ago
I think it is least likely that we will run local AI instances on Apple hardware to be honest. But yes, the comparison would be apt. I think people want to solve a problem with dependence on third party services... No, I expect GPUs with insane amount of memory surfacing and that isn't really a field for Apple yet.
zitterbewegung 3y ago
Not many laptops have 64 gb of RAM that can run llama.cpp https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3db526d5d6fd https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...
- ac29 3y ago
  Plenty of non-Apple laptops support 64GB RAM. For example, both the AMD and Intel variants of the Framework Laptop can take 64GB. All of of the large vendors have laptops that support a 64GB configuration too (HP, Dell, Lenovo).

by ftxbro 3y ago

It looks like the local-LLM enthusiasts will want to use Apple computers. This is funny to me for some reason, like if the most cost-effective work boot of 2025 is produced by Gucci.

PragmaticPulp 3y ago

Local LLMs do best on big GPUs, which aren’t an option on Macs. What Apple has done with the M-series silicon is really impressive, but it’s not as fast as big, dedicated GPU silicon. > like if the most cost-effective work boot of 2025 is produced by Gucci. Apple M-silicon laptops are high quality, but they’re not the same as overpriced luxury goods. The price of entry level M1/M2 Macs is extremely reasonable, IMO.

daniel-thompson 3y ago
> Local LLMs do best on big GPUs, which aren’t an option on Macs. For processing speed, Ms are fast but I agree not anywhere near a top Nvdia chip. For memory size, the memory on an M chip can be used as graphics memory, so a person could get an M2 today with 128GB of graphics memory for ~ $5k. Not bad considering an Nvidia chip approaching that memory size is several times that much.
- deleted 3y ago
  [deleted]
samwillis 3y ago
> it’s not as fast as big, dedicated GPU silicon It's not yet. Apples combined memory architecture enables them to significantly cut the cost of bundling more memory for various GPU, Neural or other domain specific cores. I believe they will catch up or even overtake Nvidia as the leading AI platform. Desktop AI class GPUs are hell expensive. If Apple can get something 50% as performant, but in an iPhone Pro or MacBook Air, thats going to change so very much.
- gpderetta 3y ago
  but AI class GPUs are expensive because NVidia can pretty much ask whatever it wants as it has no competition. If Apple were to become competitive in the area, things can change.
- xipix 3y ago
  It's so funny to see Apple's "combined memory architecture" praised. When Intel did this a decade+ ago with its integrated GPUs, it was roundly slated.
  - comex 3y ago
    Well, the difference is that compared to a baseline where GPUs typically get faster RAM than CPUs, Apple is providing fast RAM to both, whereas Intel provided slow RAM to both. Of course, that’s hardly Intel’s fault given that their integrated GPUs are aimed at the low end and are usually paired with external socketed RAM.
Maursault 3y ago
> Local LLMs do best on big GPUs, which aren’t an option on Macs. What Apple has done with the M-series silicon is really impressive, but it’s not as fast as big, dedicated GPU silicon. It'd be pretty silly for Apple to cram in big, dedicated GPU when 99.9% of their customers don't care and don't want to pay for it, especially considering that anyone that does want big, dedicated GPU can outboard as much GPU as they want.[1] And many seem to think that the onboard GPU along with Neural Engine should be adequate for local LLM. [1] https://support.apple.com/en-us/HT208544 https://support.apple.com/en-us/HT208544
- thoughtsimple 3y ago
  Only on Intel Macs which are going away soon (only 2019 Mac Pro is remaining outside of refurbished.) Apple Silicon doesn't currently have any provision for external graphics cards.
  - Maursault 3y ago
    There are two issues. 1) Apple Silicon won't support PCIe. 2) Apple doesn't want it to. #2 means Apple is taking nVidia and AMD head on in the GPU space. Apple wants to control everything, and allowing these competitors on their platform is giving away too much. Because Apple Silicon scales better than competitors' hw, the desire for third party GPU is probably going to evaporate within a few generations of Apple Silicon. I mean, we'll see, but that is my best guess, because it seems like that was an intentional decision rather than oversight.
    - bhj 3y ago
      AS supports Thunderbolt just fine. Isn't the lack of eGPU support due to lack of ARM drivers for them?
      - Maursault 3y ago
        To suggest it is merely lack of drivers is an oversimplification. There is a chip errata that prevents PCIe GPUs from working properly on Apple Silicon: the architectures are not compatible. Apple Silicon GPU drivers are deeply integrated into the system. Due to this integration, only graphics cards that use the same GPU architecture as Apple Silicon could be supported, and there just aren't any, and I don't see how there could be unless Apple developed one and released it.
dacryn 3y ago
they share their main memory with the GPU. There is no cheaper way to have 50gb of memory allocated to your GPU compute pipeline (buy a 64gb macbook pro or studio). So yes big, only the fast aspect remains a reason to buy expensive dedicated gpu's
diffeomorphism 3y ago
> The price of entry level M1/M2 Macs is extremely reasonable, IMO. Caveats: In the US, with low ram and low storage. Without these caveats you look at about 2000€ or so (m2 macbook air with 16gb of ram and non-halved storage performance in Germany). That can still be a decent deal for what you get, but "extremely reasonable" -- not quite.

nordsieck 3y ago

It is a little funny... but the combination of pretty good integrated graphics and (potentially) large amounts of on package ram mean that macs are actually a pretty inexpensive way to get access to a GPU with lots of graphics memory.

braymundo 3y ago
I wish gaming on Mac became more popular due to the same reason.
- heliophobicdude 3y ago
  What are some broken-down reasons why gaming on Mac didn't take off. I'm sure it's a few different reasons all together
  - braymundo 3y ago
    I don't know, but if I had to guess it would that Apple was never really interested in it or in supporting it. I fondly remember some companies (like Blizzard) who would always support Macs, until they stopped (looking at you, Diablo 4).
  - dagmx 3y ago
    A lot of people will likely blame APIs etc, but that’s just an easy scapegoat for people outside the game dev industry. The reality is likely just market share, hardware and the makeup of that market share. Macs used to actually be a big gaming platform once upon a time. Myst was a Mac exclusive (built with HyperCard) and Bungie used to make games for macs first before getting bought by Microsoft. But windows got the upper edge: 1. Much larger slice of the market 2. More of the market is made up of gaming enthusiasts than Macs which are usually either targeting light use, education or professional. 3. You can make affordable gaming PCs with windows. You can’t really do it with any kind of Mac because there’s no product for that market between the Mac Mini and iMac (which were targeted at more casual users) up to the Mac Pro and iMac Pro (which were targeted at professional users) It’s easier to target the biggest piece of the pie and the one that will buy your products. That’s why the “API” reason never makes sense, because iOS is THE dominant gaming platform despite having the same APIs as a mac. Game engines often need to support multiple graphics APIs anyway. So Mac gaming did take off, but it fumbled and never recovered. Part of that is that Apple themselves fumbled as a company, on the verge of bankruptcy before the return of Jobs. What made them successful post his return is also what killed gaming on their platforms: they targeted a different audience.

Gare 3y ago

Apple was cost-effective most of the time if you wanted high-quality tools.

foldr 3y ago

If you care about the combination of portability, battery life, performance, and display+audio quality, then Apple’s laptops have been price competitive for quite a while now. People who think they are overpriced usually don’t care about at least one of these things.

KaoruAoiShiho 3y ago

I'm not aware of any local-LLM framework that works with Apple GPUs?

Maursault 3y ago
Does this qualify?[1] [1] https://www.infoq.com/news/2023/05/mlc-llm-mobile-laptops/ https://www.infoq.com/news/2023/05/mlc-llm-mobile-laptops/

raxxorraxor 3y ago

I think it is least likely that we will run local AI instances on Apple hardware to be honest. But yes, the comparison would be apt. I think people want to solve a problem with dependence on third party services... No, I expect GPUs with insane amount of memory surfacing and that isn't really a field for Apple yet.

zitterbewegung 3y ago

Not many laptops have 64 gb of RAM that can run llama.cpp https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3db526d5d6fd https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...

ac29 3y ago
Plenty of non-Apple laptops support 64GB RAM. For example, both the AMD and Intel variants of the Framework Laptop can take 64GB. All of of the large vendors have laptops that support a 64GB configuration too (HP, Dell, Lenovo).