5 ms·

I think the author doesn’t realize that 99.9% of people with an m1 Mac don’t care at all about running local ml workloads. The people that do care about running

I think the author doesn’t realize that 99.9% of people with an m1 Mac don’t care at all about running local ml workloads. The people that do care about running local ml workloads have either a workstation with an NVidia card, access to virtual machines on the cloud with NVidia processors or some Apple computer with an M2 Max or similar processor. Put simply, this isn’t a big deal at all. Most people are not ml power users and most people are using their Mac for web apps via a browser, local office apps, and something like zoom or slack. Also, as an aside, Siri is mostly useless and I’ve never met anyone who uses it regularly. So whether it needs ml processing or not is irrelevant.

weikju 3y ago
Counterargument: most people are not running directly the ML workloads, agreed - however Apple implements them as features of the OS (like the new GPT powered text predictions). So it WILL affect the regular users over time as new features are not made available for M1 users.
- candiodari 3y ago
  The only reason you want to use bfloat16 is to train "compressed" models. Best inference performance tends to be at 4 bith quantization (and many people are trying to make it go down, even to 1 bit, and with sparsity, actually below that) So this just doesn't matter. Ironically, memory speed is what matters. And that's exactly the point where the M1 actually beats even the M3.
  - weikju 3y ago
    Thanks. I was going to rebut your point but I re-read the article and while one of the headings seem to rebut it, the actual content does not contradict anything, so I'll agree with your argument :)
fmajid 3y ago
I have a M1 Ultra Mac Studio with 128GB of RAM and the ability to run LLMs like LLaMa2 locally thanks to the unified memory architecture was a big factor. Just because you don't have a use for it doesn't mean everyone else is as blinkered as you. As soon as the M3 Ultra lands, I am getting one, preferably with 256GB RAM.
- mccrory 3y ago
  Where is my M3 Ultra MacBook Pro? I’m okay with reduced battery if I can have 256GB of ram and the processing power.
  - fmajid 3y ago
    Have you seen the size of the heatsink on the Mac Studio Ultra? It's there for a reason...
    - rsynnott 3y ago
      It's a 60W part, so it would probably be _just_ about doable (old MBPs had up to about 75W worth of stuff to cool between CPU and GPU, though they definitely struggled a bit and the battery life on the dGPU ones was never amazing).
cassianoleal 3y ago
I have an M2 Pro with 32Gb RAM. I keep ollama running in the background and use it either on the terminal for a chat interface, or as a code assistant on VS Code. It works pretty great. Of course, larger models take longer to respond or don't work at all but that hasn't been an issue for my use-case.
- minton 3y ago
  Do you mind sharing how you use ollama in VS Code?
  - cassianoleal 3y ago
    Hey, sure. It's nothing fancy really. I've been using 2 extensions: Continue.dev [0] and Llama Coder [1]. The former for a chat-like interface where you can ask questions about the code, request it to write a function, tests, refactor, etc. The second for an inline, more Intellisense-like completion. I'm on a Mac, so installed ollama from Homebrew, then `homebrew services enable ollama` to have it always in the background. Configuring the extensions to use it is well documented, I imagine you won't have any trouble but let me know if you get stuck. [0] https://continue.dev/ https://continue.dev/ [1] https://marketplace.visualstudio.com/items?itemName=ex3ndr.llama-coder https://marketplace.visualstudio.com/items?itemName=ex3ndr.l...
latexr 3y ago
> I think the author doesn’t realize that 99.9% of people with an m1 Mac don’t care at all about running local ml workloads. > (…) > Put simply, this isn’t a big deal at all. The author isn’t making a consumer argument, but a statement of fact. Which is perfectly in line with this blog, its author explores technical details of Macs but doesn’t necessarily make a value judgment on if something is good or bad, it’s about explaining how something works. The typical readers of this blog are the ones which will care about these details.
bdcravens 3y ago
Most won't know they're running ML workloads. All they know is that cool new Photoshop feature doesn't work as well on their computer as their coworkers'.
- saberience 3y ago
  But this has always been the case with older computers simply due to the inevitably faster processors coming out every year, and it hasn’t changed now. M1 processors were released more than 3 years ago. At anytime in the last 30 years of computing, if you have a three year old computer it won’t run the latest version of Photoshop, Maya, whatever videogame, as well as the newest machines. This was the case when I had a 16Mhz Mac and I saw people buying computers with 66Mhz computers and it’s still the case now. Technology moves forward and new stuff will start to run slow on older machines, it’s not revelatory news.
poulsbohemian 3y ago
>Siri is mostly useless I would normally agree with you, but both my daughter and I had experiences with it this past week where we asked it to provide information while we were driving and it actually did - correctly even. We both commented that it felt like something had changed or improved
Tagbert 3y ago
“ Siri is mostly useless and I’ve never met anyone who uses it regularly. So whether it needs ml processing or not is irrelevant.” You should get out more. I use Siri daily to switch lights, set reminders, turn on timers, and play music. It works well within limited domains like that. I don’t try to use it as an answerbot. That is a much bigger task and it is not suited for that. Certainly Siri (and Alex and Google Assistant) is not very good right now for more complex tasks. That’s why Apple’s announcement that they will be adding LLM-style AI in the next OS interesting as it could make Siri more reliable and useful for more people.
ametrau 3y ago
Yeah, why can’t you just log in remotely?

weikju 3y ago

Counterargument: most people are not running directly the ML workloads, agreed - however Apple implements them as features of the OS (like the new GPT powered text predictions). So it WILL affect the regular users over time as new features are not made available for M1 users.

candiodari 3y ago
The only reason you want to use bfloat16 is to train "compressed" models. Best inference performance tends to be at 4 bith quantization (and many people are trying to make it go down, even to 1 bit, and with sparsity, actually below that) So this just doesn't matter. Ironically, memory speed is what matters. And that's exactly the point where the M1 actually beats even the M3.
- weikju 3y ago
  Thanks. I was going to rebut your point but I re-read the article and while one of the headings seem to rebut it, the actual content does not contradict anything, so I'll agree with your argument :)

fmajid 3y ago

I have a M1 Ultra Mac Studio with 128GB of RAM and the ability to run LLMs like LLaMa2 locally thanks to the unified memory architecture was a big factor. Just because you don't have a use for it doesn't mean everyone else is as blinkered as you. As soon as the M3 Ultra lands, I am getting one, preferably with 256GB RAM.

mccrory 3y ago
Where is my M3 Ultra MacBook Pro? I’m okay with reduced battery if I can have 256GB of ram and the processing power.
- fmajid 3y ago
  Have you seen the size of the heatsink on the Mac Studio Ultra? It's there for a reason...
  - rsynnott 3y ago
    It's a 60W part, so it would probably be _just_ about doable (old MBPs had up to about 75W worth of stuff to cool between CPU and GPU, though they definitely struggled a bit and the battery life on the dGPU ones was never amazing).

cassianoleal 3y ago

I have an M2 Pro with 32Gb RAM. I keep ollama running in the background and use it either on the terminal for a chat interface, or as a code assistant on VS Code. It works pretty great. Of course, larger models take longer to respond or don't work at all but that hasn't been an issue for my use-case.

minton 3y ago
Do you mind sharing how you use ollama in VS Code?
- cassianoleal 3y ago
  Hey, sure. It's nothing fancy really. I've been using 2 extensions: Continue.dev [0] and Llama Coder [1]. The former for a chat-like interface where you can ask questions about the code, request it to write a function, tests, refactor, etc. The second for an inline, more Intellisense-like completion. I'm on a Mac, so installed ollama from Homebrew, then `homebrew services enable ollama` to have it always in the background. Configuring the extensions to use it is well documented, I imagine you won't have any trouble but let me know if you get stuck. [0] https://continue.dev/ https://continue.dev/ [1] https://marketplace.visualstudio.com/items?itemName=ex3ndr.llama-coder https://marketplace.visualstudio.com/items?itemName=ex3ndr.l...

latexr 3y ago

> I think the author doesn’t realize that 99.9% of people with an m1 Mac don’t care at all about running local ml workloads. > (…) > Put simply, this isn’t a big deal at all. The author isn’t making a consumer argument, but a statement of fact. Which is perfectly in line with this blog, its author explores technical details of Macs but doesn’t necessarily make a value judgment on if something is good or bad, it’s about explaining how something works. The typical readers of this blog are the ones which will care about these details.

bdcravens 3y ago

Most won't know they're running ML workloads. All they know is that cool new Photoshop feature doesn't work as well on their computer as their coworkers'.

saberience 3y ago
But this has always been the case with older computers simply due to the inevitably faster processors coming out every year, and it hasn’t changed now. M1 processors were released more than 3 years ago. At anytime in the last 30 years of computing, if you have a three year old computer it won’t run the latest version of Photoshop, Maya, whatever videogame, as well as the newest machines. This was the case when I had a 16Mhz Mac and I saw people buying computers with 66Mhz computers and it’s still the case now. Technology moves forward and new stuff will start to run slow on older machines, it’s not revelatory news.

poulsbohemian 3y ago

>Siri is mostly useless I would normally agree with you, but both my daughter and I had experiences with it this past week where we asked it to provide information while we were driving and it actually did - correctly even. We both commented that it felt like something had changed or improved

Tagbert 3y ago

“ Siri is mostly useless and I’ve never met anyone who uses it regularly. So whether it needs ml processing or not is irrelevant.” You should get out more. I use Siri daily to switch lights, set reminders, turn on timers, and play music. It works well within limited domains like that. I don’t try to use it as an answerbot. That is a much bigger task and it is not suited for that. Certainly Siri (and Alex and Google Assistant) is not very good right now for more complex tasks. That’s why Apple’s announcement that they will be adding LLM-style AI in the next OS interesting as it could make Siri more reliable and useful for more people.

ametrau 3y ago

Yeah, why can’t you just log in remotely?