4 ms·

I have a 5940x with 128 gb ram. It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I c

by tyfon 2y ago

I have a 5940x with 128 gb ram. It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I can offload as much of it as possible to the GPU. This works especially well with models like mixtral 8x22, but also models like llama3 and the old large bloom model. I also get the utility of running Linux instead of the closed up mac os. But running large models locally is not exclusive to mac studio, you can do the same on PC for a much lower cost.

Terretta 2y ago
I get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4. > closed up MacOS https://github.com/apple-oss-distributions/distribution-macOS https://github.com/apple-oss-distributions/distribution-macO... curl https://alx.sh | sh https://asahilinux.org/ https://asahilinux.org/ I prefer the "utility" of BSDs, but that's just a preference.
- talldayo 2y ago
  Asahi is Fischer-Price tiers of support compared to what even Nvidia, the most loathed OEM on Linux, provides for free to their users. If that's the best option available, it should be no wonder that server customers are avoiding Apple like the plague. Apple has to beg their audience to reverse-engineer their own OpenCL drivers if they want them; Nvidia ships them alongside CUDA. These two companies are not the same. Have you ever seen the inside of a datacenter? Why is it that surprising to you that nobody perks up when you start waxing on about battery life? Even terms of power-to-performance, Apple's latest chips get ethered by Nvidia's server offerings. This "Apple for Inference" meme is so dead that I can only feel sad when I see people unironically promoting it. You actually think serious customers are going to load up Asahi (even funnier, MacOS) on their Mac Pro... so they can inference half as fast as a single Blackwell GPU? You think the industry is doing this shit? I don't even think the Steve Jobs apologists are dumb enough to fall for this one, you must be a particularly aspirational shareholder.

by tyfon 2y ago

Terretta 2y ago

I get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4. > closed up MacOS https://github.com/apple-oss-distributions/distribution-macOS https://github.com/apple-oss-distributions/distribution-macO... curl https://alx.sh | sh https://asahilinux.org/ https://asahilinux.org/ I prefer the "utility" of BSDs, but that's just a preference.

talldayo 2y ago
Asahi is Fischer-Price tiers of support compared to what even Nvidia, the most loathed OEM on Linux, provides for free to their users. If that's the best option available, it should be no wonder that server customers are avoiding Apple like the plague. Apple has to beg their audience to reverse-engineer their own OpenCL drivers if they want them; Nvidia ships them alongside CUDA. These two companies are not the same. Have you ever seen the inside of a datacenter? Why is it that surprising to you that nobody perks up when you start waxing on about battery life? Even terms of power-to-performance, Apple's latest chips get ethered by Nvidia's server offerings. This "Apple for Inference" meme is so dead that I can only feel sad when I see people unironically promoting it. You actually think serious customers are going to load up Asahi (even funnier, MacOS) on their Mac Pro... so they can inference half as fast as a single Blackwell GPU? You think the industry is doing this shit? I don't even think the Steve Jobs apologists are dumb enough to fall for this one, you must be a particularly aspirational shareholder.