7 ms·

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while

by behnamoh 10mo ago

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM than more devices (centralized will always be faster than distributed). - +1TB/s bandwidth. For the past 3 generations, the speed has been 800GB/s... - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP.

tylerflick 10mo ago
> TOTALLY okay with it consuming +600W energy The 2019 i9 Macbook Pro has entered the chat.
burnt-resistor 10mo ago
Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/filer. I'm disappointed Apple has a persistent blindspot preventing them from succeeding in data center-quality gear category when they could've had virtualized servers, networking, and storage, things that would eventually find their way into my home lab after 5-7 years.
- Terretta 10mo ago
  > I'm disappointed Apple has a persistent blindspot preventing them from succeeding in ... things that would eventually find their way into my home lab after 5-7 years. I can see the dollar signs in their eyes right now. Aftermarkets are a nice reflection of durable value, and there's a massive one for iPhones and a smaller one for quick flameout startup servers, but not much money in 5 - 7 year old servers.
- donavanm 10mo ago
  Enterprise never ever mattered, and there arent enough digits available to show your “home lab” use case in the revenue numbers. Xserve, the RAID shelves, and the directory services were kinda there as a half hearted attempt for that late 90-00s AV setup. All of that fell on the cutting room floor once personal devices, esp iphone, was realized. By the time I left in ‘10 the total revenue from mac hardware was like 15% of revenue. Im honestly surprised theres anyone who cared enough to package the business services for mac minis. So if everything else is printing cash for a HUGE addressable consumer market at premium price points why would they try and compete with their own ODMs on more-or-less commodity enterprise gear?
  - SoftTalker 10mo ago
    Seems like I remember the main reason Macs survived as a product at all was because you needed one to develop for iOS. That may be an exaggeration but there certainly was a time when Macs were few and far between outside of creative shops. Certainly they were almost unseen in the corporate world, where now they are fairly common at least in laptops.
    - pjmlp 10mo ago
      Macs survived because Apple got a cash injection, survived long enough to come out with colorful iMacs with an hockey puck mouse, still running on Mac OS 8, and the iPod. Requiring one for doing iOS development they were already back into the green.
      - raw_anon_1111 10mo ago
        It’s a myth that the “cash injection” from Microsoft saved Apple. Microsoft gave Apple $250 million. The next quarter Apple turned around and spent $100 million on PowerComputing’s Mac assets. Apple lost over a billion more before it became profitable. The $150 Net wouldn’t have been make or break. Now Microsoft promising to keep Office on the Mac was a big deal
        10mo ago
- PunchyHamster 10mo ago
  For Apple, datacenter stuff is low margin business
  - spacedcowboy 10mo ago
    Considering that Apple is moving away from Linux in the datacenter to its own devices, I'm not sure that's the case. The apple machines aren't available to the consumer (they're rack-mounted, dozens of chips per PCB board, custom-made machines) but they're much less power-hungry, just as fast (or more so), much cheaper for them to make rather than buy, and natively support their own ecosystem. Some of the machine-designs that consumers are able to buy seem to have a marked resemblance to the feature-set that the datacenter people were clamouring for. Just saying...
    - mcculley 10mo ago
      > dozens of chips per PCB board Have there been leaks or something about these internal machines? I am curious to know more.
      - spacedcowboy 10mo ago
        No idea. I worked on them so I didn't really need the leak :)
    - deleted 10mo ago
      [deleted]
angoragoats 10mo ago
> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP. This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which: * Adds thousands of dollars of cost, * Adds 150W or more of power usage and the accompanying loud fan noise that comes with that, * And perhaps most importantly adds measurable latency to a networking stack that is already higher latency than the RDMA approach used by the TB5 setup in the OP.
- fenced_load 10mo ago
  Mikrotik has a switch that can do 6x200g for ~$1300 and <150W. https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_crs812_8ds_2dq_2ddq_rm_crs812_ddq_cloud_router.html/overview?ap=y&smp=Y&srsltid=AfmBOorjdgkxvg6LYd2W9GeOOQ8NXfmJAzGXk5fAejYsIV24uwzikaFfZ4k https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
  - angoragoats 10mo ago
    Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.
    - nicky_nickell 10mo ago
      how significant is the latency hit?
      - angoragoats 10mo ago
        The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt. Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also. And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.
zozbot234 10mo ago
> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of?
- fooblaster 10mo ago
  Might be helpful if they actually provided a programming model for ANE that isn't onnx. ANE not having a native development model just means software support will not be great.
  - sroussey 10mo ago
    onnx supports CoreML, is that how?
- liuliu 10mo ago
  They were talking about neural accelerators (a silicon piece on GPU): https://releases.drawthings.ai/p/metal-flashattention-v25-w-neural https://releases.drawthings.ai/p/metal-flashattention-v25-w-...
- csdreamer7 10mo ago
  > Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. Or, Apple could pay for the engineers to add it.
  - ls612 10mo ago
    Apple already paid software engineers to add Tensorflow support for the ANE hardware.
- pdpi 10mo ago
  > Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of? If you daisy chain four nodes, then traffic between nodes #1 and #4 eat up all of nodes #2 and #3's bandwidth, and you eat a big latency penalty. So, absent a switch, the fully connected mesh is the only way to have fast access to all the memory.
Dylan16807 10mo ago
> +1TB/s bandwidth. For the past 3 generations, the speed has been 800GB/s... M4 already hit the necessary speed per channel, and M5 is well above it. If they actually release an Ultra that much bandwidth is guaranteed on the full version. Even the smaller version with 25% fewer memory channels will be pretty close. We already know Max won't get anywhere near 1TB/s since Max is half of an Ultra.
wtallis 10mo ago
> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/ https://www.apple.com/mac-studio/specs/
- QuantumNomad_ 10mo ago
  Jeff mentioned in the video that only three of the ports can be used for RDMA. But it’s unclear where that limitation is coming from.
  - sroussey 10mo ago
    I bet there is one piece of silicon per two ports.
  - geerlingguy 10mo ago
    From my brief discussion with Exo/Apple, it sounds like that is just a limitation of this initial rollout, but it's not a hardware limitation. Though, I am always leery to recommend any decisions be made over something that's not already proven to work, so I would say don't bet on all ports being able to be used. They very well may be able to though.
- kappuchino 10mo ago
  He corrected that in the comment section of the youtube video. Six is actually the maximum amount. He just didn't want to buy another one. He also published the Benchmarks in Detail and with two/four Macs in Comparison: https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 https://github.com/geerlingguy/beowulf-ai-cluster/issues/17
  - re-thc 10mo ago
    > He just didn't want to buy another one. Wasn’t it loaned ie didn’t buy any at all? Apple should have loaned enough to flex.
dev_l1x_be 10mo ago
Mine is to remove the extreme Macos bloat.
Marsymars 10mo ago
> - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. I don't think the Mac Studio has a thermal design capable of dissipating 650W of heat for anything other than bursty workloads. Need to look at the Mac Pro design for that.
- jauntywundrkind 10mo ago
  The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little. Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into the chip with adequate cooling, but the gain is so unremarkable. +10% +15% +20% is almost never going to be a make or break difference for your work, and doing so at double or triple the power budget is an egregious waste. So many of the chips about are already delivered at way higher than optimum efficiency, largely for bragging rights. The exponential decay of efficiency you keep pushing for is an anti-quest, is against good. The absolute performance wins are ridiculous to seek. In almost all cases. If your problem will not scale and dumping a ton of power into one GPU or one cpu socket is all you got, fine, your problem is bad and you have to deal with that. But for 90% of people, begging for more power proces you don't actually know jack & my personal recommendation is that all such points of view deserve massive down voting by anyone with half a brain. Go back to 2018 and look at Matthew Dillon on DragobflyBSD underpowering the heck out of their 2990wx ThreadRipper. Efficiency just soars as you tell the chip to take less power. The situation has not improved! Efficiency skyrockets today at least as much as it did then by telling chips not to go all out. Good chips behave & reward. I believe Apple competent enough to thoroughly disabuse this position that this chip would be far better if we could dump 2x 3x more power into it. Just a fools position, beyond a joke, imo. https://apollo.backplane.com/DFlyMisc/threadripper.txt https://apollo.backplane.com/DFlyMisc/threadripper.txt
  - Marsymars 10mo ago
checker659 10mo ago
For a company that has repeatedly ignored macOS, your wishlist seems anything but a pipe dream. QSFP on a mac. Yeah right. If anything, they’ll double down on TB or some nonstandard interconnect. What is a computer? (Although, I do hope with the new work on supporting RDMA, the MLX5 driver shipped with macOS will finally support RDMA for ConnectX NICs) https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/ https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/
- rbanffy 10mo ago
  QSFP makes sense on a MacPro platform - and might be where Apple chooses to differentiate (one could dream of an M5 Mega, with four chiplets). The Mac Studio is a general purpose compact workstation that doesn’t need ludicrously fast networking beyond what 10Gbe and TB5 offer. It’s already overkill for the vast majority of users. Top configuration Studios are already a niche product.
  - justincormack 10mo ago
    Also given that Apple are using these in their datacenters I think they will ship much more server like hw.
  - checker659 10mo ago
    Apple already ships an MLX5 driver for ConnectX NICs.
lostmsu 10mo ago
> 3090 would be nice They would need 3x speedup over the current generation to approach 3090. A100 that has +- the 3090 compute but 80GB VRAM (so fits LLaMA 70B) does prefill at 550tok/s on a single GPU: https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacpp_benchmark_on_a100/ https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacp...
- doctorpangloss 10mo ago
  the GB10 is only the same performance as a 3090. gb10 uses way less power. i'm not sure why anyone would buy a mac studio instead of a gb10 machine for this use case.
  - villgax 10mo ago
    it's just people looking to do experiments locally on the main machine rather than just get a dedicated spark, which can be used properly as a headless box than a Mac of which you are at the mercy of system shenanigans albiet still bearable compared to windows
  - rbanffy 10mo ago
    > i'm not sure why anyone would buy a mac studio instead of a gb10 For an AI-only use case, the GB10s make sense, but they are only OK as desktop workstations, and I’m not sure for how long DGX OS will be updated, as dedicated AI machines have somewhat short lives. Apple computers, OTOH, have much longer lives, and desktops live the longest. I retired my Mac Mini a year after the machine was no longer getting OS updates, and it was still going strong.
    - FuriouslyAdrift 10mo ago
      DGX OS (based on Ubuntu) is used for all of nVidias GPU compute systems so it's probably going to be around for a while.
      - rbanffy 10mo ago
wartywhoa23 10mo ago
Would you please mind leaving some RAM to remain available for purchase at an affordable price for us mere mortals ? 1Tb for what, like, "Come on AI, make the humankind happy now"? /"s"
- RunSet 10mo ago
  AI bubble will do wonders for used RAM prices when it pops.
  - ddalex 10mo ago
    There is no popping. We cannot have enough compute for the forseeable future.
    - zhengyi13 10mo ago
      "The market can stay irrational longer than you can stay solvent"
    - SV_BubbleTime 10mo ago
      Look at this guy on his first ram shortage.
      - burnte 10mo ago
        I've been in this game so long, seen so many shortages that I'm not even worried. Right now prices are high, manufacturers are switching production, and in 6 months there's going to be a glut of supply. It's all a long game, folks. Play it long.
        brianwawok 10mo ago
        Long game is fine for optional upgrades. “I really wish my game system had 20% better graphics”. Less good when your system crashes and you need something new to work on Monday.
        10mo ago

tylerflick 10mo ago

> TOTALLY okay with it consuming +600W energy The 2019 i9 Macbook Pro has entered the chat.

burnt-resistor 10mo ago

Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/filer. I'm disappointed Apple has a persistent blindspot preventing them from succeeding in data center-quality gear category when they could've had virtualized servers, networking, and storage, things that would eventually find their way into my home lab after 5-7 years.

Terretta 10mo ago
> I'm disappointed Apple has a persistent blindspot preventing them from succeeding in ... things that would eventually find their way into my home lab after 5-7 years. I can see the dollar signs in their eyes right now. Aftermarkets are a nice reflection of durable value, and there's a massive one for iPhones and a smaller one for quick flameout startup servers, but not much money in 5 - 7 year old servers.
donavanm 10mo ago
Enterprise never ever mattered, and there arent enough digits available to show your “home lab” use case in the revenue numbers. Xserve, the RAID shelves, and the directory services were kinda there as a half hearted attempt for that late 90-00s AV setup. All of that fell on the cutting room floor once personal devices, esp iphone, was realized. By the time I left in ‘10 the total revenue from mac hardware was like 15% of revenue. Im honestly surprised theres anyone who cared enough to package the business services for mac minis. So if everything else is printing cash for a HUGE addressable consumer market at premium price points why would they try and compete with their own ODMs on more-or-less commodity enterprise gear?
- SoftTalker 10mo ago
  Seems like I remember the main reason Macs survived as a product at all was because you needed one to develop for iOS. That may be an exaggeration but there certainly was a time when Macs were few and far between outside of creative shops. Certainly they were almost unseen in the corporate world, where now they are fairly common at least in laptops.
  - pjmlp 10mo ago
    Macs survived because Apple got a cash injection, survived long enough to come out with colorful iMacs with an hockey puck mouse, still running on Mac OS 8, and the iPod. Requiring one for doing iOS development they were already back into the green.
    - raw_anon_1111 10mo ago
      It’s a myth that the “cash injection” from Microsoft saved Apple. Microsoft gave Apple $250 million. The next quarter Apple turned around and spent $100 million on PowerComputing’s Mac assets. Apple lost over a billion more before it became profitable. The $150 Net wouldn’t have been make or break. Now Microsoft promising to keep Office on the Mac was a big deal
      - 10mo ago
PunchyHamster 10mo ago
For Apple, datacenter stuff is low margin business
- spacedcowboy 10mo ago
  Considering that Apple is moving away from Linux in the datacenter to its own devices, I'm not sure that's the case. The apple machines aren't available to the consumer (they're rack-mounted, dozens of chips per PCB board, custom-made machines) but they're much less power-hungry, just as fast (or more so), much cheaper for them to make rather than buy, and natively support their own ecosystem. Some of the machine-designs that consumers are able to buy seem to have a marked resemblance to the feature-set that the datacenter people were clamouring for. Just saying...
  - mcculley 10mo ago
    > dozens of chips per PCB board Have there been leaks or something about these internal machines? I am curious to know more.
    - spacedcowboy 10mo ago
      No idea. I worked on them so I didn't really need the leak :)
  - deleted 10mo ago
    [deleted]

angoragoats 10mo ago

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP. This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which: * Adds thousands of dollars of cost, * Adds 150W or more of power usage and the accompanying loud fan noise that comes with that, * And perhaps most importantly adds measurable latency to a networking stack that is already higher latency than the RDMA approach used by the TB5 setup in the OP.

fenced_load 10mo ago
Mikrotik has a switch that can do 6x200g for ~$1300 and <150W. https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_crs812_8ds_2dq_2ddq_rm_crs812_ddq_cloud_router.html/overview?ap=y&smp=Y&srsltid=AfmBOorjdgkxvg6LYd2W9GeOOQ8NXfmJAzGXk5fAejYsIV24uwzikaFfZ4k https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
- angoragoats 10mo ago
  Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.
  - nicky_nickell 10mo ago
    how significant is the latency hit?
    - angoragoats 10mo ago
      The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt. Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also. And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.

zozbot234 10mo ago

> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of?

fooblaster 10mo ago
Might be helpful if they actually provided a programming model for ANE that isn't onnx. ANE not having a native development model just means software support will not be great.
- sroussey 10mo ago
  onnx supports CoreML, is that how?
liuliu 10mo ago
They were talking about neural accelerators (a silicon piece on GPU): https://releases.drawthings.ai/p/metal-flashattention-v25-w-neural https://releases.drawthings.ai/p/metal-flashattention-v25-w-...
csdreamer7 10mo ago
> Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. Or, Apple could pay for the engineers to add it.
- ls612 10mo ago
  Apple already paid software engineers to add Tensorflow support for the ANE hardware.
pdpi 10mo ago
> Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of? If you daisy chain four nodes, then traffic between nodes #1 and #4 eat up all of nodes #2 and #3's bandwidth, and you eat a big latency penalty. So, absent a switch, the fully connected mesh is the only way to have fast access to all the memory.

Dylan16807 10mo ago

> +1TB/s bandwidth. For the past 3 generations, the speed has been 800GB/s... M4 already hit the necessary speed per channel, and M5 is well above it. If they actually release an Ultra that much bandwidth is guaranteed on the full version. Even the smaller version with 25% fewer memory channels will be pretty close. We already know Max won't get anywhere near 1TB/s since Max is half of an Ultra.

wtallis 10mo ago

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/ https://www.apple.com/mac-studio/specs/

QuantumNomad_ 10mo ago
Jeff mentioned in the video that only three of the ports can be used for RDMA. But it’s unclear where that limitation is coming from.
- sroussey 10mo ago
  I bet there is one piece of silicon per two ports.
- geerlingguy 10mo ago
  From my brief discussion with Exo/Apple, it sounds like that is just a limitation of this initial rollout, but it's not a hardware limitation. Though, I am always leery to recommend any decisions be made over something that's not already proven to work, so I would say don't bet on all ports being able to be used. They very well may be able to though.
kappuchino 10mo ago
He corrected that in the comment section of the youtube video. Six is actually the maximum amount. He just didn't want to buy another one. He also published the Benchmarks in Detail and with two/four Macs in Comparison: https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 https://github.com/geerlingguy/beowulf-ai-cluster/issues/17
- re-thc 10mo ago
  > He just didn't want to buy another one. Wasn’t it loaned ie didn’t buy any at all? Apple should have loaned enough to flex.

dev_l1x_be 10mo ago

Mine is to remove the extreme Macos bloat.

Marsymars 10mo ago

> - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. I don't think the Mac Studio has a thermal design capable of dissipating 650W of heat for anything other than bursty workloads. Need to look at the Mac Pro design for that.

jauntywundrkind 10mo ago
The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little. Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into the chip with adequate cooling, but the gain is so unremarkable. +10% +15% +20% is almost never going to be a make or break difference for your work, and doing so at double or triple the power budget is an egregious waste. So many of the chips about are already delivered at way higher than optimum efficiency, largely for bragging rights. The exponential decay of efficiency you keep pushing for is an anti-quest, is against good. The absolute performance wins are ridiculous to seek. In almost all cases. If your problem will not scale and dumping a ton of power into one GPU or one cpu socket is all you got, fine, your problem is bad and you have to deal with that. But for 90% of people, begging for more power proces you don't actually know jack & my personal recommendation is that all such points of view deserve massive down voting by anyone with half a brain. Go back to 2018 and look at Matthew Dillon on DragobflyBSD underpowering the heck out of their 2990wx ThreadRipper. Efficiency just soars as you tell the chip to take less power. The situation has not improved! Efficiency skyrockets today at least as much as it did then by telling chips not to go all out. Good chips behave & reward. I believe Apple competent enough to thoroughly disabuse this position that this chip would be far better if we could dump 2x 3x more power into it. Just a fools position, beyond a joke, imo. https://apollo.backplane.com/DFlyMisc/threadripper.txt https://apollo.backplane.com/DFlyMisc/threadripper.txt
- Marsymars 10mo ago

checker659 10mo ago

For a company that has repeatedly ignored macOS, your wishlist seems anything but a pipe dream. QSFP on a mac. Yeah right. If anything, they’ll double down on TB or some nonstandard interconnect. What is a computer? (Although, I do hope with the new work on supporting RDMA, the MLX5 driver shipped with macOS will finally support RDMA for ConnectX NICs) https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/ https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/

rbanffy 10mo ago
QSFP makes sense on a MacPro platform - and might be where Apple chooses to differentiate (one could dream of an M5 Mega, with four chiplets). The Mac Studio is a general purpose compact workstation that doesn’t need ludicrously fast networking beyond what 10Gbe and TB5 offer. It’s already overkill for the vast majority of users. Top configuration Studios are already a niche product.
- justincormack 10mo ago
  Also given that Apple are using these in their datacenters I think they will ship much more server like hw.
- checker659 10mo ago
  Apple already ships an MLX5 driver for ConnectX NICs.

lostmsu 10mo ago

> 3090 would be nice They would need 3x speedup over the current generation to approach 3090. A100 that has +- the 3090 compute but 80GB VRAM (so fits LLaMA 70B) does prefill at 550tok/s on a single GPU: https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacpp_benchmark_on_a100/ https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacp...

doctorpangloss 10mo ago
the GB10 is only the same performance as a 3090. gb10 uses way less power. i'm not sure why anyone would buy a mac studio instead of a gb10 machine for this use case.
- villgax 10mo ago
  it's just people looking to do experiments locally on the main machine rather than just get a dedicated spark, which can be used properly as a headless box than a Mac of which you are at the mercy of system shenanigans albiet still bearable compared to windows
- rbanffy 10mo ago
  > i'm not sure why anyone would buy a mac studio instead of a gb10 For an AI-only use case, the GB10s make sense, but they are only OK as desktop workstations, and I’m not sure for how long DGX OS will be updated, as dedicated AI machines have somewhat short lives. Apple computers, OTOH, have much longer lives, and desktops live the longest. I retired my Mac Mini a year after the machine was no longer getting OS updates, and it was still going strong.
  - FuriouslyAdrift 10mo ago
    DGX OS (based on Ubuntu) is used for all of nVidias GPU compute systems so it's probably going to be around for a while.
    - rbanffy 10mo ago

wartywhoa23 10mo ago

Would you please mind leaving some RAM to remain available for purchase at an affordable price for us mere mortals ? 1Tb for what, like, "Come on AI, make the humankind happy now"? /"s"

RunSet 10mo ago
AI bubble will do wonders for used RAM prices when it pops.
- ddalex 10mo ago
  There is no popping. We cannot have enough compute for the forseeable future.
  - zhengyi13 10mo ago
    "The market can stay irrational longer than you can stay solvent"
  - SV_BubbleTime 10mo ago
    Look at this guy on his first ram shortage.
    - burnte 10mo ago
      I've been in this game so long, seen so many shortages that I'm not even worried. Right now prices are high, manufacturers are switching production, and in 6 months there's going to be a glut of supply. It's all a long game, folks. Play it long.
      - brianwawok 10mo ago
        Long game is fine for optional upgrades. “I really wish my game system had 20% better graphics”. Less good when your system crashes and you need something new to work on Monday.
      - 10mo ago