5 ms·
Apple Is Developing AI Chips for Data Centers, Seeking Edge in Arms Race
- verdverm 2y agoMost of the profits in AI chips for data centers will be in selling and renting said chips. Is Apple going to start offering a cloud or selling chips directly? That would be very unexpected Of all the big tech companies, Apple seems to have missed the boat more than others
- qbxk 2y agothey're going to make the chips, and build a cloud with them and then they're going to sell services with that cloud directly to the consumer. they're going to own the entire stack, from fab to finger
- verdverm 2y agoApple doesn't own a fab currently afaik, that would be new as well
- runjake 2y agoApple is in no position to own fab and besides, we’d hear about it many months prior. They aren’t going to be able to keep that secret. My guess is that they’d use TSMC or perhaps even Intel, for fab. But probably TSMC.
- HeatrayEnjoyer 2y agoApple has more cash on hand than nearly every private sector entity on Earth. They could build a fab with zero investment or credit activity.
- verdverm 2y agoYet they are choosing to do a $110B stock buyback, which apparently exceeds their cash on hand in the books > Since Apple’s $110 billion stock buyback exceeds its balance sheet cash, how will the iPhone maker pay for the share buyback? https://www.forbes.com/sites/petercohan/2024/05/05/warren-buffett-dumps-13-of-apple-stake-before-110-billion-buyback/?sh=c68fd0e3c509#:~:text=Since%20Apple%E2%80%99s%20%24110%20billion%20stock%20buyback%20exceeds%20its%20balance%20sheet%20cash%2C%20how%20will%20the%20iPhone%20maker%20pay%20for%20the%20share%20buyback%3F https://www.forbes.com/sites/petercohan/2024/05/05/warren-bu...
- dagmx 2y agoBuybacks happen over a period. Cash on hand isn’t the only factor, cash flow in is a large one. Apple stated several years ago that they’re aiming to be cash neutral https://www.fool.com/investing/2018/02/02/understanding-apples-new-net-cash-neutral-goal.aspx https://www.fool.com/investing/2018/02/02/understanding-appl... In the earnings call last week they stated they’re still a while away from that goal. These buybacks are part of the net cash neutral strategy. Other forms they could take are upping R&D (but they already seem to be pretty high), or increasing employee wages (that would be much appreciated I’m sure) or reducing product margins (also probably appreciated by a different group). Though the latter two are harder to back out of if things ever go south.
- nxobject 2y agoCan you ELI5 for someone who doesn't know corporate finance why constant cash surpluses would be unfavorable for them?
- runjake 2y agoEven if they could, they haven’t. And if they did, it would take months/years and we would hear about it.
- oldpersonintx 2y ago[dead]
- nxobject 2y agoRight after buying Disney, too.
- HeatrayEnjoyer 2y agoBuying Disney?
- nxobject 2y agoNo doubt Apple can negotiate some form of super-exclusivity deal with a fab, in the same way that their license from ARM is probably more permissive than any other of their partners. (I wouldn't put it past them to go with Intel – after all, during the PPC era, it also would've been unexpected to go with Intel!)
- MBCook 2y agoThey’ve gotten exclusive access to nodes by buying up the full capacity before.
- mr_toad 2y agoI doubt Apple would sell the chips, or even rent them. And they likely won’t even make the models directly available, even as a cloud service, they’ll be embedded in products like Xcode.
- MBCook 2y agoI think that’s how they’d rent them. You’d set up a training project in Xcode and press a button and as part of your Xcode cloud subscription it would run on Apple’s hardware instead of locally. Maybe you are billed by time used. Maybe you have to buy hours upfront. I could see either.
- MBCook 2y agoThey already have Xcode Cloud. Perhaps you could run your training jobs there and they would run on Apple’s new custom silicon. You’d pay for it as a part of your existing subscription. That + internal use would make sense to me.
- Havoc 2y agoBit strange. I thought they don’t even do their own storage (at least partially on GCP) Maybe it’s supposed to be complementary to their on device strategy. Ie split tasks by difficulty
- snowwrestler 2y agoNetwork observations have shown Apple traffic going in and out of AWS and GCP. But they also own their own data centers, for example: https://www.apple.com/newsroom/2017/08/apples-next-us-data-center-will-be-built-in-iowa/ https://www.apple.com/newsroom/2017/08/apples-next-us-data-c...
- pvsnp 2y agoI think this makes sense for Apple to also provide the cloud capabilities similar to NVIDIA DGX so that any AI models that are targeted to Apple chips can run scaled up in data centers but Apple owns the front end for it as well as any ongoing optimizations
- chucke1992 2y agoIt really feels like Apple does not have a real plan now and are just trying to catch up.
- nxobject 2y agoYes – or a use case. Apple seems very happy to advertise that they keep their inference on-device.
- ein0p 2y agoThere’s no way to do freeform chat on device. On-device models can realistically be used for constrained tasks like summarization, RAG, email/instant message replies, that kind of thing. You aren’t going to get anything GPT4 like unless your “device” can fit a model with at least tens, if not hundreds of billions of parameters. Which, to be fair, a number of higher end Apple SKUs can.
- easton 2y agoI think Apple’s said before that they don’t really get the chat usecase, their priority is integrating ML backed features into the OS (text highlighting in images, etc). It seems that even the state of the art models can’t be used to make a more reliable Siri without another few breakthroughs, so it’s probable that these new AI features aren’t chat stuff.
- anon373839 2y agoOh, I think we’ve passed that point already. Llama 3 8B works incredibly well for chat. On the LMSYS English-only leaderboard, it’s only 3 points below GPT-4-0314. (!) It will hallucinate, as every other LLM will also do, so it shouldn’t be used as a store of knowledge. But grounded with context, it is astonishing in its capability. For that matter, Phi-3 at half the size is not bad at all.
- ein0p 2y agoYeah, if you’re OK with it making everything up, it “works incredibly well”. I’m not ok with that, so I actually prefer models where hallucination is minimized as much as possible because they encode massive amounts of knowledge. Points don’t matter. Of all SOTA models, GPT4, the largest one, hallucinates the least. Almost not at all, in fact.
- dagmx 2y agoObviously nobody is competing with NVIDIA right now given their strong CUDA+Mellanox advantage. But that second place is a very important spot. I think if Apple sold to businesses, it would be really bad news for AMD. For comparison, take an MI300A which is AMDs H100 competitor. It’s under half the price ($20k) of an H100 (~$30-50k) with more HB3 memory. It maxes out at something like 128GB of memory. I can’t find the price of a GH100 in todays market. Then there’s Apple. An M2 Ultra Mac Studio is <$7k for 192GB of memory. Granted it’s much slower memory, but that’s not a huge factor for many workloads. So this M2 Ultra machine is 14% of the H100 and 35% of the MI300A. It has more memory and a faster per-core processor that consumes 40% of the power of the MI300A. Now of course there aren’t good benchmarks to compare the exact loadouts and configurations here, and like I mentioned the memory is slower LPDDR5 instead of HBM3. but even then, you can get multiple Mac Studios for the closest competitor. It’ll cost you significantly less to run. The extreme memory bandwidth of HBM3 is not unique to anyone and the price difference would likely more than cover it. So then there’s the software moat. NVIDIA has CUDA and then after them, Apple seemingly has the widest support of ML libraries. AMD has improved a lot with Rocm but they’re still lagging behind Apple and way behind NVIDIA. So anyway, if Apple decides to bring this to market, I would be very worried for anyone vying for second place. The biggest issues will be whether macOS is a turnoff (doubt it) and whether Apple can do enterprise (remains to be seen).
- google234123 2y agoTensorflow seems to run fine on Google TPUs.
- dagmx 2y agoYes but Google don’t sell their hardware to others, so there’s no pricing information to compare it with.
- edward28 2y agoThe mi300x is AMDs h100/200 competitor.
- 2y ago
- gdiamos 2y agoI feel like we screwed up the AI chip segmentation with separate “inference” and “training” chips Every machine learning application ever involves both training and inference. What is the point of a model that can’t learn? What is the point of learning a great model that is never used? It seems like these lead to different optimizations, and they do. But the bigger lock-in is the SW stack. If you own one side of it, you have a huge advantage in SW compatibility that you can leverage into the other.
- dagmx 2y agoMost inference can be easily abstracted over multiple ISV implementations so the inference side rarely is locked in. It’s usually the training side or software above the inference layer that’s locked in. But as to your question as to why: silicon size and frequency of use. You’re almost always going to have a high ratio of inference to training. Silicon for training needs to be much more capable due to how intensive it is. That’s then a block of silicon that isn’t used to its full potential, takes up valuable space and in turn jacks up cost to buy and cost to run (energy). It’s simply not a worthwhile trade off.
- flembat 2y agoThe point of the separation is economic, owners create and train models, and renters are charged to execute them on their edge machines.
- RamblingCTO 2y ago> Every machine learning application ever involves both training and inference. Not really, no. There are a lot of "classical" models that you don't want to update on the fly. I'd even argue that it's a minority.
- RedStarComrade 2y agohttps://www.msn.com/en-us/money/other/apple-is-developing-ai-chips-for-data-centers-seeking-edge-in-arms-race/ar-BB1lVtFy https://www.msn.com/en-us/money/other/apple-is-developing-ai...