14 ms·
The first two custom silicon chips designed by Microsoft for its cloud
- spandextwins 3y agoSo does NVIDIA, just turns out NVIDIA can profit more because of their software and the software ecosystem around it adds so much value, nobody can compete. It's gonna take a lot of work and many years to approach that. Even by leveraging AI. And by diverging with their own chips, they're gonna miss out on the mainstream.
- ugh123 3y agoI don't know much about the software-side of nvidia/gpus + LLMs. Can you catch me up on what software they've created means as a differentiator? Is that CUDA? How does this relate to things like tensorflow with Google's chips?
- jedberg 3y agoIt was only a matter of time. Google announced theirs years ago, Amazon announced theirs last year. Right now NVIDIA has the lead because they have the better software, but they can't make the chips fast enough. Will be interesting to see if their better software continues to keep them in the lead or if people are more interested in getting the capacity in any form.
- rehitman 3y agoI would say software is very important. We already have tons of different models, standards, library, etc. I usually have smooth experience if I am using NVDA, but any other variation, I have to spend some times to get things started. Supporting the common librarires that I use is very important for me to chose the cloud platform.
- Jagerbizzle 3y agoConsidering that Jensen is on stage with Satya at the moment sharing the keynote of Microsoft Ignite, I suspect NVIDIA won't be going anywhere anytime soon.
- paulpan 3y agoA bit surprised to see Jensen's stage appearance since clearly Microsoft's success with its own AI chips means less business for Nvidia's chips. Because different than the ARM chip also announced in the same Ignite event, Microsoft doesn't exactly "need" nor can fully utilize an AI chip. Google trains its foundational models (e.g. Gemini) on its own TPU hardware but Microsoft's is heavily reliant on OpenAI for its generative AI serving needs. Unless Microsoft is planning to acquire OpenAI fully and switch over from Nvidia hardware...
- echelon 3y ago> Unless Microsoft is planning to acquire OpenAI fully They're going to play a modified version of the old Rareware trick. It's also a pretty great game to buy up OpenAI equity, which ultimately gets spent on Microsoft compute. Two birds, one stone.
- WanderPanda 3y agoI don’t think the margins here are so considerable that we can assume revenue = profit
- echelon 3y agoThey can be losing money, but gaining in market share and moat. Almost nobody in this game cares about profit right now.
- kmeisthax 3y ago...they're going to overbid on a studio that was actively falling apart, after being rebuffed from buying one of the biggest giants in the business[0], all as part of an ill-advised attempt to muscle into a game business they didn't understand? [0] Microsoft tried to buy Nintendo very early on
- rapsey 3y agonVidia still has the monopoly on training. Everyone else is just making chips for inference.
- muro 3y agohttps://cloud.google.com/tpu/docs/training-on-tpu-pods https://cloud.google.com/tpu/docs/training-on-tpu-pods
- adrian_b 3y agoWhile NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar. The funny thing is that this fact has been shown inadvertently by NVIDIA: https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x-better-performance-per-dollar-than-its-h100/ https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...
- sgillen 3y agoThat article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.
- adrian_b 3y agoThat 50X is only if you can afford one thousand NVIDIA H100. There cannot be more than a handful of companies in the entire world that could afford such a huge price (tens of millions of $). In comparison with a still extremely expensive cluster of 64 NVIDIA H100, the difference in speed would reduce to only two to three times, and paying several times less for the entire training becomes very attractive.
- lazide 3y agoYou’ve got a really warped sense of scale if you think only a handful of companies couldn’t afford tens of millions of dollars. That’s complete pocket change for any of the Fortune 500.
- belval 3y ago> Amazon announced theirs last year Inferentia (inf1) was GA'ed in December 2019 so it's actually almost 4 years old now. The trainium (trn1) chips and the Inferentia 2 (inf2) refresh is indeed 1 year old though.
- jedberg 3y agoI was referring to trainium.
- ShamelessC 3y agoYikes those names are horrible.
- belval 3y agoWhy? Inferentia => inference, trainium => training. Given the usually naming of AWS product, having one where the name roughly matches what it does is pretty good? TPU is pretty good but is associated with Google. MTIA is an acronym but still maps to what the chip does. ~~"Cobalt" is worse as it does not mean anything~~ . Cobalt is the CPU chip, MAIA is the accelerator so this matches Meta's naming.
- ShamelessC 3y ago> Why? Inferentia => inference, trainium => training. Funny, that's precisely why I think the names are bad. It's like if Google had chosen "Search-ola" as their name. Way too on the nose and/or lazy. Having said that, I don't really care all that much and I imagine that may have been the spirit of those who chose the names.
- mdaniel 3y agoheh, as someone who has to deal with this nonsense all day <https://aws.amazon.com/products/ https://aws.amazon.com/products/> I would for sure welcome some straightforward naming. $(echo "AWS Fargate" | sed s/Fargate/ServerlessContainerium/)
- scarface_74 3y agoIf they “can’t make the chips fast enough” being TSMC’s second highest volume customer behind Apple and probably second in priority, what chance does Microsoft have getting enough of TSMCs capacity?
- donatzsky 3y agoMicrosoft does have the benefit of not having any customers other than themselves, so volumes are smaller.
- tambre 3y agoThey're more constrained by advanced packaging (CoWoS) capacity rather than the manufacturing of the silicon.
- tyfighter 3y agoBut from the pictures, the Maia 100 is also using CoWoS packaging. That will be necessary for any new HBM chip.
- localhost 3y agoThe latest Stratechery interview had Pat Gelsinger on it and he was talking about how Intel is happy to package up TSMC dies in their foundries. Relevant quote: > So it really brings together many of the things that Intel is doing as an IDM, now bringing it together in a heterogeneous environment where we’re taking TSMC dies. We’re going to be using other foundries in the industry, we’re standardizing that with UCIe. So I really see ourself as the front end of this multi-chip chiplet world doing so in the Intel way, standardizing it for the industry’s participation with UCIE, and then just winning a better technology.
- tw04 3y ago> what chance does Microsoft have getting enough of TSMCs capacity? Looking at their balance sheet: plenty. TSMC is all about pay to play. Apple is first in line because they’re willing to pay to be first in line. I have no doubt Microsoft can justify spending the money they save not buying nvidia into getting some priority access from TSMC. Also keep in mind Apple is now on 3nm so there’s likely spare 5nm.
- sofixa 3y ago> Amazon announced theirs last year Amazon has had Graviton chips (after they acquired Annapurna Labs) since 2018. Or do you mean specifically AI-oriented chips?
- kristianp 3y agohttps://archive.is/jzxDG https://archive.is/jzxDG
- pseudosavant 3y agoNot a lot of information about the chips yet. About 100B transistors in the AI chip. For comparison, an RTX 4090 has 76B, and an H100 has about 80B. So the Maia chip is pretty massive.
- hulitu 3y agoThis is like performance review based on written line of code.
- Slartie 3y agoGPUs (and AI chips) are highly parallel, containing thousands upon thousands of the same compute units. The performance of these chips is very much dependent on having a sheer number of transistors to form into as many compute units as possible. If we assume that Microsoft is roughly able to architect compute units of a similar performance-to-number-of-transistors ratio as nVidia is, then having twice the number of transistors should roughly result in twice the performance. That is very different than it is with typical software. If you give a programmer who needs to write 100 lines of code to solve a given problem 100 more lines to fill, he won't simply be able to copy-paste his 100 lines another time and by that action be twice as fast at solving whatever problem you tasked him with. With GPU compute units, such copy-pasting of compute units is exactly what's being done (at least until you hit the limits of other resources such as management units, memory bandwidth etc.).
- kergonath 3y agoThey did not mention performance, just size.
- pseudosavant 3y agoIn a way, it is the opposite. Code is what you execute. The transistors are the engines that do the execution. They are going to expect a chip they designed with 105B transistors to perform (speed/efficiency/whatever) in the same ballpark as a high=end GPU for their AI workloads. It is like knowing the kind of engine a car has. Not all V8 gas engines produce the same power, but knowing that it is a V8 instead of an inline three cylinder does give you an idea of the expected performance characteristics.
- virgildotcodes 3y agoThis seems a little unclear to me. They're saying this isn't meant to compete with Nvidia, and that it's more of an ARM based CPU? So more of an SOC a la AWS graviton or Apple Silicon than a pure GPU?
- jsnell 3y agoThere's two separate chip announcements here. One is a ML accelerator (Maia). The other is an ARM CPU (Cobalt).
- virgildotcodes 3y agoAh, my bad, thanks for pointing that out.
- conradev 3y ago“Microsoft gave few technical details that would allow gauging the chips' competitiveness versus those of traditional chipmakers” Clearly. All I got was “using ARM IP” and “TSMC N5”
- buildbot 3y agoAlso the new MX datatypes (https://www.opencompute.org/blog/amd-arm-intel-meta-microsoft-nvidia-and-qualcomm-standardize-next-generation-narrow-precision-data-formats-for-ai https://www.opencompute.org/blog/amd-arm-intel-meta-microsof...), from the article: "Manufactured on a 5-nanometer TSMC process, Maia has 105 billion transistors — around 30 percent fewer than the 153 billion found on AMD’s own Nvidia competitor, the MI300X AI GPU. “Maia supports our first implementation of the sub 8-bit data types, MX data types, in order to co-design hardware and software,” says Borkar. “This helps us support faster model training and inference times.”"
- conradev 3y agoGood catch! That is very cool
- minedwiz 3y agoLooking over another story on this at https://www.zdnet.com/article/microsoft-unveils-first-ai-chip-maia-100-and-cobalt-cpu/ https://www.zdnet.com/article/microsoft-unveils-first-ai-chi..., Cobalt seems to be a general purpose ARM CPU.
- mr_toad 3y agoIt doesn’t get as much attention but it’s the one that the average HN’er is much more likely to end up using.
- monlockandkey 3y agoArm in inevitable for the server. It's interesting how now days, efficiency/power consumption is a consideration over pure raw performance.
- wargames 3y agoI genuinely don't see how x86 architecture will continue to survive the next 10 years. It will of course take longer to change home desktop users to new architectures; they will be the last segment to switch, but it seems all but inevitable. BTW, I'm not even speaking to whether x86 can compete at the same power per watt... I think it just won't make sense financially to be out of sync with the industry.
- helf 3y ago[dead]
- jimmaswell 3y agoI care vastly more about raw performance than energy usage for my home systems. I also have good reasons to care about the best single core performance. I don't see x86 going away that fast.
- magnio 3y agoFWIW, the current king of single core Geekbench is the M3 chips. Even the base M3 scores as high as the i9-14900K and higher than the Ryzen 9 7950X3D, at less than half their TDPs.
- tester756 3y ago>FWIW, the current king of single core Geekbench is the M3 chips Are we reading the same scores? https://browser.geekbench.com/v6/cpu/singlecore https://browser.geekbench.com/v6/cpu/singlecore
- 3y ago
- andy_xor_andrew 3y agoDamn, Microsoft is playing all sides of this, huh. Within a ten minute window: - Satya announced GPT-4 runs (at least partly) on a new AMD offering - Satya announced an in-house chip for ML acceleration - Satya brings NVidia CEO Jensen Huang on stage they've got every horse in the race, huh (disclaimer, I work for MS but all the stuff talked about here far is waaaay above my paygrade haha, and all brand new info to me)
- onlyrealcuzzo 3y agoMicrosoft really likes being valued as a growth stock... Obviously they're going to play every angle.
- kyboren 3y ago> - Satya brings NVidia CEO Jensen Huang on stage This should be ringing alarm bells at FTC and DoJ. You know what's even better than trust busting and breaking up cartels? Preventing the formation of cartels and trusts in the first place.
- scarface_74 3y agoRaising alarm bells because Microsoft is actually dealing with multiple companies? Would it be better for competition if Microsoft only used one supplier?
- cplusplusfellow 3y agoI would certainly want to spread my risk around to several other large companies.
- kccqzy 3y agoEvery reasonable company in that situation needs to play all sides of this. Even the comically bad management at Google understands this: Google has their own LLMs, their own hardware for ML, and they also buy a ton of nvidia chips for cloud customers. It's a no brainer.
- 3y ago
- gary_0 3y ago> Microsoft said it does not plan to sell the chips Add it to the list of things you can't buy at any price, and can only rent. That list is getting pretty long, especially if you count "any electronic device you can't fully control or modify".
- ronsor 3y agoGoogle does the same thing with their TPUs. The masses will be left with the NVidia monopoly, while large companies will be able to free themselves from that.
- wmf 3y agoMI300 is coming.
- machinekob 3y agoThis time AMD for sure will fight with NV (its only failed 20 times already copium)
- wmf 3y agoOn one hand this is a fair prediction but Triton exists now and it didn't exist last time.
- pjmlp 3y agoIf the software still sucks with what CUDA can do, in graphics tooling, polyglot support and optmized libraries, across at very least Windows and GNU/Linux, than whatever the card is capable of doing isn't much relevant.
- FuriouslyAdrift 3y agoDecember 6 launch date: https://ir.amd.com/news-events/press-releases/detail/1168/amd-announces-amd-instinct-mi300-accelerator-launch-event https://ir.amd.com/news-events/press-releases/detail/1168/am...
- aiman3 3y agoNVIDIA should start offer AI cloud, buy DigitalOcean, counterattack microsoft, google, amazon, since they are attacking NVIDIA's territory.
- stanac 3y agoThey already have gaming cloud, wouldn't be unthinkable to offer gpgpu cloud.
- solardev 3y agoIt's a hell of a cloud too. Geforce Now performs several times better than Microsoft's crappy offering, xCloud (or whatever it's called now). Nvidia made some really amazing strides in the past few years, taking over cloud gaming where Onlive and Stadia utterly failed, making DLSS, etc. I just hope they don't abandon us gamers for their AI stuff :( Probably the entire gaming market is way smaller than the potential AI market, just hopefully not too small to matter.
- collegeburner 3y agothey invested heavily in coreweave and gave them H100 allocation
- tmikaeld 3y agoAnother AI chip made by TSMC, they make all of them?
- jon-wood 3y agoTSMC manufacture (more or less) all of them. There's very few companies in the world capable of manufacturing high performance chips.
- dragontamer 3y agoThat's like complaining that all books are made by Penguin Press or something, ignoring the effort individual authors make. Most of the value of chips is in their design, which is owned by different entities. Manufacturing is important too (only TSMC can make these advanced designs at scale and at lower costs than the competition). The question I have is if Cobalt has any innovations in its design, or if its just bog-standard ARM Neoverse cores. Its not too big of a deal to download ARM's latest designs and slap them into... erm... your designs. But hopefully Microsoft added value somewhere along the road (The Uncore remains important: cache sizes, communications, and the like).
- adrian_b 3y agoMicrosoft claims that Cobalt has a much lower power consumption than any other Arm CPUs that they have used. Presumably this means that Cobalt has a much lower power consumption than the current Ampere CPUs used by Azure. Most of the power consumption reduction for a given performance may have come from using a more recent TSMC process, together with a more recent Arm Neoverse core, but perhaps there might be also some other innovation in the MS design.
- solardev 3y agoIt's not a matter of giving due credit, but supply constraints. Books aren't limited by the availability of printing presses, are they? (maybe they are and I just didn't know?) But if TSMC is the only company that can do this, they're a bottleneck for the entire world. Not to mention a strategic and geopolitical risk for the West. It's be nice if some domestic companies invested in fabs again...
- lnsru 3y agoI wish I could work in some team designing these chips. Maia is probably my dream product to work on. Super new, super cool and one of it’s kind.
- geodel 3y agoI mean that looks cool and exciting if it is really small colocated team if one is lead engineer or director of large team of engineers so that they can learn do things that interest them and assign boring/routine work to individuals. Otherwise it would be just another job where people work on assigned JIRA stories and go home in evening.
- buildbot 3y agoWell, azure devops in this case ;)
- aleph_minus_one 3y ago> I wish I could work in some team designing these chips. Maia is probably my dream product to work on. Super new, super cool and one of it’s kind. You likely became bewitched by their glamorous marketing side. I'd bet that the real work that the team does is very similar to the work that basically every ASIC design team does.
- asdfman123 3y agoThink of all the requirements meetings!
- hulitu 3y ago> Maia is probably my dream product to work on. I bet you haven't used any Microsoft product before. /s
- lmm 3y agoA lot of Microsoft hardware is best-in-class, strangely enough.
- neilv 3y agoInitially looks pretty, but will mess you up: https://en.wikipedia.org/wiki/Cobalt#Health_issues https://en.wikipedia.org/wiki/Cobalt#Health_issues
- opcode84 3y agoThe Microsoft chip has MX data types: https://news.ycombinator.com/item?id=37930663 https://news.ycombinator.com/item?id=37930663 https://arxiv.org/abs/2310.10537 https://arxiv.org/abs/2310.10537
- buildbot 3y agoYes! Great catch :)
- tibbydudeza 3y agoDoes it run Linux or the Windows for Azure (definitely not a run of the mill Windows for DataCenter).
- mr_toad 3y agoThe Cobalt chip will run ARM Linux I’m sure. It’s basically a competitor for AWS’s Graviton.
- xnx 3y agoChip manufacturers (including Nvidia) really missed where the market was going if customers like Microsoft, Amazon, etc. feel the need to make their own chips.
- lotsofpulp 3y agoAs far as I understand, Nvidia does not manufacture chips, they only design them and create the software.
- adgjlsfhk1 3y agoI think they got the direction right but the price wrong. They are used to dealing with super-computers as the main server clients who aren't big enough to fight back if the prices creep to high.
- mr_toad 3y agoI think in Intel’s case it was both the cost and the power consumption/heat that led to cloud vendors and Apple looking for alternatives.
- airstrike 3y agoMicrosoft, Amazon, etc. feel the need to make their own chips so that they don't let NVIDIA take all the profits, not because they think NVIDIA is incompetent
- Affric 3y agoBingo. There are real profits in the chip space and considering that there are 3 fabs and one clear leader who will make anything for anyone this is a sign that NVIDIA are doing a great job.
- vb-8448 3y agoMy guess is that it's more about the wish of cloud vendors to control everything from the hw to sw: it's called vertical integration, and it's common in a lot of businesses. It makes a lot of sense from the point of view of cloud giants.
- MR4D 3y agoEvery day we get closer and closer to the hypothetical "five computers". [0] Software was first, and now every one of them have hardware too. Amazon Apple Goole Microsoft [0] - https://engines.egr.uh.edu/episode/1059 https://engines.egr.uh.edu/episode/1059
- deleted 3y ago[deleted]
- asdfman123 3y agoFive mainframes, maybe. I've got at least five terminals for those mainframes attached to, or inside of, my TV alone.
- pjmlp 3y agoThe irony being, they might even all be running some flavour of an UNIX clone, and it won't matter to the general public.
- deleted 3y ago[deleted]
- kmod 3y agoI thought an interesting point was the liquid cooling -- unclear how important this is to them, but I'm guessing it means that they designed it with a TDP that requires liquid cooling. This (wanting higher density) is the opposite of the trade-off that I was expecting. In my (limited and out of date) experience, power was the limiting factor before space, and I believe AI racks have very high power draws already. I would have guessed this would be because larger nodes would be better for AIs tight communication patterns, but they specifically call out datacenter space as the constraint. Curious if anyone knows more about this
- sanxiyn 3y agoMy understanding is that if you are renting your data center space right now, power still is the limiting factor and you will need to leave some rack space empty to be able to run GPUs. On the other hand, if you are building your own data center, which is the case for Microsoft, presumably you can arrange high power zone to run GPUs.
- asdfman123 3y agoWhat would be even more impressive would be making the chip in the US so we weren't so completely reliant on TMSC.
- sanxiyn 3y agoGroq is planning to do that. https://twitter.com/GroqInc/status/1691436573014020096 https://twitter.com/GroqInc/status/1691436573014020096
- ThinkBeat 3y agoI am a little confused on all this. I thought conventional wisdom was that creating "super good/ top performant " computer processors was hard, and unless you could produce them at scale prohibitable expensive. Is this done as a bridge until/if Nvidia is able to deliver their processors fast enough? I would think that competitors to Nvidea would have serious competitors on the market already if competing can be done by Microsoft for whom producing hardware is not their main business focus.
- p1esk 3y agoIt remains to be seen how well these chips will compete with Nvidia.
- jacquesm 3y agoNote the huge difference in the kind of hardware required to do training and to do interference.
- s0rce 3y agoPresumably Microsoft won't be making the chips, I guess they will also be made by TSMC.
- thetrb 3y agoWhy would Microsoft not produce them at scale? With Azure and internal use cases I'm sure they're at a good scale.
- AdamN 3y agoSecond largest cloud on the planet isn't enough scale?
- spandextwins 3y ago[dead]
- karmasimida 3y agoAI, or creation of AI, will very soon leave the hands of common folks at this rate. The cost will soon unattainable.
- er4hn 3y agoThis is likely to be cyclical though. Fast cars[1], large amounts of processing power[2], access to cryptographic algorithms, are all things that started out expensive to be on the cutting edge, some still are expensive on the cutting edge, but then became more affordable for the consumer over time. AI has already had explorations on training models with limited resources. It's feasible, just has tradeoffs that will hopefully get better over time. [1] A 1970 Corvette 427 has a 0 - 60 mph of 5.3 seconds (src: https://www.caranddriver.com/features/g15379023/the-chevrolet-corvette-a-history-in-zero-to-60-times/?slide=13 https://www.caranddriver.com/features/g15379023/the-chevrole...) and cost around $44k inflation adjusted dollars. You can buy a 2008 Nissasn 350Z Enthusiast that will do it in 5.2 (src: https://www.zeroto60times.com/vehicle-make/nissan-0-60-mph-times/fast-slow-0-60/ https://www.zeroto60times.com/vehicle-make/nissan-0-60-mph-t...?) for around $13k today. [2] I'm too lazy to calculate relative cost / cycle in old warehouse computers vs phones but it's gotten _better_.
- vGPU 3y agoThe article is actually incorrect. There is no 1970 427. There was a 350 and a 454. Considering the amount of torque it put down, it would likely be a good bit faster with modern tires. You do have to also realize that the 1970 vette was right when emissions controls started destroying performance - the 69 427 put down a good bit more power than the 70 454.
- mackid 3y ago“People who are really serious about software should make their own hardware.” - Alan Kay
- hulitu 3y ago[flagged]
- jasonwatkinspdx 3y agoWow, you sure owned them ... Could we just not make these kind of comments? They aren't useful.
- ss1996 3y agoI thought it was funny. Is this is no fun zone?
- Aromasin 3y agoNote that Microsoft has been using "custom chips" for years; they've just been on FPGAs not ASICs. They've developed IP to accelerate a whole bunch of processes, so it's not like they've suddenly come around to a magic sauce that Amazon and Google have been doing this whole time. It would surprise me if over half of the new chip is just based on RTL from their FPGA design. The only thing that's changed is that they're scaling like crazy now and can justify overhead that comes with designing ASICs versus using off the shelf parts.
- nl 3y agoMS has offered FPGAs for years. AFAIK they aren't generally using them themselves, although I'd be interested to read otherwise.
- gavin_gee 3y agoCapitalism comes to the rescue. With h100's at 40K its obvious that other players want to drive that price down.
- xutao323 3y agoCobalt is based on Arm CSS model, so it's Neoverse series probably N2 (based on 128 core count). AWS Graviton 2/3 is N1/V1, and Nvidia Grace is V2. Ampere Altra is N1 and the new Ampere One is publicly unknown. Several Chinese companies also develop chips based on N2, like Alibaba T-head and ZTE Sanechips. I worked on software tunings for both of them. It's good to see more and more Arm chips.
- tenebrisalietum 3y ago> The Azure Cobalt CPU, named after the blue pigment, is a 128-core chip that’s built on an Arm Neoverse CSS design and customized for Microsoft. > Microsoft is currently testing its Cobalt CPU on workloads like Microsoft Teams and SQL server, Teams is so bloated it needs its own 128-core CPU to run well. /s
- jbverschoor 3y agoI really expected Apple to provide arm cloud and eat a piece of the cloud pie, as the multi billion contracts with Amazon are ending soon