14 ms·
Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
- lvl102 3y agoSeems Nvidia is going for maximum margin as they see competition ahead.
- deleted 3y ago[deleted]
- theGnuMe 3y agoAnd they can build a big moat with cuda.
- paulpauper 3y agoStock unchanged in afterhours. A lot of people were hoping for a big pop on some big development.
- synergy20 3y agonot only that, it lost steam during the day, maybe it was overheated too much and no more news can pump it up any further.
- deleted 3y ago[deleted]
- dagmx 3y agoI imagine it’ll pop in the morning
- dxbydt 3y agoguy is messing it up bigtime and in real-time as well. sheesh. none of his jokes are landing. “we had 2 customers. we have more now”. long pause. screen behind him covered with logos of all his customers. pause. pause. finally applause. ok on to the next tidbit. whole conference has been proceeding like this now. look if you invite cramer and the wall street crowd, you should throw in some dollar figures. like - who is paying for all this. how much. and why. talk is entirely about token generation bandwidth, exaflops and petaflops, data parallel vs tensor parallel vs pipeline parallel - do you honestly think cramer knows the difference between an ml pipeline and an oil pipeline ? i am watching this conf with my kid - proper GenZ member - who got up after 5 mins and said man who is this comedian, his jokes are so bad, and left :(
- gmerc 3y agoNah, wallstreet doesn’t understand what it’s looking at. That’s fine, it’s a developer conference for a founder lead company that hasn’t reached the “stock price is the product” state. He’s not trying to optimize the next 5 days of stock. There’s a full ecosystem grab with Nim there, a new GPU that forces every major datacenter to adopt (or their competitors will massively increase their compute density)
- anon291 3y agoThis is a developer's conference, not a financial one.
- Takennickname 3y agoCramer is an entertainer. Not a developer or an investor.
- smallmancontrov 3y agoYou might not like it, but this is what peak performance looks like.
- Deasiomlo 3y agoWhy would a Gen z kid care about this conference? It's not a hipster event. And the pauses are clearly his style of presenting. Having a artificial pause to tell the audience it would be a good time to react. You clearly don't like it, I don't mind it. But just to be clear: we see peak human ingenuity. This right now will be history on how we as humans build AGI, full human robots etc (in case we don't nuke ourselves). You can read the conference results easily at any it news portal. There is no requirement from Nvidia entertaining you or your kid.
- TheAlchemist 3y agoWell, stock price is not a good short term indicator about Nvidia developments, nor any company for that matter. Nvidia is doing a very good job. That being said, their stock is absolutely and hilariously overvalued.
- costcofries 3y agoTell me more about why you believe their stock is hilariously overvalued.
- Takennickname 3y agoBecause he missed the train. My guess.
- Workaccount2 3y agoThey are priced as if they are the only ones who are capable of creating chips that can crunch LLM algos. But AMD, Google, Intel, and even Apple are also capable. Apple is in talks with Google to bring Gemini to the iPhone, and it will obviously also be on android phones. So almost every phone on earth is poised to be using Gemini in the near future, and Gemini runs entirely on Google's own custom hardware (which is at parity or better than nVidia's offerings anyway).
- jerf 3y agoThis seems as good a place as any to be Corrected by the Internet, so... correct me if I'm wrong. Making a graphics chip that is as good as Nvidia: Very difficult. Huge moat, huge effort, lots of barriers, lots of APIs, lot of experience, lots of decades of experience to overcome. Making something that can run a NN: Much, much easier. I'd guess, start-up level feasible. The math is much simpler. There's a lot of it, but my biggest concern would be less about pulling it off and more around whether my custom hardware is still the correct custom hardware by the time it is released. You'd think you could even eke out a bit of a performance advantage in not having all the other graphics stuff around. LLMs in their current state are characterized by vast swathes of input data and unbelievably repetitive number crunching, not complicated silicon architectures and decades-refined algorithms. (I mean, the algorithms are decades refined, but they're still simple as programs go.) I understand nVidia's graphics moat. I do not understand the moat implied by their stock valuation, that as you say, they are the only people who will ever be able to build AI hardware. That doesn't seem remotely true. So... correct me Internet. Explain why nVidia has persistent advantages in the specific field of neural nets that can not be overcome. I'm seriously listening, because I'm curious; this is a deliberate Cunningham's Law invocation, not me speaking from authority.
- rvz 3y ago> A lot of people were hoping for a big pop on some big development They are waiting for earnings projections for such a pop since right now it is extremely overbought and struggling to move past >$1,000 per share. For now, Microsoft and OpenAI will use these chips, but in the long term they are just looking at this and plotting to build their own chips and reducing their dependence on Nvidia and will be ready to switch once their contracts have run out.
- swalsh 3y agoAt 2 trillion, it's all baked in already
- deleted 3y ago[deleted]
- _factor 3y agoNvidia is no secret. Whatever hidden value is in the stock, is likely already represented.
- dagmx 3y agoFP8 being 2.5x Hopper is kind of disappointing after such a long time. Since its 2 fused chips, that means it’s 25% effective delta. though it seems most of the progress has been on memory throughput and power use which is still very impressive. I wonder how this will trickle down to the consumer segment.
- azeirah 3y agoJensen revealed later that the LLM inference is 30x due to architectural improvements, it's massive. I don't know if it's latency or just 2-3x performance boost with 30x more customers served in the same chip. Either way, 30x is massive.
- ephemeral-life 3y ago30x is the type of number that when you see it in a generational improvement, you should ignore it as marketing fluff.
- azeirah 3y agoFrom how I understood it, it means they optimised the entire stack from CUDA to the networking interconnects specifically for data centers, meaning you get 30x more inference per dollar for a datacenter. This is probably not fluff, but it's only relevant for a very very specific use-case, ie enterprises with the money to buy a stack to serve thousands of users with LLMs. It doesn't matter for anyone who's not microsoft, aws or openai or similar.
- acchow 3y agoThey showed 30x was for FP4. Who is using FP4 in practice?
- KaoruAoiShiho 3y agoBut maybe you should. Once the software stack is ready for it there'll be more people since the performance gains are so massive.
- jairuhme 3y agoI haven't listened to Jensen speak before, but am I the only one who thought the presentation wasn't very polished? Not a knock on anything he has accomplished, just an observation that sorta surprised me
- deleted 3y ago[deleted]
- acchow 3y agoHe has more important things to do than perfecting a presentation. He likes his employees to message him freely with things they think he can help with.
- PartiallyTyped 3y agoI read about this around and tbh I respect it a lot. Somehow Jensen has managed to scale the company and still keep it focused on the product with high eng practices.
- jefozabuss 3y agoWhen he had that slide up with generating everything I was kind of expecting that he'd say this whole keynote is generated including him. That'd have been crazy.
- azeirah 3y agoHe said he didn't rehearse well. I think it makes him come across very genuinely, not some dumb hyperpolished corporate blabla
- CaptainFever 3y agoIn comparison to Apple keynotes, precise, polished, practiced and pre-recorded.
- ipsum2 3y agoYeah he said he didn't rehearse and it really shows.
- bluedino 3y agoThey acquired Bright Cluster Manager a few years ago, who would be next on their list to acquire? It seems like they want to provide customers with the whole stack.
- shiftpgdn 3y agoCanonical is a ripe target. Canonical has been trying to grow Ubuntu and other tools in the enterprise world for the last few years without significant success, and much of the Nvidia devkit stuff is built around Ubuntu.
- echelon 3y agoPlease do not give them this idea. Ubuntu is actually a pretty great daily driver desktop Linux, and I'd hate for that to lose priority and disappear. I'm not a fan of what happened to the Red Hat ecosystem for exactly the same reasons.
- xmprt 3y agoAs someone who has used Ubuntu in the past and has now moved onto greener pastures, I appreciate everything Canonical and Ubuntu have done for the Linux community but there are many better options today and Canonical is already far from the company it once used to be.
- bhickey 3y ago[flagged]
- deleted 3y ago[deleted]
- stevethomas 3y agoTime to sell. When they start becoming a platform, it means they have nothing more concrete in the near future. Sell now and buy again later once the price corrects.
- golergka 3y agoDoes a company like Nvidia has to have anything more concrete than newer, bigger and faster chips?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- belter 3y agoDon't bet against a CEO who knows what is talking about, has 80% market share and an arm tattoo of his own company logo.... :-) So far the short sellers, learned that bitter lesson.
- deleted 3y ago[deleted]
- pvg 3y agoIf you held and then sold Nvidia stock when they announced CUDA or GeForce Live, you'd now be now a big pile of negative money richer.
- Keyframe 3y agoThey still have to announce you can send an email through their platform.
- mark336 3y agoI wouldn't be 100% against them, but it is disappointing that they don't have any better ideas.
- hsuz 3y ago
- hakdbha 3y ago[dead]
- qwertox 3y agoWhat is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698 https://youtu.be/Y2F8yisiS6E?t=4698
- sipjca 3y agoyes
- deleted 3y ago[deleted]
- fancyfredbot 3y agoThat's right.There was mention of a precision aware transformer engine which might make it easier to use fp4, but it's not 30x faster in a like for like way. This shouldn't be surprising since it's more or less two hoppers next to one another on a slightly improved process node. 2.5x seems more likely in cases where you don't exploit a new feature like that or the increased memory.
- Havoc 3y ago>bit misleading. Only partially, because in LLMs FP4 isn't half as useful as FP8. So if you have gear that crushes at FP4 then that's what you use and you benefit from that increased speed (at minimal accuracy loss). Definitely some marketing creativity in there, but its not entirely wrong as a measure of real world usage
- tamimio 3y agoI think at this point, they should stop making it video “cards” but rather video “stations”, a full tower station with power supply and one giant “card” inside with proper cooling, etc., might also justify the crazy prices anyway.
- deleted 3y ago[deleted]
- ufocia 3y agoProbably better to stick to the GPUs. Integration is a low margin game.
- caycep 3y agogranted, at this point we plug the computer into the GPU, so it might not make a difference...
- georgyo 3y agoI'd prefer they stick to GPUs, but I think you're over simplifying. Dell proves that selling complete units is very profitable. Apple shows that owning the entire stack is immensely profitable. Nvidia already has significant hardware and software investment. They very well could fully integrate and grab larger slices of the pie. In fact, Nvidia already has complete appliance like fully integrated machines. But enterprises like to install their own OS and run their own software stack. These appliances have not caught on, at least not yet.
- zer00eyz 3y ago>> Apple shows that owning the entire stack is immensely profitable. Apple shows no such thing. Apple, sells pretty, reliable and safe. A car is a car, but apple is a sports car, or a saloon. Vertical integration is the way they chose to deliver that, and pretty and reliable are all normal people care about. Nvidia is gonna have to think long and hard about the "whole stack". 20 years ago they might have been able to pull a next, but right now anything that isnt LINUX is a rounding error, and they dont want to turn into SUN (no one at nividia is smart enough to make them the next sun). Nvidia architecture + Nvidia os is not something that I see them pulling off for the datacenter.
- herecomethefuzz 3y ago"Platform company" means multi-chip in this case? Seems logical since it's becoming impractical to cram so many transistors on a single die.
- 1oooqooq 3y agono it means rent seeking. imagine aws if they also sold all computers in the world, now you can only rent from them
- throwaway11460 3y agoSo like IBM at the beginning of computers
- maximus-decimus 3y ago"For only 100$ a month, you'll be able to turn on the gpu you already paid for" --Nvidia, pretty soon
- bpye 3y agoThis is sort of already a reality. Their vGPU functionality (partitioning a single physical GPU into multiple virtual GPUs) is already separately licensed - https://www.nvidia.com/en-us/data-center/buy-grid/ https://www.nvidia.com/en-us/data-center/buy-grid/ And that's once you've bought an expensive Tesla/Quadro GPU too.
- 1oooqooq 3y agogood. that only impact disgusting cloud providers who are hoarding all hardware everywhere. no sympathy for them.
- 0xcde4c3db 3y agoI don't really understand the bird's-eye view of the product line, but judging by some of the raw physical numbers and configurations Jensen was bragging about, it means that they want to basically play the mainframe game of locking high-end applications into proprietary middleware running on proprietary chassis with proprietary cluster interconnect (hello, Mellanox acquisiton).
- amelius 3y ago"platform co." Platform company, as in they're allowing developers on their AI platform and opening an app store?
- fnordpiglet 3y agoThe headline is editorialized by the submitter, the actual headline is “Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’” which is arguably worse. The article doesn’t discuss becoming a platform co but instead discussed ways their existing platform subscription model is evolving to add backwards compatibility testing.
- deleted 3y ago[deleted]
- tzm 3y agoPlatform co seems fitting, considering Nvidia's data center revenue in the fourth quarter of 2023 was a record $18.4 billion, which is 27% higher than the previous quarter and 409% higher than the previous year. Seems revenue from inference is growing at a significant clip.
- deleted 3y ago[deleted]
- ec109685 3y agoData center revenue includes sales to companies like Meta that run their chips in their own data centers.
- Deasiomlo 3y agoDouble digit peta flop mass produced. "The computing power needed to replicate the human brain’s relevant activities has been estimated by various authors, with answers ranging from 10^12 to 10^28 FLOPS." Petaflop is 10^15 Crazy times.
- teaearlgraycold 3y agoI’ll be happy with this if we use it to design viable fusion power plants. And I’ll be severely disappointed if it’s mostly used for ad targeting.
- steelframe 3y ago> I’ll be severely disappointed if it’s mostly used for ad targeting Obligatory Rick and Morty: https://www.youtube.com/watch?v=xerLPWdyX-M https://www.youtube.com/watch?v=xerLPWdyX-M
- sbstp 3y agoYou are about to be severely disappointed.
- __loam 3y agoWhy? More compute helps with fluid dynamics, particle physics, plasma control, and material science. Can't imagine it hurts.
- belter 3y agoFusion plants will be used to power ad targeting Nanoprobes.
- geor9e 3y agoLike a lot of the commenters here, I have a problem with this headline. They don't "seek to become a platform company", as in making their own cloud platform where they rent GPU time, meaning they stop selling GPUs to other cloud platforms. That easy misinterpretation makes good clickbait, but no, that's not what the article says - the article has Huang bragging that CUDA already is a parallel computing platform, for a decade or more, and Blackwell Architecture is so integrated and customizable with CUDA (with all its user-extendable kernels and community) that it's thought of as a platform rather than just chip architecture.
- deleted 3y ago[deleted]
- baobabKoodaa 3y agoSomeone please "chip in" and confirm or deny if this interpretation is correct?
- wmf 3y ago"Platform" can have several different meanings and some people in this thread are picking the most evil meaning as an excuse to shit on Nvidia. It's true that Nvidia is providing the GPU, the CPU, the server, the rack, the network, the drivers, and the orchestration software to run AI training/inference. (If you want that stuff. You could just buy GPUs if you want.) It's fair to call that a platform. Nvidia is not becoming a cloud provider (beyond a small eval environment perhaps).
- basiccalendar74 3y agothey announced a cloud service NIM that hosts LLMs using Nvidia software stack. https://developer.nvidia.com/blog/nvidia-nim-offers-optimized-inference-microservices-for-deploying-ai-models-at-scale/ https://developer.nvidia.com/blog/nvidia-nim-offers-optimize... so there is a push towards a platform company, but probably not well explained in this article.
- jakobov 3y agoThey are claiming a 25x reduction in power consumption. That can't be right. Anyone understand where this number is coming from?
- LTL_FTC 3y agoDid you read that in the linked article? I couldn’t find it. But maybe due to the better efficiency with regard to the performance boost (5x) and the ability to now use 27 trillion parameters versus 1.7 Trillion, one can presumably finish the same amount of work in 1/25th of the time and bam, reduction in power consumption. As you say, I’m skeptical the max power draw itself is 25x lower.
- cavisne 3y agoComes from here [1]. Basically 100 racks of H vs 8 racks of B. I think there may be a typo though, I assume this also includes liquid-cooled vs air-cooled. [1] https://nvdam.widen.net/s/xqt56dflgh/nvidia-blackwell-architecture-technical-brief https://nvdam.widen.net/s/xqt56dflgh/nvidia-blackwell-archit...
- adamnemecek 3y agoWhich Blackwell is it named after?
- wmf 3y agohttps://en.wikipedia.org/wiki/David_Blackwell https://en.wikipedia.org/wiki/David_Blackwell
- cebert 3y ago> “Nvidia … is becoming less of a mercenary chip provider and more of a platform provider, like Microsoft or Apple, on which other companies can build software. I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see some acquisitions or competing custom solutions in the future. Fortunately for Nvidia, a lot of AI is still dependent on CUDA. I’m interested to see how this plays out.
- dheera 3y agoMy prediction is eventually there will be anti-trust ligitation, they will be required to open the CUDA standard, after which AMD will become a competitor. NVIDIA could voluntarily open the standard to avoid this ligitation if they wanted to, though, and IMO it would be the smart thing to do, but almost every corporation in history has chosen the ligitation instead.
- wmf 3y agoIt would be kind of genius for Nvidia to "open" the CUDA APIs (which have already been unofficially reverse engineered anyway) but not the code. Maybe they'd also officially support HIP and SYCL. Maybe they could open SXM after all competitors have already committed to OAM. They'd create the appearance of opening up while giving up very little.
- sitkack 3y agoBy "Opening Up" they cement their leadership position. AI frameworks are already targeting CL, SPIR-V, etc. The low level details will fade and so will Nvidias api dominance. The MI300 smokes the H100 yet here we are.
- incrudible 3y agoJust because they are a target doesn’t mean things just work. Historically, AMD hardware for GPGPU becomes obsolete well before the software landscape catches up. I am not going to risk my time and money finding out whether history repeats itself, just for a few potential FLOPS per dollar.
- wmf 3y agoMore technical details: https://news.ycombinator.com/item?id=39752500 https://news.ycombinator.com/item?id=39752500
- __mharrison__ 3y agoMy take from being at the keynote and the content I've seen so far at the conference is that Nvidia's is moving up the stack (like all good hardware vendors are prone to do). Obviously they are going to keep doing bigger. But the takeaway for me is that they are building "docker for llms" - NIM. They are building a container system where you can download/buy(?) NIMs and easily deploy them on their hardware. Going to be fun to watch what this does to all the AI startups...
- deleted 3y ago[deleted]
- djtango 3y agoI'm not that abreast of all the developments in the AI space. What specific class of AI startups do you have in mind here? AI-aaS startups who provide the "infra"?
- fisf 3y agoAI startups who just wrap a (standard) API or model in a thin UI layer. The backend part will be a commodity and the UI layer offers no value proposition.
- flessner 3y agoWon't do anything to most consumer facing AI, the UI & convenience is already a major selling point. A bigger threat is that the feature the business is built around makes it into mainline software... there is no demand for (paid) background removal anymore as every iPhone can do it nowadays. Generally if whatever AI product you have can easily just be a feature in whatever application businesses already use, then you are running a business on borrowed time.
- Kinrany 3y agoOh no, turns out the thing people really want from AIs is generality because everything else can be done in stupid software!
- 3y ago
- gammalost 3y agoI wonder when we as an industry will start to address the scaling issues in LLMs.It is obviously in Nvidias interest to keep pushing out bigger and better GPUs, but what is the collective interest? It is already proven that good language models are possible given enough resources. The challenge now is to put these models in a solution which do not require unfathomable amounts of resources for the average use cases.
- bayindirh 3y agoWasteful software development is easy and keeps momentum for development. As long as growth is king, quick and dirty will always beat well optimized and smaller systems. This is not a problem with AI only, but with every software we use. Only two groups try to optimize things and try to fit into smaller systems. Passionate programmers and people who is paid to do this (e.g.: phone manufacturers' software teams, etc.).
- Difwif 3y agoNot sure it's fair to characterize modern LLMs as 'wasteful software development" or unoptimized. The implementations do quite an impressive level of optimization with what hardware is available. New theoretical methods w.r.t quantization represent most of our software optimization techniques and we're probably hitting the limit of that shortly with ternary or binary gates. To your point enthusiasts and developers with strong financial motivation have fairly optimized code. AI definitely falls into the later group. These are not like your typical web app :)
- bayindirh 3y agoCurrently AI folks create bigger models and feed larger training data sets, because we still don't know the efficiency limits. IOW, currently we can't accelerate the learning beyond a certain point with less training data. This is esp. true on LLM/GenAI space. On the other areas where NNs are used, what I can see is training models with less data is not only plausible, but very possible, esp. in image processing, probably an image carries more information than a single sentence. I think AI falls on the spectrum with a slight bias to the latter group. Because if you can shrink a model 10% and lose a month, you'd rather have a 10% bigger model now, and reap the money^H^H^H^H^H fame. I don't know anything about a typical web app, because I'm not a typical developer developing web apps. :)
- deleted 3y ago[deleted]
- bilsbie 3y agoDoes this support that 1-2 bit stuff we were hearing about a few weeks ago?
- nojvek 3y agoThe issue with bigger LLM or even MMMs is that the bigger they are the more they are cramming and regurgitating the training data, and that opens up to lawsuits. Making NNs generalize the way humans do it is still a hard problem.
- lewhoo 3y agoIs this indeed established ? Could you provide a link or three ?
- nojvek 3y agohttps://avidml.org/blog/llm-privacy-plaigarism/ https://avidml.org/blog/llm-privacy-plaigarism/ https://moreisdifferent.blog/p/weird-properties-of-large-language https://moreisdifferent.blog/p/weird-properties-of-large-lan... https://www.sarus.tech/post/llms-have-become-excellent-at-memorizing-its-a-problem-for-privacy https://www.sarus.tech/post/llms-have-become-excellent-at-me...
- lewhoo 3y agoThank you. That's certainly interesting.
- nojvek 3y agoPoor Mayoshi Sun from Softbank. He held a $3.1B position in Nvidia in 2019. That would be 19x at $60B today. He was super bullish on AI and robotics. Just ahead of his time.
- flakiness 3y agoNit: Masayoshi Son
- guy-om 3y agoNo, we need less AI, less robots, more (alive) humans.
- karakot 3y agowe've got quite a lot of humans already.