17 ms·

GPUs Go Brrr

1104 points by nmstoker 2y ago

latchkey 2y ago
Really impressed by the writing style of this post and very much looking forward to this on AMD MI300x. Let me know if you want some time on mine.
- jsemrau 2y ago
  Really? It gives me PTSD from the Wallstreetbets days.
  - forrestthewoods 2y ago
    I also enjoyed the article's style. I utterly despise "academic paper speak". It is, imho, not the most effective style to communicate complex ideas. I find it so much easier to learn from a more casual "blog post" or in-person presentation over stiff, rigid academic speak.
    - kaycey2022 2y ago
      I find both to be useful in different stages. The casual style is very helpful when starting out. But once I have put in a few weeks or months of study in, then the rigor and preciseness of academic style is good as well. I agree with you in the sense that something has "died" in writings the follow academic paper speak these days. Just yesterday I saw an ancient article surfaced by Scientific American and Peter Norvig on System Analysis by Strachey. It uses quite a bit of formal language but is super approachable at the same time. That kind of skill is rarely seen these days.
  - david927 2y ago
    > the Wallstreetbets days. https://twitter.com/TheRoaringKitty/status/1790041813379850491 https://twitter.com/TheRoaringKitty/status/17900418133798504...
- globular-toast 2y ago
  Good writing is clear and unambiguous. With speech there is an opportunity to interrupt and ask for clarification. Writing has one chance to get the message across. A reader shouldn't have to consult knowyourmeme.com to figure out what the heck the authors are trying to say. I don't even know what the title means here. That's how far they've missed the mark.
  - _obviously 2y ago
    Wow that really sucks for you. I just read it in 5 minutes and feel much more informed about the subject pf nvidia memory twizzlization. It's kind of funny to me that presumably young college guys are writing in a style that's very readable for my old ass.
    - unethical_ban 2y ago
      >that really sucks for you How can I put this in your vernacular... "Most polite genZ meme enjoyer"
  - aetimmes 2y ago
    Even if you're not familiar with the "go brrr" meme (which is the only use of meme-idiom in the article and is used exactly twice), its meaning is easily inferred via context clues from the opening paragraphs. Good writing is also entertaining and engaging.
    - globular-toast 2y ago
      Keyword being also.
    - throwaway1492 2y ago
      As someone who witnessed A-10 CAS fuck some stuff up in a combat zone ie the real “brrrrt” I’ve been mystified by the meme and current useage. No one knows where it comes from nor the slaughter it represents.
      - 2y ago
- tracker1 2y ago
  Have you done much AI work against AMD products? I'm not going to plunk down $2500+ for an RTX 4090, but have been considering an RX 7900XTX for playing around with, or at least getting started. Just curious how well it will or won't work in practice, or if saving a bit more and getting a 7900 XT over the XTX might be a better option, and how much less vram might impact usefulness in practice.
  - latchkey 2y ago
    My only work with consumer AMD GPUs was mining ethereum, I had 150,000 of them. If you want to use enterprise AMD gpus, I'm renting them. That said, I haven't even had a chance to run/play with them myself yet, they have been rented since I got them last month. Yes, we are getting more.
  - PeterisP 2y ago
    Caveat emptor and your mileage may vary; but unlike nVidia where you could just assume that everything is compatible with everything, for AMD I'd strongly recommend that you try before you buy - consider renting a cloud machine with that GPU to check if the software works for your needs before committing to a large purchase.
    - latchkey 2y ago
      Agreed! The problem is that you cannot rent a MI300x or other high end AMD. They all go into HPC. A problem that I love to work on.
panki27 2y ago
Warp scheduler, 4 quadrants, tensor memory accelerator, unswizzled wgmma layouts... The line between GPU lingo and Star Trek technobabble fades away further and further.
- Agentlien 2y ago
  Your comment prompted me to take a step back and look at these terms with new eyes. That made me smile, because you're so right.
  - deleted 2y ago
    [deleted]
- araes 2y ago
  There was some awareness reading the article, yet "we're warping through the quadrant in our tensor accelerator" is pretty Trek. Have had that thought occasionally with some of the other articles. What it must read like to somebody who gets a ref link for an article over here. Wandered into some Trek nerd convention discussing warp cores.
  - winwang 2y ago
    I mean, if we're talking about "accelerating by modifying the metric tensor" then yeah, that would be pretty sci-fi :) https://en.wikipedia.org/wiki/Metric_tensor_(general_relativity) https://en.wikipedia.org/wiki/Metric_tensor_(general_relativ...
apsec112 2y ago
Interesting! Would this support fp8? Does anyone know how it would compare to Triton?
renonce 2y ago
> NVIDIA’s lies. This is an extraordinarily misleading representation of the actual 128b swizzled wgmma layout. This diagram cost us three weeks of life that we will not get back, hence the public shaming. Wondering if anyone would be surprised that a huge amount of progress in AI is on the engineering side (optimizing matmuls), and that a huge portion of the engineering is about reverse engineering NVIDIA chips
- DeathArrow 2y ago
  Architecture doesn't make a difference. Big enough models trained with big enough data tend to give the same results regardless of architecture. So yes, most advances in AI are mostly due to the fact we can now multiply matrices very fast.
  - elcomet 2y ago
    That's not completely true. The architecture must behave well for scaling, which is not trivial. Basic multi-layer perceptrons do not scale well for example, the gradient will vanish or explode deeper in the network.
    - 3abiton 2y ago
      And data quality. Ensuring the sourcing and quality is very important to get a good model.
      - fleischhauf 2y ago
        this, if you have money to spend in improving your model, more training data is the first thing I'd take a look at
    - Tarrosion 2y ago
      How do modern foundation models avoid multi-layer perceptron scaling issues? Don't they have big feed-forward components in addition to the transformers?
brcmthrowaway 2y ago
NVIDIA needs to be broken up
- huhlig 2y ago
  Into what? Where would you draw such lines?
  - robocat 2y ago
    Into tiles ;-p GPU compute is already broken up - there is a supply chain of other cooperating players that work together to deliver GPU compute to end users: TSMC, SK hynix, Synopsys, cloud providers (Azure/Amazon etcetera), model providers (OpenAI/Anthropic etcetera). Why single out NVidia in the chain? Plus the different critical parts of the chain are in different jurisdictions. Split up NVidia and somebody else will take over that spot in the ecosystem. This interview with Synopsys is rather enlightening: https://www.acquired.fm/episodes/the-software-behind-silicon-with-synopsys-founder-aart-de-geus-and-ceo-sassine-ghazi https://www.acquired.fm/episodes/the-software-behind-silicon... How does the profit currently get split between the different links? Profit is the forcing variable for market cap and profit is the indicator of advantage. Break up NVidia and where does the profit move?
- latchkey 2y ago
  The better alternative is to root for AMD and others to develop their own products so that regardless of breaking NV up or not, there are alternative solutions for people to use. They all leapfrog each other with new releases now any way. Why put all your eggs into one basket.
  - simondotau 2y ago
    George Hotz went down the AMD rabbit hole for a while and concluded that the driver software — more precisely the firmware which runs on the cards themselves — is so badly written that there's no hope of them becoming serious contenders in AI without some major changes in AMD's priorities.
benreesman 2y ago
[flagged]
- spullara 2y ago
  please publish it!
  - deleted 2y ago
    [deleted]
  - benreesman 2y ago
    I’m working on building it, it’s time and resource-intensive. If anyone wants to join the effort, we’re hiring. I said that it was trivial to see that better is possible. I never said that doing better is trivial: actually doing better is hard work, takes time, and costs money. But it can be done and we will.
diginova 2y ago
What should I do if I want to understand such articles in complete? where to start on the roadmap?
- kolinko 2y ago
  This is a good course on gpu programming. Around 4.0 lesson you’ll get the required basics: https://youtube.com/playlist?list=PLzn6LN6WhlN06hIOA_ge6SrgdeSiuf9Tb&si=CE4C7wrkz55YCUqG https://youtube.com/playlist?list=PLzn6LN6WhlN06hIOA_ge6Srgd... Also, write your own cuda kernel to do vector-matrix multiplication (if you use pycuda, you can focus on the kernel, and write everything else with python). Just tell chatgpt that you want to write your own implementation that multiplies a 4000-element vector by 4000x12000 matrix, and to guide you through the whole process. For renting gpus, runpods is great - right now they have everything from lower tier gpus to h100s. You can start with a lesser gpu at the beginning.
- abstractcontrol 2y ago
  For a deep dive, maybe take a look at the Spiral matrix multiplication playlist: https://www.youtube.com/playlist?list=PL04PGV4cTuIWT_NXvvZsnlqYEUy6OmIc3 https://www.youtube.com/playlist?list=PL04PGV4cTuIWT_NXvvZsn... I spent 2 months implementing a matmult kernel in Spiral and optimizing it.
  - justplay 2y ago
    sorry for noob question, how gpu programming is helpful ?
    - abstractcontrol 2y ago
      NNs for example are (mostly) a sequence of matrix multiplication operations, and GPUs are very good at those. Much better than CPUs. AI is hot at the moment, and Nvidia is producing the kind of hardware that can run large models efficiently which is why it's a 2 trillion-dollar company right now. However, in the Spiral series, I aim to go beyond just making an ML library for running NN models and break new ground. Newer GPUs actually support dynamic memory allocation, recursion, and the GPU threads have their own stacks, so you could in fact treat them as sequential devices and write games and simulators directly on them. I think once I finish the NL Holdem game, I'll be able to get over 100x fold improvements by running the whole program on the GPU versus the old approach of writing the sequential part on a CPU and only using the GPU to accelerate a NN model powering the computer agents. I am not sure if this is a good answer, but this is how GPU programming would be helpful to me. It all comes down to performance. The problem with programming them is that the program you are trying to speed up needs to be specially structured, so it utilizes the full capacity of the device.
DeathArrow 2y ago
So do their kernels and library also speed up RTX 4090?
cl3misch 2y ago
> The unswizzled shared memory layouts suffer from very poor coalescing If I didn't know any better I'd consider it technobabble
imiric 2y ago
Hasn't this research been done by teams building NPUs today? E.g. chips built by Groq use an architecture built specifically for AI, which is why they're able to deliver the performance they do. On the consumer side, Apple silicon is also quite capable. I'm not in this field at all, but it seems to me that using general purpose processors that communicate over (relatively) slow lanes can only get us so far. Rethinking the design at the hardware level, and eventually bringing the price down for the consumer market seems like a better long-term strategy.
- resource_waste 2y ago
  >On the consumer side, Apple silicon is also quite capable. I am not sure that is true. A glance/or long stay at the reddit localllama subreddit basically has a bunch of frustrated CPU users trying their absolute best to get anything to work at useful speeds. When you can get an Nvidia GPU for a few hundred dollars or a full blown gaming laptop with a 4050 6gb vram for $900, its hard to call a CPU based AI capable. Heck we don't have GPUs at work, and CPU based is just not really reasonable without using tiny models and waiting. We ended up requesting GPU computers. I think there is a 'this is technically possible', and there is a 'this is really nice'. Nvidia has been really nice to use. CPU has been miserable and frustrating.
  - imiric 2y ago
    I don't think NVIDIA's reign will last long. The recent AI resurgence is not even a decade old. We can't expect the entire industry to shift overnight, but we are seeing rapid improvements in the capability of non-GPU hardware to run AI workloads. The architecture change has been instrumental for this, and Apple is well positioned to move the field forward, even if their current gen hardware is lacking compared to traditional GPUs. Their silicon is not even 5 years old, yet it's unbeatable for traditional workloads and power efficiency, and competitive for AI ones. What do you think it will be capable of in 5 years from now? Same for Groq, and other NPU manufacturers. Betting on NVIDIA doesn't seem like a good long-term strategy, unless they also shift their architecture.
  - serialx 2y ago
roschdal 2y ago
ChatGPT - the largest electricity bill in the world.
winternewt 2y ago
I believe that reducing the power consumption and increasing the speed of AI inference will be best served by switching to analog, approximate circuits. We don't need perfect floating-point multiplication and addition, we just need something that takes an two input voltages and produces an output voltage that is close enough to what multiplying the input voltages would yield.
- brap 2y ago
  I don’t know why you’re being downvoted, that’s an active area of research AFAIK
  - gitfan86 2y ago
    Maybe because that is a VERY different problem than the one discussed here. Building a single analog chip with 1 billion neurons would cost billions of dollars in a best case scenario. A Nvidia card with 1 billion digital neurons is in the hundreds of dollars of range. Those costs could come down eventually, but at that point CUDA may be long gone.
  - cyanide911 2y ago
    Do you have any references of papers/people working on this? I'm very interested in the possibilities that lie here, but have no idea where to start
- brazzy 2y ago
  Sounds pretty impossble to me do that with a sufficient combination of range and precision.
  - atoav 2y ago
    What do you mean with inpossible? You are aware that what radio equipment does is often equivalent of analog operations like multiplication, addition, etc. just at high frequencies? Sure accuracy is an issue, but this is not as impossible as you may think it would be. The main question will be if the benefits by going analog outweigh the issues arising from it.
_spl 2y ago
It reminds me of when I first read about superscalar CPU architecture and was amazed. GPUs are really next level.
DeathArrow 2y ago
It would be nice if such improvements find their way in pytorch and scikit-learn.
- kmacdough 2y ago
  I'm sure they will. Right now it's, though, it's bleeding edge and it'll take some time for these ideas to mature and be adapted to the particular idioms of these more stable packages.
bombela 2y ago
I cannot tell for sure if units are really all power of 10. I found some datasheet that states 80GB of VRAM, and a BAR of 80GiB. All caches are also in power of two. The bandwidth are all power of 10 though. https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/data-center/h100/PB-11133-001_v01.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/da...
joaquincabezas 2y ago
wow their graphs at the GitHub README (https://github.com/HazyResearch/ThunderKittens/blob/main/attn.png https://github.com/HazyResearch/ThunderKittens/blob/main/att...) make me extremely dizzy. Are these wavy bars even legal? :P
- bogtog 2y ago
  I second this. It's like they're trying to incorporate some optical illusion. I'd even prefer just seeing numbers without any bars
- hoosieree 2y ago
  It looks like the xkcd theme for matplotlib[1]. But I agree the waves are too extreme. [1]: https://matplotlib.org/stable/gallery/showcase/xkcd.html#sphx-glr-gallery-showcase-xkcd-py https://matplotlib.org/stable/gallery/showcase/xkcd.html#sph...
badgersnake 2y ago
That’s the whole point, VCs invested heavily in GPUs anticipating a crypto boom and when that never happened they had to find some other snake oil to peddle that happened to require GPUs.
- verbify 2y ago
  My experience is that when crypto was in the news, my non-technical friends, family, and colleagues would ask me what is bitcoin and were generally confused. My experience with the AI boom couldn't be more different - everyone from my colleagues to my mum are using chatgpt as a daily tool. I really don't think that AI and crypto are comparable in terms of their current practical usage.
  - kkielhofner 2y ago
    Comparing crypto and AI is really tired and you make the best point - real people are using these GPUs to actually do things of value and improve their daily lives. At the peak of the crypto boom/hype cycle I took on a little project to look at the top 10 blockchain networks/coins/whatever. From what I could tell a very, very, very generous estimate is that crypto at best has MAUs in the low tens of millions. ChatGPT alone got to 100 million MAUs within a year of release and has only grown since. ChatGPT 10x'd actual real world usage of GPUs (and resulting power and other resources) in a year vs ~15 years for crypto. > I really don't think that AI and crypto are comparable in terms of their current practical usage. A massive understatement!
    - latchkey 2y ago
      GPUs stopped being used for crypto because Ethereum switched from PoW to PoS and that decimated the whole gpu mining industry. Ethereum was the only profitable thing to mine, that also had a usecase. The rest of the chains dumped in price and became unprofitable to mine at scale. Not enough market depth to unload the tokens at scale. In other words, it has nothing to do with AI.
      - 2y ago
weinzierl 2y ago
"For this post, we’re going to focus on the NVIDIA H100 [... because] we think the trends it implies are going to continue in future generations, and probably from other manufacturers, too." Is it though? Wouldn't we expect to see more advanced packaging technology eventually? If that happens the increased memory bandwidth could be an enabler for a unified memory architecture like in the Nvidia Jetson line. In turn that would make a lot of what the article says make GPU go Brr today moot.
lucidrains 2y ago
would be interested to see thunderkittens (great name!) tackle the flash attention backwards pass, which is an order of magnitude harder than the forward
- Aaryan44 2y ago
  good news - we've actually included optimized causal and non-causal versions of the flash attention backwards pass with TK - would love for you to check them out! causal: https://github.com/HazyResearch/ThunderKittens/blob/main/examples/attn_causal/h100_train.cu https://github.com/HazyResearch/ThunderKittens/blob/main/exa... non-causal: https://github.com/HazyResearch/ThunderKittens/blob/main/examples/attn/h100/h100_train.cu https://github.com/HazyResearch/ThunderKittens/blob/main/exa...
  - lucidrains 2y ago
    amazing work! thank you!
    - Aaryan44 2y ago
      Thanks @lucidrains :)
  - pama 2y ago
    Awesome. Do you happen to have a benchmark against the latest (v9.1) cuDNN implementation?
    - Aaryan44 2y ago
      @pama, if useful - here are utilization numbers for our attention backwards kernels (causal and non-causal, head dim = 64): https://github.com/HazyResearch/ThunderKittens/blob/main/attn.png https://github.com/HazyResearch/ThunderKittens/blob/main/att...
LordShredda 2y ago
Standford research team just published an article with a wojak in it. That by itself is bigger news than AI
chefandy 2y ago
One of my biggest struggles in doing AI stuff on consumer hardware is heat. I noticed zero discussion of this so I assume it's an implementation detail on small systems that doesn't really factor into more robust setups. Is that the really case, or is this just diving into the comp sci layer of hardware utilization and ignoring things like heat because it's not salient to this subtopic?
- nostrebored 2y ago
  It factors into robust setups but is part and parcel of doing any HPC where you're pushing through a ton of TFLOPS. It's a problem that is assumed to have been solved when you're doing this kind of work.
danjl 2y ago
I bet traditional image processing would love to be implemented in ThunderKitten.
Cyberdog 2y ago
I remember when "compute" was a verb.
wmab 2y ago
The amount of comma splicing, (parentheses for extra points) -- and em dashes for good measure! that this post has makes it entirely unreadable.
- eimrine 2y ago
  This is common in Russian texts but I haven't found any other signs of that suppose.
hi-v-rocknroll 2y ago
NVIDIAs stock will plummet in 3-4 years after Microsoft and Meta stop spending tens of billions without having a specific use for H100's and end up with a ridiculous amount of excess capacity. Hopefully, that means some H100-based systems will end up on eBay in ~5-8 years for home lab use.
rajnathani 2y ago
Not related to the substance of the post: They should've really avoided the 4chan like drawings in the post.
nmstoker 2y ago
Some related material here too: https://twitter.com/bfspector/status/1789749117104894179?t=kruiIMW5J9cDq_RNrHHINg&s=19 https://twitter.com/bfspector/status/1789749117104894179?t=k...
jauntywundrkind 2y ago
The ThunderKittens mascot has great kitten/Sony-Aibo vibes. Nicely generated, AI (I presume). https://github.com/HazyResearch/ThunderKittens https://github.com/HazyResearch/ThunderKittens
- layer8 2y ago
  It looks off because the head isn’t centered on the neck.
  - john_minsk 2y ago
    Great attention to detail! I, like the parent, was surprised by the quality as well. However now I can't unsee it:-)
  - Satam 2y ago
    Easy fix: https://imgur.com/a/Ahwt6tr https://imgur.com/a/Ahwt6tr (although not sure which one is actually better)
perfmode 2y ago
This article rekindles the joy I experienced during CS 149 Parallel Programming.
- figbert 2y ago
  Appreciate the recommendation, will check out the course!
- Aaryan44 2y ago
  Kayvon and Kunle are amazing - I took CS149 Parallel Programming two quarters ago and loved it :)
  - perfmode 2y ago
    lucky! i took it 11 years ago. would love to revisit the material, especially in this new era of specialized processing units and UMA.
behnamoh 2y ago
tangential: When @sama talks about "Universal Basic Compute" (UBC) as a substitute for Universal Basic Income, obviously he means GPU, right? Who's going to benefit from such policies? Only nvidia? It just seems such a dystopian future to live in: imagine you can sell your UBC to others who know better how to use it, or you can use it to mine bitcoin or whatever. But all the compute is actually created by one company. There are many reasons to hate nvidia, but honestly if this UBC policy is even remotely being considered in some circles, I'd join Linus Torvalds and say "nvidia, fuck you".
- callalex 2y ago
  You’re looking for logic. The only logic is “when a sucker buys WorldCoin, sama bank account go brrrr”. That’s the whole logic.
- jra101 2y ago
  You're blaming NVIDIA for Sam Altman's dumb idea?
  - behnamoh 2y ago
    nvidia's CEO literally keeps saying "the more you buy GPUs, the more you save"—it's hard to believe nvidia has nothing to do with such ideas.
    - WanderPanda 2y ago
      Him saying this always puts me off. Gives hard old sales-guy vibes. I really wonder who/which demographic is influenced in nvidias favor by this rethoric.
      - deleted 2y ago
        [deleted]
    - coffeebeqn 2y ago
      GPU CEO wants to sell more GPUs? What on earth
Animats 2y ago
"And we ask: if your matrix multiply is smaller than 16x16, are you sure what you’re doing is AI? From a philosophical point of view, we think a frame shift is in order. A “register” certainly shouldn’t be a 32-bit word like on the CPUs of old. And a 1024-bit wide vector register, as CUDA uses, is certainly a step in the right direction. But to us a “register” is a 16x16 tile of data. We think AI wants this." The hardware needs of AI are starting to focus. GPUs, after all, were designed for an entirely different job. They're used for AI because they have good matrix multiply hardware. "AI GPUs" get to leave out some of the stuff in a real GPU (does an H100 even have texture fill units?). Then there's a trend towards much shorter numbers. 16 bit floating point? 8 bit? 2 bit? 1 bit? That will settle out at some point. This paper indicates that hardware that likes 16x16 tiles makes a lot of sense. It's certainly possible to build such hardware. Someone reading this is probably writing it in VHDL right now, or will be soon. Then we'll see somewhat simpler, less general, and cheaper devices that do "AI" operations with as little excess hardware baggage as possible. Nice.
- deleted 2y ago
  [deleted]
- choppaface 2y ago
  “NVidia’s LIES.. On kernels such as flash attention, TMA and the L2 cache are both fast enough so as to hide these problems reasonably well. But to make the full use of the hardware, memory request must be coalesced and bank conflicts avoided ” The depth of the competition is also starting to become apparent. There’s no way the documentation error was totally an accident. Diagrams are the easiest to steal / copy and there must have been some utility for nvidia to have left this in place. Remember when Naveen Rao’s Nervana was writing NVidia Maxwell drivers that out-performed NVidia’s own? Not every documentation mishap in a high-growth product is a competition counter-measure, but given that the researchers spent so long reverse-engineering wgmma and given the China-US political situation of the H100 in particular, it seems NVidia is up to its old tricks to protect its moat. So don’t over-study the H100 peculiarities, as “what hardware does AI want?” really encompasses the commercial situation as well.
  - 2y ago
uyzstvqs 2y ago
What is needed are true NPUs as dedicated co-processors, especially for prosumer desktop systems (devs, other professionals, gamers). GPUs work in the enterprise, but they're a hassle to use for AI on the personal computing side of the market. Especially VRAM limitations, but also the lack of a standard open API other than Vulkan (again, using video stuff for AI).
- dartos 2y ago
  Fwiw, Vulkan isn’t specifically a graphics api and has had compute specific features for a while now. (Potentially since its inception)
  - the__alchemist 2y ago
    Compared to CUDA, Vulkan is... not fun to code compute in! The serialization bridge and duplicating data structures and functions between CPU and GPU is tedious.
    - dartos 2y ago
      I hear both CUDA and Vulkan are not fun to code in. But yeah Vulkan is famously verbose. It takes about 1000 LoC to draw a triangle
      - KeplerBoy 2y ago
        CUDA is very much fun to code in! Nvidia provides devs with great tools (Nsight Systems and Nsight Compute), so you know where you have to optimize.
jokoon 2y ago
this is why people should better study neuroscience, psychology if they want to advance research in AI. also things related to graph topology in neural networks maybe, but probably not related to artificial NN. I was given this video, which I found was pretty interesting: https://www.youtube.com/watch?v=nkdZRBFtqSs https://www.youtube.com/watch?v=nkdZRBFtqSs (How Developers might stop worrying about AI taking software jobs and Learn to Profit from LLMs - YouTube)
- renewiltord 2y ago
  There are loads of psychologists and neuroscientists today. Has any of them in the last few years produced anything advancing AI? The proof of the pudding is in the eating so if they have at a higher rate than just straight CS/Mathematics and related then there’s probably some truth to it.
- chmod775 2y ago
  I can't seem to figure out the connection between this comment and the article at hand, except that they're both about AI.
- dartos 2y ago
  I don’t think psychology will have any bearing on AI. I doubt neuroscience will either, but I’m not as sure on that. The more impressive AI systems we have moved further away from the neuron analogy that came from perceptions. The whole “intelligence” and “neural” part of AI is a red herring imo. Really poor ambiguous word choice for a specific, technical idea.
  - sva_ 2y ago
    > I doubt neuroscience will either, but I’m not as sure on that The stuff on spiking networks and neuromorphic computing is definitely interesting and inspired by neuroscience, but it currently seems mostly like vaporware
    - dartos 2y ago
      Yep, I’ve heard about spiking networks, but haven’t read into them much yet.
WanderPanda 2y ago
Is this "just" CUTLASS in user friendly?
phinnaeus 2y ago
FYI the caption of the "spirit animals" image says "canadian goose" instead of "Canada Goose".
- downrightmike 2y ago
  Don't worry, the Geese are en route to location, resolution incoming. Stand by.
  - hoherd 2y ago
    In my experience, Canadian geese are never en route to anywhere. They stay next to the pond year round and crap everywhere you might want to step. EG: https://sanjosespotlight.com/going-to-santa-clara-central-park-watch-where-you-step-canada-goose-geese-poop-droppings/ https://sanjosespotlight.com/going-to-santa-clara-central-pa...
- fastball 2y ago
  Canadian goose seems better in [current year], to avoid confusion with the clothing brand.
- wglb 2y ago
  An error too often made.
- adzm 2y ago
  Likely a regional thing; they are consistently called Canadian Geese where I grew up and where I currently live.
- bombcar 2y ago
  It’s a Canada Goose from Canada. A Canadian Canada Goose, or Canadian Goose.
  - gosub100 2y ago
    https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...

1104 points by nmstoker 2y ago

Really impressed by the writing style of this post and very much looking forward to this on AMD MI300x. Let me know if you want some time on mine.

jsemrau 2y ago
Really? It gives me PTSD from the Wallstreetbets days.
- forrestthewoods 2y ago
  I also enjoyed the article's style. I utterly despise "academic paper speak". It is, imho, not the most effective style to communicate complex ideas. I find it so much easier to learn from a more casual "blog post" or in-person presentation over stiff, rigid academic speak.
  - kaycey2022 2y ago
    I find both to be useful in different stages. The casual style is very helpful when starting out. But once I have put in a few weeks or months of study in, then the rigor and preciseness of academic style is good as well. I agree with you in the sense that something has "died" in writings the follow academic paper speak these days. Just yesterday I saw an ancient article surfaced by Scientific American and Peter Norvig on System Analysis by Strachey. It uses quite a bit of formal language but is super approachable at the same time. That kind of skill is rarely seen these days.
- david927 2y ago
  > the Wallstreetbets days. https://twitter.com/TheRoaringKitty/status/1790041813379850491 https://twitter.com/TheRoaringKitty/status/17900418133798504...
globular-toast 2y ago
Good writing is clear and unambiguous. With speech there is an opportunity to interrupt and ask for clarification. Writing has one chance to get the message across. A reader shouldn't have to consult knowyourmeme.com to figure out what the heck the authors are trying to say. I don't even know what the title means here. That's how far they've missed the mark.
- _obviously 2y ago
  Wow that really sucks for you. I just read it in 5 minutes and feel much more informed about the subject pf nvidia memory twizzlization. It's kind of funny to me that presumably young college guys are writing in a style that's very readable for my old ass.
  - unethical_ban 2y ago
    >that really sucks for you How can I put this in your vernacular... "Most polite genZ meme enjoyer"
- aetimmes 2y ago
  Even if you're not familiar with the "go brrr" meme (which is the only use of meme-idiom in the article and is used exactly twice), its meaning is easily inferred via context clues from the opening paragraphs. Good writing is also entertaining and engaging.
  - globular-toast 2y ago
    Keyword being also.
  - throwaway1492 2y ago
    As someone who witnessed A-10 CAS fuck some stuff up in a combat zone ie the real “brrrrt” I’ve been mystified by the meme and current useage. No one knows where it comes from nor the slaughter it represents.
    - 2y ago
tracker1 2y ago
Have you done much AI work against AMD products? I'm not going to plunk down $2500+ for an RTX 4090, but have been considering an RX 7900XTX for playing around with, or at least getting started. Just curious how well it will or won't work in practice, or if saving a bit more and getting a 7900 XT over the XTX might be a better option, and how much less vram might impact usefulness in practice.
- latchkey 2y ago
  My only work with consumer AMD GPUs was mining ethereum, I had 150,000 of them. If you want to use enterprise AMD gpus, I'm renting them. That said, I haven't even had a chance to run/play with them myself yet, they have been rented since I got them last month. Yes, we are getting more.
- PeterisP 2y ago
  Caveat emptor and your mileage may vary; but unlike nVidia where you could just assume that everything is compatible with everything, for AMD I'd strongly recommend that you try before you buy - consider renting a cloud machine with that GPU to check if the software works for your needs before committing to a large purchase.
  - latchkey 2y ago
    Agreed! The problem is that you cannot rent a MI300x or other high end AMD. They all go into HPC. A problem that I love to work on.

panki27 2y ago

Warp scheduler, 4 quadrants, tensor memory accelerator, unswizzled wgmma layouts... The line between GPU lingo and Star Trek technobabble fades away further and further.

Agentlien 2y ago
Your comment prompted me to take a step back and look at these terms with new eyes. That made me smile, because you're so right.
- deleted 2y ago
  [deleted]
araes 2y ago
There was some awareness reading the article, yet "we're warping through the quadrant in our tensor accelerator" is pretty Trek. Have had that thought occasionally with some of the other articles. What it must read like to somebody who gets a ref link for an article over here. Wandered into some Trek nerd convention discussing warp cores.
- winwang 2y ago
  I mean, if we're talking about "accelerating by modifying the metric tensor" then yeah, that would be pretty sci-fi :) https://en.wikipedia.org/wiki/Metric_tensor_(general_relativity) https://en.wikipedia.org/wiki/Metric_tensor_(general_relativ...

apsec112 2y ago

Interesting! Would this support fp8? Does anyone know how it would compare to Triton?

renonce 2y ago

> NVIDIA’s lies. This is an extraordinarily misleading representation of the actual 128b swizzled wgmma layout. This diagram cost us three weeks of life that we will not get back, hence the public shaming. Wondering if anyone would be surprised that a huge amount of progress in AI is on the engineering side (optimizing matmuls), and that a huge portion of the engineering is about reverse engineering NVIDIA chips

DeathArrow 2y ago
Architecture doesn't make a difference. Big enough models trained with big enough data tend to give the same results regardless of architecture. So yes, most advances in AI are mostly due to the fact we can now multiply matrices very fast.
- elcomet 2y ago
  That's not completely true. The architecture must behave well for scaling, which is not trivial. Basic multi-layer perceptrons do not scale well for example, the gradient will vanish or explode deeper in the network.
  - 3abiton 2y ago
    And data quality. Ensuring the sourcing and quality is very important to get a good model.
    - fleischhauf 2y ago
      this, if you have money to spend in improving your model, more training data is the first thing I'd take a look at
  - Tarrosion 2y ago
    How do modern foundation models avoid multi-layer perceptron scaling issues? Don't they have big feed-forward components in addition to the transformers?

brcmthrowaway 2y ago

NVIDIA needs to be broken up

huhlig 2y ago
Into what? Where would you draw such lines?
- robocat 2y ago
  Into tiles ;-p GPU compute is already broken up - there is a supply chain of other cooperating players that work together to deliver GPU compute to end users: TSMC, SK hynix, Synopsys, cloud providers (Azure/Amazon etcetera), model providers (OpenAI/Anthropic etcetera). Why single out NVidia in the chain? Plus the different critical parts of the chain are in different jurisdictions. Split up NVidia and somebody else will take over that spot in the ecosystem. This interview with Synopsys is rather enlightening: https://www.acquired.fm/episodes/the-software-behind-silicon-with-synopsys-founder-aart-de-geus-and-ceo-sassine-ghazi https://www.acquired.fm/episodes/the-software-behind-silicon... How does the profit currently get split between the different links? Profit is the forcing variable for market cap and profit is the indicator of advantage. Break up NVidia and where does the profit move?
latchkey 2y ago
The better alternative is to root for AMD and others to develop their own products so that regardless of breaking NV up or not, there are alternative solutions for people to use. They all leapfrog each other with new releases now any way. Why put all your eggs into one basket.
- simondotau 2y ago
  George Hotz went down the AMD rabbit hole for a while and concluded that the driver software — more precisely the firmware which runs on the cards themselves — is so badly written that there's no hope of them becoming serious contenders in AI without some major changes in AMD's priorities.

benreesman 2y ago

[flagged]

spullara 2y ago
please publish it!
- deleted 2y ago
  [deleted]
- benreesman 2y ago
  I’m working on building it, it’s time and resource-intensive. If anyone wants to join the effort, we’re hiring. I said that it was trivial to see that better is possible. I never said that doing better is trivial: actually doing better is hard work, takes time, and costs money. But it can be done and we will.

diginova 2y ago

What should I do if I want to understand such articles in complete? where to start on the roadmap?

kolinko 2y ago
This is a good course on gpu programming. Around 4.0 lesson you’ll get the required basics: https://youtube.com/playlist?list=PLzn6LN6WhlN06hIOA_ge6SrgdeSiuf9Tb&si=CE4C7wrkz55YCUqG https://youtube.com/playlist?list=PLzn6LN6WhlN06hIOA_ge6Srgd... Also, write your own cuda kernel to do vector-matrix multiplication (if you use pycuda, you can focus on the kernel, and write everything else with python). Just tell chatgpt that you want to write your own implementation that multiplies a 4000-element vector by 4000x12000 matrix, and to guide you through the whole process. For renting gpus, runpods is great - right now they have everything from lower tier gpus to h100s. You can start with a lesser gpu at the beginning.
abstractcontrol 2y ago
For a deep dive, maybe take a look at the Spiral matrix multiplication playlist: https://www.youtube.com/playlist?list=PL04PGV4cTuIWT_NXvvZsnlqYEUy6OmIc3 https://www.youtube.com/playlist?list=PL04PGV4cTuIWT_NXvvZsn... I spent 2 months implementing a matmult kernel in Spiral and optimizing it.
- justplay 2y ago
  sorry for noob question, how gpu programming is helpful ?
  - abstractcontrol 2y ago
    NNs for example are (mostly) a sequence of matrix multiplication operations, and GPUs are very good at those. Much better than CPUs. AI is hot at the moment, and Nvidia is producing the kind of hardware that can run large models efficiently which is why it's a 2 trillion-dollar company right now. However, in the Spiral series, I aim to go beyond just making an ML library for running NN models and break new ground. Newer GPUs actually support dynamic memory allocation, recursion, and the GPU threads have their own stacks, so you could in fact treat them as sequential devices and write games and simulators directly on them. I think once I finish the NL Holdem game, I'll be able to get over 100x fold improvements by running the whole program on the GPU versus the old approach of writing the sequential part on a CPU and only using the GPU to accelerate a NN model powering the computer agents. I am not sure if this is a good answer, but this is how GPU programming would be helpful to me. It all comes down to performance. The problem with programming them is that the program you are trying to speed up needs to be specially structured, so it utilizes the full capacity of the device.

DeathArrow 2y ago

So do their kernels and library also speed up RTX 4090?

cl3misch 2y ago

> The unswizzled shared memory layouts suffer from very poor coalescing If I didn't know any better I'd consider it technobabble

imiric 2y ago

Hasn't this research been done by teams building NPUs today? E.g. chips built by Groq use an architecture built specifically for AI, which is why they're able to deliver the performance they do. On the consumer side, Apple silicon is also quite capable. I'm not in this field at all, but it seems to me that using general purpose processors that communicate over (relatively) slow lanes can only get us so far. Rethinking the design at the hardware level, and eventually bringing the price down for the consumer market seems like a better long-term strategy.

resource_waste 2y ago
>On the consumer side, Apple silicon is also quite capable. I am not sure that is true. A glance/or long stay at the reddit localllama subreddit basically has a bunch of frustrated CPU users trying their absolute best to get anything to work at useful speeds. When you can get an Nvidia GPU for a few hundred dollars or a full blown gaming laptop with a 4050 6gb vram for $900, its hard to call a CPU based AI capable. Heck we don't have GPUs at work, and CPU based is just not really reasonable without using tiny models and waiting. We ended up requesting GPU computers. I think there is a 'this is technically possible', and there is a 'this is really nice'. Nvidia has been really nice to use. CPU has been miserable and frustrating.
- imiric 2y ago
  I don't think NVIDIA's reign will last long. The recent AI resurgence is not even a decade old. We can't expect the entire industry to shift overnight, but we are seeing rapid improvements in the capability of non-GPU hardware to run AI workloads. The architecture change has been instrumental for this, and Apple is well positioned to move the field forward, even if their current gen hardware is lacking compared to traditional GPUs. Their silicon is not even 5 years old, yet it's unbeatable for traditional workloads and power efficiency, and competitive for AI ones. What do you think it will be capable of in 5 years from now? Same for Groq, and other NPU manufacturers. Betting on NVIDIA doesn't seem like a good long-term strategy, unless they also shift their architecture.
- serialx 2y ago

roschdal 2y ago

ChatGPT - the largest electricity bill in the world.

winternewt 2y ago

I believe that reducing the power consumption and increasing the speed of AI inference will be best served by switching to analog, approximate circuits. We don't need perfect floating-point multiplication and addition, we just need something that takes an two input voltages and produces an output voltage that is close enough to what multiplying the input voltages would yield.

brap 2y ago
I don’t know why you’re being downvoted, that’s an active area of research AFAIK
- gitfan86 2y ago
  Maybe because that is a VERY different problem than the one discussed here. Building a single analog chip with 1 billion neurons would cost billions of dollars in a best case scenario. A Nvidia card with 1 billion digital neurons is in the hundreds of dollars of range. Those costs could come down eventually, but at that point CUDA may be long gone.
- cyanide911 2y ago
  Do you have any references of papers/people working on this? I'm very interested in the possibilities that lie here, but have no idea where to start
brazzy 2y ago
Sounds pretty impossble to me do that with a sufficient combination of range and precision.
- atoav 2y ago
  What do you mean with inpossible? You are aware that what radio equipment does is often equivalent of analog operations like multiplication, addition, etc. just at high frequencies? Sure accuracy is an issue, but this is not as impossible as you may think it would be. The main question will be if the benefits by going analog outweigh the issues arising from it.

_spl 2y ago

It reminds me of when I first read about superscalar CPU architecture and was amazed. GPUs are really next level.

DeathArrow 2y ago

It would be nice if such improvements find their way in pytorch and scikit-learn.

kmacdough 2y ago
I'm sure they will. Right now it's, though, it's bleeding edge and it'll take some time for these ideas to mature and be adapted to the particular idioms of these more stable packages.

bombela 2y ago

I cannot tell for sure if units are really all power of 10. I found some datasheet that states 80GB of VRAM, and a BAR of 80GiB. All caches are also in power of two. The bandwidth are all power of 10 though. https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/data-center/h100/PB-11133-001_v01.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/da...

joaquincabezas 2y ago

wow their graphs at the GitHub README (https://github.com/HazyResearch/ThunderKittens/blob/main/attn.png https://github.com/HazyResearch/ThunderKittens/blob/main/att...) make me extremely dizzy. Are these wavy bars even legal? :P

bogtog 2y ago
I second this. It's like they're trying to incorporate some optical illusion. I'd even prefer just seeing numbers without any bars
hoosieree 2y ago
It looks like the xkcd theme for matplotlib[1]. But I agree the waves are too extreme. [1]: https://matplotlib.org/stable/gallery/showcase/xkcd.html#sphx-glr-gallery-showcase-xkcd-py https://matplotlib.org/stable/gallery/showcase/xkcd.html#sph...

badgersnake 2y ago

That’s the whole point, VCs invested heavily in GPUs anticipating a crypto boom and when that never happened they had to find some other snake oil to peddle that happened to require GPUs.

verbify 2y ago
My experience is that when crypto was in the news, my non-technical friends, family, and colleagues would ask me what is bitcoin and were generally confused. My experience with the AI boom couldn't be more different - everyone from my colleagues to my mum are using chatgpt as a daily tool. I really don't think that AI and crypto are comparable in terms of their current practical usage.
- kkielhofner 2y ago
  Comparing crypto and AI is really tired and you make the best point - real people are using these GPUs to actually do things of value and improve their daily lives. At the peak of the crypto boom/hype cycle I took on a little project to look at the top 10 blockchain networks/coins/whatever. From what I could tell a very, very, very generous estimate is that crypto at best has MAUs in the low tens of millions. ChatGPT alone got to 100 million MAUs within a year of release and has only grown since. ChatGPT 10x'd actual real world usage of GPUs (and resulting power and other resources) in a year vs ~15 years for crypto. > I really don't think that AI and crypto are comparable in terms of their current practical usage. A massive understatement!
  - latchkey 2y ago
    GPUs stopped being used for crypto because Ethereum switched from PoW to PoS and that decimated the whole gpu mining industry. Ethereum was the only profitable thing to mine, that also had a usecase. The rest of the chains dumped in price and became unprofitable to mine at scale. Not enough market depth to unload the tokens at scale. In other words, it has nothing to do with AI.
    - 2y ago

weinzierl 2y ago

"For this post, we’re going to focus on the NVIDIA H100 [... because] we think the trends it implies are going to continue in future generations, and probably from other manufacturers, too." Is it though? Wouldn't we expect to see more advanced packaging technology eventually? If that happens the increased memory bandwidth could be an enabler for a unified memory architecture like in the Nvidia Jetson line. In turn that would make a lot of what the article says make GPU go Brr today moot.

lucidrains 2y ago

would be interested to see thunderkittens (great name!) tackle the flash attention backwards pass, which is an order of magnitude harder than the forward

Aaryan44 2y ago
good news - we've actually included optimized causal and non-causal versions of the flash attention backwards pass with TK - would love for you to check them out! causal: https://github.com/HazyResearch/ThunderKittens/blob/main/examples/attn_causal/h100_train.cu https://github.com/HazyResearch/ThunderKittens/blob/main/exa... non-causal: https://github.com/HazyResearch/ThunderKittens/blob/main/examples/attn/h100/h100_train.cu https://github.com/HazyResearch/ThunderKittens/blob/main/exa...
- lucidrains 2y ago
  amazing work! thank you!
  - Aaryan44 2y ago
    Thanks @lucidrains :)
- pama 2y ago
  Awesome. Do you happen to have a benchmark against the latest (v9.1) cuDNN implementation?
  - Aaryan44 2y ago
    @pama, if useful - here are utilization numbers for our attention backwards kernels (causal and non-causal, head dim = 64): https://github.com/HazyResearch/ThunderKittens/blob/main/attn.png https://github.com/HazyResearch/ThunderKittens/blob/main/att...

LordShredda 2y ago

Standford research team just published an article with a wojak in it. That by itself is bigger news than AI

chefandy 2y ago

One of my biggest struggles in doing AI stuff on consumer hardware is heat. I noticed zero discussion of this so I assume it's an implementation detail on small systems that doesn't really factor into more robust setups. Is that the really case, or is this just diving into the comp sci layer of hardware utilization and ignoring things like heat because it's not salient to this subtopic?

nostrebored 2y ago
It factors into robust setups but is part and parcel of doing any HPC where you're pushing through a ton of TFLOPS. It's a problem that is assumed to have been solved when you're doing this kind of work.

danjl 2y ago

I bet traditional image processing would love to be implemented in ThunderKitten.

Cyberdog 2y ago

I remember when "compute" was a verb.

wmab 2y ago

The amount of comma splicing, (parentheses for extra points) -- and em dashes for good measure! that this post has makes it entirely unreadable.

eimrine 2y ago
This is common in Russian texts but I haven't found any other signs of that suppose.

hi-v-rocknroll 2y ago

NVIDIAs stock will plummet in 3-4 years after Microsoft and Meta stop spending tens of billions without having a specific use for H100's and end up with a ridiculous amount of excess capacity. Hopefully, that means some H100-based systems will end up on eBay in ~5-8 years for home lab use.

rajnathani 2y ago

Not related to the substance of the post: They should've really avoided the 4chan like drawings in the post.

nmstoker 2y ago

Some related material here too: https://twitter.com/bfspector/status/1789749117104894179?t=kruiIMW5J9cDq_RNrHHINg&s=19 https://twitter.com/bfspector/status/1789749117104894179?t=k...

jauntywundrkind 2y ago

The ThunderKittens mascot has great kitten/Sony-Aibo vibes. Nicely generated, AI (I presume). https://github.com/HazyResearch/ThunderKittens https://github.com/HazyResearch/ThunderKittens

layer8 2y ago
It looks off because the head isn’t centered on the neck.
- john_minsk 2y ago
  Great attention to detail! I, like the parent, was surprised by the quality as well. However now I can't unsee it:-)
- Satam 2y ago
  Easy fix: https://imgur.com/a/Ahwt6tr https://imgur.com/a/Ahwt6tr (although not sure which one is actually better)

perfmode 2y ago

This article rekindles the joy I experienced during CS 149 Parallel Programming.

figbert 2y ago
Appreciate the recommendation, will check out the course!
Aaryan44 2y ago
Kayvon and Kunle are amazing - I took CS149 Parallel Programming two quarters ago and loved it :)
- perfmode 2y ago
  lucky! i took it 11 years ago. would love to revisit the material, especially in this new era of specialized processing units and UMA.

behnamoh 2y ago

tangential: When @sama talks about "Universal Basic Compute" (UBC) as a substitute for Universal Basic Income, obviously he means GPU, right? Who's going to benefit from such policies? Only nvidia? It just seems such a dystopian future to live in: imagine you can sell your UBC to others who know better how to use it, or you can use it to mine bitcoin or whatever. But all the compute is actually created by one company. There are many reasons to hate nvidia, but honestly if this UBC policy is even remotely being considered in some circles, I'd join Linus Torvalds and say "nvidia, fuck you".

callalex 2y ago
You’re looking for logic. The only logic is “when a sucker buys WorldCoin, sama bank account go brrrr”. That’s the whole logic.
jra101 2y ago
You're blaming NVIDIA for Sam Altman's dumb idea?
- behnamoh 2y ago
  nvidia's CEO literally keeps saying "the more you buy GPUs, the more you save"—it's hard to believe nvidia has nothing to do with such ideas.
  - WanderPanda 2y ago
    Him saying this always puts me off. Gives hard old sales-guy vibes. I really wonder who/which demographic is influenced in nvidias favor by this rethoric.
    - deleted 2y ago
      [deleted]
  - coffeebeqn 2y ago
    GPU CEO wants to sell more GPUs? What on earth

Animats 2y ago

"And we ask: if your matrix multiply is smaller than 16x16, are you sure what you’re doing is AI? From a philosophical point of view, we think a frame shift is in order. A “register” certainly shouldn’t be a 32-bit word like on the CPUs of old. And a 1024-bit wide vector register, as CUDA uses, is certainly a step in the right direction. But to us a “register” is a 16x16 tile of data. We think AI wants this." The hardware needs of AI are starting to focus. GPUs, after all, were designed for an entirely different job. They're used for AI because they have good matrix multiply hardware. "AI GPUs" get to leave out some of the stuff in a real GPU (does an H100 even have texture fill units?). Then there's a trend towards much shorter numbers. 16 bit floating point? 8 bit? 2 bit? 1 bit? That will settle out at some point. This paper indicates that hardware that likes 16x16 tiles makes a lot of sense. It's certainly possible to build such hardware. Someone reading this is probably writing it in VHDL right now, or will be soon. Then we'll see somewhat simpler, less general, and cheaper devices that do "AI" operations with as little excess hardware baggage as possible. Nice.

deleted 2y ago
[deleted]
choppaface 2y ago
“NVidia’s LIES.. On kernels such as flash attention, TMA and the L2 cache are both fast enough so as to hide these problems reasonably well. But to make the full use of the hardware, memory request must be coalesced and bank conflicts avoided ” The depth of the competition is also starting to become apparent. There’s no way the documentation error was totally an accident. Diagrams are the easiest to steal / copy and there must have been some utility for nvidia to have left this in place. Remember when Naveen Rao’s Nervana was writing NVidia Maxwell drivers that out-performed NVidia’s own? Not every documentation mishap in a high-growth product is a competition counter-measure, but given that the researchers spent so long reverse-engineering wgmma and given the China-US political situation of the H100 in particular, it seems NVidia is up to its old tricks to protect its moat. So don’t over-study the H100 peculiarities, as “what hardware does AI want?” really encompasses the commercial situation as well.
- 2y ago

uyzstvqs 2y ago

What is needed are true NPUs as dedicated co-processors, especially for prosumer desktop systems (devs, other professionals, gamers). GPUs work in the enterprise, but they're a hassle to use for AI on the personal computing side of the market. Especially VRAM limitations, but also the lack of a standard open API other than Vulkan (again, using video stuff for AI).

dartos 2y ago
Fwiw, Vulkan isn’t specifically a graphics api and has had compute specific features for a while now. (Potentially since its inception)
- the__alchemist 2y ago
  Compared to CUDA, Vulkan is... not fun to code compute in! The serialization bridge and duplicating data structures and functions between CPU and GPU is tedious.
  - dartos 2y ago
    I hear both CUDA and Vulkan are not fun to code in. But yeah Vulkan is famously verbose. It takes about 1000 LoC to draw a triangle
    - KeplerBoy 2y ago
      CUDA is very much fun to code in! Nvidia provides devs with great tools (Nsight Systems and Nsight Compute), so you know where you have to optimize.

jokoon 2y ago

this is why people should better study neuroscience, psychology if they want to advance research in AI. also things related to graph topology in neural networks maybe, but probably not related to artificial NN. I was given this video, which I found was pretty interesting: https://www.youtube.com/watch?v=nkdZRBFtqSs https://www.youtube.com/watch?v=nkdZRBFtqSs (How Developers might stop worrying about AI taking software jobs and Learn to Profit from LLMs - YouTube)

renewiltord 2y ago
There are loads of psychologists and neuroscientists today. Has any of them in the last few years produced anything advancing AI? The proof of the pudding is in the eating so if they have at a higher rate than just straight CS/Mathematics and related then there’s probably some truth to it.
chmod775 2y ago
I can't seem to figure out the connection between this comment and the article at hand, except that they're both about AI.
dartos 2y ago
I don’t think psychology will have any bearing on AI. I doubt neuroscience will either, but I’m not as sure on that. The more impressive AI systems we have moved further away from the neuron analogy that came from perceptions. The whole “intelligence” and “neural” part of AI is a red herring imo. Really poor ambiguous word choice for a specific, technical idea.
- sva_ 2y ago
  > I doubt neuroscience will either, but I’m not as sure on that The stuff on spiking networks and neuromorphic computing is definitely interesting and inspired by neuroscience, but it currently seems mostly like vaporware
  - dartos 2y ago
    Yep, I’ve heard about spiking networks, but haven’t read into them much yet.

WanderPanda 2y ago

Is this "just" CUTLASS in user friendly?

phinnaeus 2y ago

FYI the caption of the "spirit animals" image says "canadian goose" instead of "Canada Goose".

downrightmike 2y ago
Don't worry, the Geese are en route to location, resolution incoming. Stand by.
- hoherd 2y ago
  In my experience, Canadian geese are never en route to anywhere. They stay next to the pond year round and crap everywhere you might want to step. EG: https://sanjosespotlight.com/going-to-santa-clara-central-park-watch-where-you-step-canada-goose-geese-poop-droppings/ https://sanjosespotlight.com/going-to-santa-clara-central-pa...
fastball 2y ago
Canadian goose seems better in [current year], to avoid confusion with the clothing brand.
wglb 2y ago
An error too often made.
adzm 2y ago
Likely a regional thing; they are consistently called Canadian Geese where I grew up and where I currently live.
bombcar 2y ago
It’s a Canada Goose from Canada. A Canadian Canada Goose, or Canadian Goose.
- gosub100 2y ago
  https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...