10 ms·

What every developer should know about GPU computing

465 points by Anon84 3y ago

abhi9u 3y ago
Thank you for sharing!
Jeff_Brown 3y ago
Not every developer. I'm not trying to be snarky. I think there's an unhelpful compunction to want to know everything about everything among STEM types like programmers (of which I am one). Specialization is fundamental to the success, not just of whole economies, but of the individuals in them. It can feel like a paintful sacrifice to admit that you'll never (have time to) learn, say, the entire Python language specification, or how type inference works, or any number of other things someone might tell you is critical knowledge. But it's often liberating, and more often than that, mandatory. (Maybe I was trying to be a little snarky initially, but I'm not any more.)
- Jiro 3y ago
  Over-specialization is hard on your ability to find another job if you lose your current one.
  - Jeff_Brown 3y ago
    Completely true. The analogy to hydration holds, I think -- too much and too little can both be problematic. But I see seem to run into much more under-specialization than over-specialization. More than 90% of the wildest successes I know went really deep on something pretty narrow.
    - HPsquared 3y ago
      Specialization is like putting everything on one number in roulette. It's great if you win (i.e. your specialty is in demand), but you're more likely to lose everything you invested.
      - Jeff_Brown 3y ago
        It is a risk, but not as bad as you suggest, because (1) in work you can get an incremental sense of what is worth specializing in, and (2) there's always some degree to which skills are transferrable and experience is impressive even to someone who doesn't need you to do what you did again.
  - andrewSC 3y ago
    And so it’s a balance :^) I personally take the approach of specializing in what I believe to be my natural talents (i.e. seemingly able to pick up X much easier than whoever) while just being generally aware of new technologies or other spaces/sectors. Also tinkering and/or having a general interest in tech helps lol.. IME doing this has helped me pivot into a new tech or other areas where I may not be specialized in, but may need to eventually be for a new role. Also worth mentioning is taking a pragmatic approach to problem solving in general.. I’ve personally found that if you’re able to demonstrate solid reasoning and/or problem solving, generally learning a new lang or specializing in something different than what you’re used to, isn’t too too bad. I’m not really sure how specialists outside of tech/in other sectors can transition into other roles “easily” though… hm. Edit: I wanted to mention that IME in STEM, most? More often than not? are goal oriented… It is _completely_ okay to not have any “working” thing at the end of your tinkering/learning… The journey into that “thing” can be a learning experience in and of itself. I’ve often started to learn things but after a certain point have told myself “I’m gonna stop here and that’s okay. I don’t need to have a solid understanding of this, at this time”. YMMV…
- nabla9 3y ago
  This is just minimum for those don't need to know details. Even general knowledge has depth. Even if you are generalist, or specialist in different area you should deepen your knowledge at every area gradually.
  - abhi9u 3y ago
    True. This is probably what first few lectures of a first course on GPU computing would cover. The nitty gritty of how to write parallel algorithms for maximum throughput on the GPU is where things become too specific. :)
- cratermoon 3y ago
  I dunno. The section "Latency Tolerance, High Throughput and Little’s Law" has many applications in programming. Ever need to scale your cache, or size a connection pool? Little's Law.
  - abhi9u 3y ago
    Thank you for reading! :)
    - cratermoon 3y ago
      Thanks for writing this. I note you mention that GPUs are massively parallel, and I think it would strengthen the article to add a paragraph or two discussing what kinds of loads lend themselves to GPU computing. The sort of "embarrassingly parallel" things like linear algebra vs inherently serial example of numerical analysis.
      - abhi9u 3y ago
        Yes, that's a good feedback. Few other people also mentioned this. I was trying to save space, to spend time more on the "how things work" part. But I see your point that some motivation behind parallelization would make it a more interesting read.
- yodsanklai 3y ago
  > It can feel like a paintful sacrifice to admit that you'll never (have time to) learn And even if you do have time, you'll probably forget most of it if you don't make use of it in your daily job. Also if you do want to learn about a new topic, it'll take commitment. I remember going through an MIT OS project, it took me something like 50 hours of hours to complete the project. Pretty much impossible when you have a full time job (I didn't at the time). And despite that, I still consider myself a newbie in OS development. That being said, a little extra knowledge can come handy and make a difference in an interview for instance, or reduce ramp-up time when changing teams. This is also what school is for. Give you full time and a structured program to pick the fundamentals. There's only so much you can learn once you have a demanding job. It's actually pretty sad we don't get to go back to school in the middle of our career. Edit: still an interesting article ;)
  - abhi9u 3y ago
    Thank you :)
  - owl57 3y ago
    > I remember going through an MIT OS project, it took me something like 50 hours of hours to complete the project. Pretty much impossible when you have a full time job (I didn't at the time). And despite that, I still consider myself a newbie in OS development. I took an OS course loosely based on MIT one, and of course wouldn't consider myself an experienced OS developer as well, but I think the residual knowledge is sometimes really helpful when debugging bad application performance at work.
- pjmlp 3y ago
  Indeed, one thing that most seniors learn is humility and being able to say I don't know, without caring about consequences.
  - bobwaycott 3y ago
    An invaluable life skill, as well.
- loeg 3y ago
  At this point, I think the running shtick / inside joke of "Every Developer Should Know ..." headlines is that of course every developer doesn't need to know the contents of the article that follows.
  - bachmeier 3y ago
    Not true. Anyone experienced in programming discussions knows that "software development" overlaps exactly with the programming done by the commenter, and everything else is solving toy problems with tools that don't scale.
  - wongarsu 3y ago
    > of course every developer doesn't need to know the contents of the article that follows. Things every developer should know about English, part 1: why word order matters in negation /s Obviously you meant "not every developer needs to know" instead of "every developer doesn't need to know", but I see this switch so often lately that I'm beginning to wonder if it's a dialect thing (similar to double negative implying a positive to some and an emphasis of the negative to others)
    - deleted 3y ago
      [deleted]
    - a1369209993 3y ago
      Actually, that seems to be a legitimate precedence ambiguity. The probably more common way to parse/interpret it is: > [Foreach developer D, D] needs to know this. > [Foreach developer D, D] doesn't need to know this. (No developers.) But you could also have: > [The logical conjuction of all developers[0]] needs to know this. > [The logical conjuction of all developers] doesn't need to know this. (Not all developers.) The former convention is clearly better unless I've missed something rather significant, but inherently broken features in a number of programming languages (eg perl 6 'junctive' operators) suggest the latter is well-established as a thing that exists. 0: ie, a linguistic fiction that has only those properties / needs only those things that are common to every developer.
  - 3y ago
- cj 3y ago
  One thing I find interesting about the software industry is our lack of descriptive job titles. At big companies you have SWE L1, L2, L3, senior, staff, principal, etc. You also have SRE and maybe some devops or architect roles. Smaller companies you have lots of people generic “engineer” titles or “full stack engineers”, etc. Why don’t we encode people’s specialties in their titles if most engineers are working on narrow sections of software? E.g. “React & Node Developer” instead of “Full stack engineer” … etc I suppose the easiest rationale is generic titles allow for easier mobility between disciplines.
- crabbone 3y ago
  You are interpreting the title literally, while all this article is trying to do is to give an introduction to anyone who wants to get into GPU programming. Now, about every developer. Ideally, developers should know something, in general, about all fields related to programming. Similar to how in medicine you need to learn about different branches of medicine, even though you'll specialize in one, and so it is in mathematics, physics and so on. Presently, the demand for quality professionals in programming is very low. There aren't any good testing or certification programs that can tell a good programmer from a bad one. The industry is generally happy with "specialists" who perhaps only know to do one thing, somewhat. So, presently you don't need to know anything about GPU or any other field that's not directly related to your job description. ---- Now, about the article itself. While it gives a lot of valuable factual information, it's missing the forest for the trees. It's very dedicated to how CUDA works or some other particular aspects of NVidia's GPUs. The part that's missing is the part that could make it, potentially, a candidate for the kind of introduction to GPU programming that would make it worth reading to expand your general understanding of how computers work. If you ever paid attention to how encyclopedic articles are written: the structure of a definition given by encyclopedia often has two components. First puts the object of the definition into the more general category, second explains how the object of the definition is different from other elements of the category. What the GPU article is missing is the first component: putting GPU programming into the more general category. This, in practical terms, means that questions like "is DPU programming anything like GPU programming?" or "can `smart' SSDs (with FPGAs) be treated similar to GPUs?" unanswered.
- deleted 3y ago
  [deleted]
- nativeit 3y ago
  I can appreciate what you’re getting at in mourning the absence of greater opportunities for in-depth learning, but I personally value and appreciate the learning process such that I am overwhelmed by gratitude that I will likely never be without something to hold my interest. I have a deep understanding of the things I use in my daily work, but I think holding a breadth of knowledge is also useful in that you have a higher appreciation for what other specialists know and do, and in the event you need something novel, you may have a head start in getting to the knowledge you need at the time. I view it as unreservedly positive to audit many subjects even if you cannot engage with them further.
- Agingcoder 3y ago
  I’d say very few actually ( and I say this from the perspective of someone who used to work in hpc ). If they need to know about hw, most devs need to know about their primary platform, ie cpu. Gpus for general purpose computing ( I’m deliberately excluding games here, and even then it’s not obvious ) and programmed by people who don’t write ml/hpc libraries are far from ubiquitous. Yes, you want to know as much as possible ( helps debugging/zooming in on issues since you don’t need to introduce an outsider to your problem, helps avoid errors), yes you need to specialize somewhere, no you can’t know everything and often don’t need to
cebert 3y ago
> Most programmers have an intimate understanding of CPUs and sequential programming because they grow up writing code for the CPU Maybe it’s just where I work, but I feel like even this isn’t true. A lot of the newer/young employees don’t even seem to have a lot of OS/system understanding. A lot of the higher-level languages abstract the immediate need of having any “intimate understanding” of CPUs.
- Rizu 3y ago
  Do you have any recommendations to learn CPUs and Systems well ?, thanks
  - mackrevinack 3y ago
    not sure if this is what you have in mind but Code by Charles Petzold might be worth checking out
  - United857 3y ago
    The Patterson Hennessy computer architecture textbook is the classic and still the best IMO.
  - mavelikara 3y ago
    Not the GP, but I recommend reading (or watching lectures based on) the book “Computer Systems: A Programer’s Perspective.”
  - jocaal 3y ago
    Onur Mutlu puts his computer architecture lectures on youtube, their really good.
- dr_kiszonka 3y ago
  It's not just where you work, in my experience.
- abhi9u 3y ago
  True. It's becoming a lost art. Although, I think anyone who takes a CS program probably learns enough about CPUs in a Computer Organization or Computer Architecture course. Even if they never have to use that knowledge ever again. :) On the other hand I agree that people who take the non-CS route may not be that intimately familiar with all the details.
mannyv 3y ago
Now I understand why ML uses floats for precision. It wasn't a choice, it was because graphics code uses them. Another piece in the "why is ML so inefficient" puzzle! I wonder what that memory copying overhead is IRL. If it's like normal stuff it'll be brutal. I mean, they offload tcp processing into hardware to avoid that. This is way more data, though it is done in bigger chunks.
- choppaface 3y ago
  For a lot of larger modern networks the GPU compute time in computing gradients and doing the backwards pass is so slow that copying float data over the pcie bus is no bottleneck. I.e. copying a minibatch of float images is still plenty fast because the gradients / SGD iteration is so slow and requires so much compute (even with mixed precision). For shallower networks, there can be an advantage to copying just the original compressed data to GPU memory and then doing decompression etc there. But modern GPUs haven’t adopted pcie 5 yet because the raw compute is more important. Lastly tensor cores have had a big impact, depending on the network they can be fast enough to be very under-utilized.
  - touisteur 3y ago
    I would say that some one modern GPU has adopted PCIe 5.0, the NVIDIA H100 - if you want more (FP32) 'compute' (and no HBM...) in one GPU your sourceable choice is the L40 which didn't get PCIe 5.0. This feels like another market/product segmentation (that and L40 didn't get nvlink).
- oivey 3y ago
  I don’t think the choice of using floating point numbers is particularly inefficient. If frameworks were fixed point by default it would be tricky to get the dynamic ranges right all the way through a network. The math in the training assumes the numbers are continuous, too.
- Dylan16807 3y ago
  What's inefficient about floats? ML seems to get a big benefit out of having access to multiple/many orders of magnitude.
Const-me 3y ago
> During execution the registers allocated to a thread are private to it, i.e., other threads cannot read/write those registers. Wave intrinsics in HLSL, and similar CUDA things can read registers from different threads within the current wavefront. Also, in the paragraph about memory architecture, I would mention the caches provide no coherency guarantees across threads of the same dispatch/grid, but there’s a special functional block global to the complete chips which implements atomics on global memory.
toppy 3y ago
Why are they still called GPU? PPU (Parallel Processing Unit) sounds like a better name.
- croes 3y ago
  CPUs are PPUs too
  - rational_indian 3y ago
    A better name for GPUs would then be "Massively parallel processing units" or MPPUs.
  - toppy 3y ago
    Touche!
- speed_spread 3y ago
  Vector Processing Unit would be more appropriate.
  - KeplerBoy 3y ago
    Why stop there? Matrix Processing Unit or Tensor Processing Unit if you want to trigger the physicists or are a fan of the google cloud.
    - nologic01 3y ago
      Differential Geometric Processing Unit would do it for me.
    - speed_spread 3y ago
      Those things actually exists and are called NPU (Neural Processing Unit). Many recent ARM CPU have them (Apple, Qualcomm, etc.) and even new RISC-V CPU. They're different from GPU in being even more constrained in programming requiring usage of a fixed function vendor-supplied library.
    - 3y ago
01100011 3y ago
> Copying Data from Host to Device Surprised there's no mention of async copies here. If you want to get the most out of the GPU, you don't want it idle when copying data between the host and the GPU. Many frameworks provide for a mechanism to schedule async copies which can execute along side async work submission. The post is sort of GPU 101 but there's a whole world of tricks and techniques beyond that once you start doing real-world GPU programming where you want to squeeze as much out of the expensive GPU as possible. Profiling tools help a lot here because, like much of optimizing now, there are hidden cliffs and non-linearities all over that you have to be aware of.
- nine_k 3y ago
  Since you likely use 64-bit (double) floats, not every GPU would help much, especially compared to a beefy CPU. But if you use a GPU with a large number of FP64 units, it may speed things up a lot. These are generally not gaming GPUs, but if you have a 4060 sitting around anyway, it has about 300 GFLOPS FP64 performance, likely more than your CPU. Modern CPUs are mighty in this regard though, able to issue many FP64 operations per clock per core.
  - 01100011 3y ago
    Did you reply to the wrong comment?
permo-w 3y ago
>Most programmers have an intimate understanding of CPUs maybe this article is brilliant, but when the first line is something so blatantly untrue it really makes it hard to take the rest seriously
- anonporridge 3y ago
  Definitely not true about most programmers, but maybe the author meant CS educated engineers. Going through a formal CS program will give you an intimate understanding of CPUs, especially when compared to the very light coverage of GPUs.
  - croes 3y ago
    Maybe a generation thing, I remember reading Michael Abrash's Graphics Programming Black Book and learning a lot about cycle eaters in the CPU.
  - usea 3y ago
    > Going through a formal CS program will give you an intimate understanding of CPUs I did an undergrad in CS, where I did well. I don't feel like I understand CPUs very well. Certainly not anywhere in the realm of "intimate."
  - galangalalgol 3y ago
    Are most programmers self trained now? I can see if someone self trains for fe or even full stack with an eye on compensation, they wouldn't understand program counters and the like. But so many people seem motivated by video games to get into the industry that I'd expect them to be reading about things like the fast inverse square root or similar.
    - crooked-v 3y ago
      A lot of game dev now happens in high-level engines, often with their own extra layer of scripting engine. Even working on the nitty-gritty of spatial logic rather than using engine-based colliders/detection would be pretty rare.
zackmorris 3y ago
This is a great writeup. And GPUs are more advanced/performant for what they do than anything I could ever come up with. But I put SIMD in the category of something that isn't necessary once one has learned other (more flexible) paradigms. I prefer MIMD and clusters/transputers, which seem to have died out by the 2000s. Today's status quo puts the onus on developers to move data manually, write shaders under arbitrary limitations on how many memory locations can be accessed simultaneously, duplicate their work with separate languages for GPU/CPU, know if various hardware is available for stuff like ray tracing, get locked into opinionated frameworks like OpenGL/Metal/Vulkan, etc etc etc. GPUs are on a side tangent that can never get me to where I want to go, so my experience over the last 25 years has been of a person living on the wrong timeline. I've commented about it extensively but it just feels like yelling into the void now. Loosely, a scalable general purpose CPU working within the limitations of the end of Moore's law is multicore with local memories, sharing data through a copy-on-write content-addressable memory or other caching scheme which presents a single unified address space to allow the user to freely explore all methods of computation in a desktop computing setting. It uses standard assembly language but is usually programmed with something higher level like Erlang/Go, Octave/MATLAB or ideally a functional programming language like Julia. 3D rendering and AI libraries are written as a layer above that, they aren't fundamental. It's interesting that GPUs have arrived at roughly the multicore configuration that I spoke of, but with drivers that separate the user from the bare-metal access needed to do general purpose MIMD. I had thought that FPGAs were the only way to topple GPU dominance, but maybe there is an opportunity here to write a driver that presents GPU hardware as MIMD with a unified memory. I don't know how well GPU cores handle integer math, but that could be approximated with the 32 bit int portion of a 64 bit float. Those sorts of tradeoffs may result in a MIMD machine running 10-100 times slower than a GPU, but still 10-100 times faster than a CPU. But scalable without the over-reliance on large caches and fast busses which stagnated CPUs since around 2007 when affordability and power efficiency took priority over performance due to the mobile market taking over. And MIMD machines can be clustered and form distributed compute networks like SETI@home with no changes to the code. To get a sense of how empowering that could be to the average user: it's like comparing BitTorrent to FTP, but for compute instead of data.
paulddraper 3y ago
SIMD programming is f---ing wild. Want to run a calculation for every pixel on your screen? No problem. Want to have a branching condition? Ouchie.
- eimrine 3y ago
  Want to have eval? Stop everything.
  - touisteur 3y ago
    Vectorized emulation is very interesting and fun, Brandon Falk (gamozolabs) has this series of hours-long streams and rust projects, you can start here: https://news.ycombinator.com/item?id=18222729 https://news.ycombinator.com/item?id=18222729
- winwang 3y ago
  To be fair, this makes sense: making a smart decision is "harder" than scaling a simple calculation out to a bunch of workers.
jokoon 3y ago
I wish it was easier to program a GPU... I've already refrained myself to learn vulkan because it scares me, but similarly, opengl and cuda are a bit mysterious to me, and I don't really know how I could take advantage of it, since most computing tasks cannot be made parallel. I've read there are data structures that are somehow able to take advantage of a GPU as an alternative to the CPU (for example a database running on a GPU), but it seems a very new domain, and I don't have the skill to explore it.
- HalfCrimp 3y ago
  If you are comfortable with C++ already then look at Thrust. It's nvidia's analogue to the standard library arrived at GPU computing. Writing and launching raw cuda kernels is too low level for me, but writing with Thrust makes it feel pretty similar to writing regular C++ code. You still need to deal with moving data from host to device and back, but that's as simple as assigning a `thrust::device_vector` to a `thrust::host_vector`
  - rnrn 3y ago
    or look at using standard C++ with an implementation that uses the GPU: https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algorithms/index.html https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algor...
nologic01 3y ago
It is anybody's guess what the future will bring, but on past form GPU programming will remain a niche for specific highly-tuned (HPC) applications and mere mortals can focus on somewhat easier multi-core CPU programming instead. The main reason GPU gets so much attention these days is that CPU manufacturers (Intel in particular) simply cannot get their act together. Intel had promised significant breakthroughs with Xeon Phi like a decade ago. In the meantime people have invented more and more applications that need significant computational power. But it will eventually get there. E.g., AMD' latest epyc features 96 cores. Importantly, that computational power is available in principle with simpler / more familiar programming models.
natanye 3y ago
[flagged]
m3kw9 3y ago
Never really needed to know this stuff because compiler does all the translation, but low level programming like training AI would be useful. Is that a correct assumption?
MrRolleyes 3y ago
TL;DR most developers don't need to know any of this.
osigurdson 3y ago
>> Most programmers have an intimate understanding of CPUs I'd say the mental model for most programmers is: lines of text, in their language of choice, zipping by really fast.
- quickthrower2 3y ago
  Like an interpreted language I suppose
- jacquesm 3y ago
  By the time you're looking at using GPUs my assumption would be that you've left that particular mental model behind long ago.
qwertox 3y ago
Let's assume I have an array of 10.000 lat/lng-pairs. I want to compute the length of the track. I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. Then I use a vectorized haversine algorithm on these arrays to obtain a third one with the distances between each "row" of the two arrays. With NumPy this is fast, but I guess that a GPU could also perform this and likely do it much faster. Would it be worth it if one has to consider that the data needs to be moved to the GPU and the result be retrieved?
- rational_indian 3y ago
  You need to try it and see if it is any faster. NVIDIA has a drop-in replacement for numpy: https://developer.nvidia.com/cunumeric https://developer.nvidia.com/cunumeric.
- dataflow 3y ago
  Not the answer to your question, but: > I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. I assume/hope this is only what you're doing logically, not physically? Otherwise you might as well just compute the length using n - 1 points from the existing array, then do the remaining portion manually and add it to the existing sum. That would avoid the copying of the whole array.
- dist-epoch 3y ago
  > Would it be worth it if one has to consider that the data needs to be moved to the GPU and the result be retrieved Depends on your batch size. If the computation on the CPU is less than lets say 200 ms it's probably not worth it. Also consider that integrated GPUs don't have separate memory, I'm not sure, but they might not have a high cost of moving data to memory.
- tlb 3y ago
  Probably not. The computation is only a few trig instructions per array element, so most of the time is moving data on either CPU or GPU.
godelski 3y ago
I thought I'd share something with my experience with HPC that applies to many areas, especially in the rise of GPUs. The main bottleneck isn't compute, it is memory. If you go to talks you're gonna see lots of figures like this one[0] (typically also showing disk speeds, which are crazy small). Compute is increasing so fast that at this point we finish our operations long faster than it takes to save those simulations or even create the visualizations and put on disk. There's a lot of research going into this, with a lot of things like in situ computing (asynchronous operations, often pushing to a different machine, but needing many things like flash buffers. See ADIOS[1] as an example software). What I'm getting at here is that we're at a point where we have to think about that IO bottleneck, even for non-high performance systems. I work in ML now, which we typically think of as compute bound, but being in the generative space there are still many things where the IO bottlenecks. This can be loading batches into memory, writing results to disk, or communication between distributed processes. It's one beg reason we typically want to maximize memory usage (large batches). There's a lot of low hanging fruit in these areas that aren't going to be generally publishable works but are going to have lots of high impact. Just look at things like LLaMA CPP[2], where in the process they've really decreased the compute time and memory load. There's also projects like TinyLLaMa[3] who are exploring training a 1B model and doing so on limited compute, and are getting pretty good results. But I'll tell you from personal experience, small models and limited compute experience doesn't make for good papers (my most cited work did this and has never been published, gotten many rejections for not competing with models 100x it's size, but is also quite popular in the general scientific community who work with limited compute). Wfiw, companies that are working on applications do value these things, but it is also noise in the community that's hard to parse. Idk how we can do better as a community to not get trapped in these hype cycles, because real engineering has a lot of these aspects too, and they should be (but aren't) really good areas for academics to be working in. Scale isn't everything in research, and there's a lot of different problems out there that are extremely important but many are blind to. And one final comment, there's lots of code that is used over and over that are not remotely optimized and can be >100x faster. Just gotta slow down and write good code. The move fast and break things method is great for getting moving but the debt compounds. It's just debt is less visible, but there's so much money being wasted from writing bad code (and LLMs are only going to amplify this. They were trained on bad code after all). And no, pytorch isn't going to optimize everything for you automagically (no one can. Optimization is often situationally dependent). You're still gonna need to understand the distributed package and it is worth learning MPI. [0] https://drivenets.com/wp-content/uploads/2023/05/blog-networking-bottlenecks-part2-full-1536x803.png https://drivenets.com/wp-content/uploads/2023/05/blog-networ... [1] https://github.com/ornladios/ADIOS2 https://github.com/ornladios/ADIOS2 [2] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp [3] https://github.com/jzhang38/TinyLlama https://github.com/jzhang38/TinyLlama
einpoklum 3y ago
> Most programmers have an intimate understanding of CPUs Not at all. Very few programmers have an intimate understanding of CPUs. I'm not one of them, for example. And that's without mentioning how most programmers actually write Excel scripts, and most of the rest write interpreted code like Javascript or Python which is somewhat removed from the CPU. > because they grow up writing code for the CPU There is not really such thing as "The CPU". The design space of CPUs is large enough for it to contain quite a variation of beasts. For example, > CPUs were designed to execute sequential instructions Originally, yes, but - one cannot really make this argument about CPUs since Dennard scaling hit the wall, and scale-out began in earnest. ----- While the mental model is indeed different, hardware-wise, you should think of GPUs as CPUs, except that... * You only ever use SIMD registers: 32 lanes x 32bit * Large register file (up to 256 SIMD registers) * No branch prediction * Predicated execution supported, but no if-then-else (sort of). * Very little stack to speak of * ~10x L1 cache latency * A piece of L1 "cache" which doesn't cache anything and you can just write to it like regular memory. * ~100x L2 cache latency, no coherence, and it's smaller * ~100x main memory latency (and no L3) * A bunch of special registers for implementing geometric thread & lane indexing. * ~10x memory bandwidth * ~5x-10x cores than on a CPU (numbers are for NVIDIA cards, AMD are slightly different; and of course individual cards differ etc.) ----- Now take all of those hardware differences, and conceive a programming model. What you get is the fanciful tale of a "not-like-a-CPU" processor. Programming for the CPU, we often do the opposite, i.e. ignore the cores we have, ignore things like latency hiding, careful management of what's in registers and what's in the cache, and just write naive serial programs.
fatih-erikli 3y ago
I think GPU computing should not be done in application layer. It's way too low-level.
- jacquesm 3y ago
  That's the only layer where it makes sense because that's where you know what it is that you are trying to achieve. The overhead in GPU programming is such that if you make one small assumption that doesn't hold true in practice you may end up sinking your performance in a terrible way. So you need a lot of control over where and how things are laid out. For more generic stuff there are libraries, but those too run at the behest of your application. As this technology matures you'll see more and more abstraction and automation of the parts that squeeze out the most performance. But for now that's where you can make the biggest gains, just like any other kind of special purpose co-processor.
  - deleted 3y ago
    [deleted]
  - fatih-erikli 3y ago
    Makes sense.
fancyfredbot 3y ago
This article is pretty good but, looking at comments so far, nobody seems to have made the obvious/predictable point that it's terribly Nvidia specific. That would be understandable perhaps a few years ago. But the era where there was no reasonable alternative is over. Sycl is a good language which performs well across multiple hardware vendors. Sapphire rapids is very good hardware. AMD's MI300 looks amazing. Nvidia has run out of GPUs. It's finally time for GPU development to stop meaning CUDA, but I guess it's going to take the world a while to realise this.
- jacquesm 3y ago
  At this point the massive investment in software is what drives this, hardware differences may no longer be dominant but the only way to unseat NV at this point is drop in replacements and/or drop in replacement libraries. And that's getting there. Which is good because GPUs are too expensive and have too little memory, some competition might help move things along rather than to give NV more time to milk their precious digital cows.
  - tormeh 3y ago
    I really wish the ML researchers would have stayed with Vulkan or OpenCL instead of standardising on CUDA. Everyone must have known how it would end.
    - behnamoh 3y ago
      Those ml researchers wanted to get hired…
      - p1esk 3y ago
        Those ML researchers (myself included) wanted to get the job done.
    - jacquesm 3y ago
      Sometimes people just have a job to do rather than to play politics and in this case doing it with OpenCL or Vulkan would have ceded the market for their application to the competition due the difference in speed and money spent for a given amount of compute. That's starting to change but the competition has reacted much too late. NVidia figured out that software is almost as important as the hardware early enough that betting on that with a sizeable dedicated team gave them a tremendous lead. I've used CUDA and have tried to use OpenCL but the number of hurdles I had to jump through with OpenCL made it a non-starter and that was before factoring in the cost. NV made it easy and made it work right out of the box. Quite often the most visible constraint is developer time and reducing developer friction is a great way to capture a larger slice of the market as well as to actually grow that market.
diimdeep 3y ago
Also check out this talk and slides from few years ago about CPU and GPU nitpicks Alexander Titov — Know your hardware: CPU memory hierarchy https://youtu.be/QOJ2hsop6hM https://youtu.be/QOJ2hsop6hM https://github.com/alexander-titov/public/blob/master/conferences/cpp-meetup-march-2019/ https://github.com/alexander-titov/public/blob/master/confer... Know Your Hardware - CPU Memory Hierarchy -- Alexander Titov -- C%2B%2B Moscow Meetup March 2019.pdf https://github.com/alexander-titov/public/blob/master/conferences/corehard-2019/ https://github.com/alexander-titov/public/blob/master/confer... GPGPU - what it is and why you should care -- Alexander Titov -- CoreHard 2019.pdf
RevEng 3y ago
This is one of the better explanations I've seen of how GPU programming works. I'll be using this for my mentees in the future. Well done!
kdwikzncba 3y ago
> We can understand this with the help of Little’s law from queuing theory. It states that the average number of requests in the system (Qd for queue depth) is equal to the average arrival rate of requests (throughput T) multiplied by the average amount of time to serve a request (latency L). First off, this is obviously false. If you can serve 9req/s and you're getting 10req/s the size of the queue depth is growing at a rate of 1req/s. It's not stationary. Second, what's the connection between this and gpus? What's the queue? What's the queue depth? What are the requests? Seems to me that the article focuses more on being smart than actually learning.
- minitoar 3y ago
  The scenario is that you’re calculating Qd given a static average latency. Absent that, this formula doesn’t give you a way to compute Qd. What is the average amount of time to service a request in a system where the queue depth is growing without bound?
- ssivark 3y ago
  > First off, this is obviously false. If you can serve 9req/s and you're getting 10req/s the size of the queue depth is growing at a rate of 1req/s. It's not stationary. I haven’t formally studied any queuing theory, but I think: 1. The rule assumes you have enough processing power to service the average load (otherwise it fails catastrophically like you mentioned) 2. The rule is trying to model the fluctuations in the pending load (which might determine wait time or whatever else).
rakkhi 3y ago
Imagine with NVIDIA banned in china, how well the Chinese local companies will do in GPU's for AI: https://x.com/BeijingDai/status/1715861773495279743?s=20 https://x.com/BeijingDai/status/1715861773495279743?s=20
- kevingadd 3y ago
  It's going to take them a while to catch up to CUDA, though. Even with stolen IP it's going to be tough to make 1:1 drop-in replacements. The amount of existing investment into AMD and NVIDIA architectures is huge, as evidenced by how bad the Moore Threads GPUs are - it simply isn't easy to enter the market as a serious competitor. Even Intel is struggling.
guidedlight 3y ago
One thing I don’t understand is how the architecture of Apple Silicon is different from NVidia’s. Looking at this quote: > the Nvidia H100 GPU has 132 SMs with 64 cores per SM, totalling a whopping 8448 cores. 8448 cores sure sounds impressive. But the Apple M2 Ultra only has 76 cores?! How can the NVidia H100 GPU have over 110x more cores? Clearly it doesn’t have 110x more performance over the M2 Ultra, so what is going on here?
- kevingadd 3y ago
  NVIDIA's SMs are most comparable to the 'CUs' on AMD GPUs or cores on Apple GPUs, generally speaking. The "cores" are subsets of the SM that perform individual operations, IIRC. See this diagram from an nvidia blog post: https://developer-blogs.nvidia.com/wp-content/uploads/2021/guc/raD52-V3yZtQ3WzOE0Cvzvt8icgGHKXPpN2PS_5MMyZLJrVxgMtLN4r2S2kp5jYI9zrA2e0Y8vAfpZia669pbIog2U9ZKdJmQ8oSBjof6gc4IrhmorT2Rr-YopMlOf1aoU3tbn5Q.png https://developer-blogs.nvidia.com/wp-content/uploads/2021/g... ( https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/ https://developer.nvidia.com/blog/nvidia-ampere-architecture... )
- subharmonicon 3y ago
  NVIDIA is intentionally being obtuse and frankly dishonest calling what’s effectively a vector lane a “core” and similarly using “thread” in “SIMT” to mean the execution of one of those vector lanes. Yes, their architecture is different from many in that they support a separate program counter per lane (which is why they feel justified in calling this a “thread”), but ultimately it’s the rate and throughput of ALUs that matter.
  - freeone3000 3y ago
    It’s not even a seperate PC per lane, you only get that per block — lane level execution goes to an execution mask lut per insn. Branchy code that’s not branchy uniformly in the block executes a lot of noops.
    - sakras 3y ago
      As of Volta, they have independent PCs with a warp optimizer that dynamically groups threads with the same program counter, so branches aren’t nearly as bad as they used to be.
markhahn 3y ago
it's not just nvidia-specific, but nvidia-biased. the misuse of "core", pretending that CPUs are backwards, etc. every developer should know about more than nvidia's spin.
dang 3y ago
Someone emailed to complain about this: https://twitter.com/abhi9u/status/1715753871564476597 https://twitter.com/abhi9u/status/1715753871564476597 That is against HN's rules. In fact, it's the one thing that's important enough to be in both the site guidelines and FAQ. HN users feel extremely strongly about this. Q: Can I ask people to upvote my submission? A: No. Users should vote for a story because they personally find it intellectually interesting, not because someone has content to promote. We penalize or ban submissions, accounts, and sites that break this rule, so please don't. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html Don't solicit upvotes, comments, or submissions. Users should vote and comment when they run across something they personally find interesting—not for promotion. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- abhi9u 3y ago
  Sorry, I was not aware of this rule. Although it was submitted by someone I don't know. But I won't do it again now that I know.

465 points by Anon84 3y ago

Thank you for sharing!

Not every developer. I'm not trying to be snarky. I think there's an unhelpful compunction to want to know everything about everything among STEM types like programmers (of which I am one). Specialization is fundamental to the success, not just of whole economies, but of the individuals in them. It can feel like a paintful sacrifice to admit that you'll never (have time to) learn, say, the entire Python language specification, or how type inference works, or any number of other things someone might tell you is critical knowledge. But it's often liberating, and more often than that, mandatory. (Maybe I was trying to be a little snarky initially, but I'm not any more.)

Jiro 3y ago
Over-specialization is hard on your ability to find another job if you lose your current one.
- Jeff_Brown 3y ago
  Completely true. The analogy to hydration holds, I think -- too much and too little can both be problematic. But I see seem to run into much more under-specialization than over-specialization. More than 90% of the wildest successes I know went really deep on something pretty narrow.
  - HPsquared 3y ago
    Specialization is like putting everything on one number in roulette. It's great if you win (i.e. your specialty is in demand), but you're more likely to lose everything you invested.
    - Jeff_Brown 3y ago
      It is a risk, but not as bad as you suggest, because (1) in work you can get an incremental sense of what is worth specializing in, and (2) there's always some degree to which skills are transferrable and experience is impressive even to someone who doesn't need you to do what you did again.
- andrewSC 3y ago
  And so it’s a balance :^) I personally take the approach of specializing in what I believe to be my natural talents (i.e. seemingly able to pick up X much easier than whoever) while just being generally aware of new technologies or other spaces/sectors. Also tinkering and/or having a general interest in tech helps lol.. IME doing this has helped me pivot into a new tech or other areas where I may not be specialized in, but may need to eventually be for a new role. Also worth mentioning is taking a pragmatic approach to problem solving in general.. I’ve personally found that if you’re able to demonstrate solid reasoning and/or problem solving, generally learning a new lang or specializing in something different than what you’re used to, isn’t too too bad. I’m not really sure how specialists outside of tech/in other sectors can transition into other roles “easily” though… hm. Edit: I wanted to mention that IME in STEM, most? More often than not? are goal oriented… It is _completely_ okay to not have any “working” thing at the end of your tinkering/learning… The journey into that “thing” can be a learning experience in and of itself. I’ve often started to learn things but after a certain point have told myself “I’m gonna stop here and that’s okay. I don’t need to have a solid understanding of this, at this time”. YMMV…
nabla9 3y ago
This is just minimum for those don't need to know details. Even general knowledge has depth. Even if you are generalist, or specialist in different area you should deepen your knowledge at every area gradually.
- abhi9u 3y ago
  True. This is probably what first few lectures of a first course on GPU computing would cover. The nitty gritty of how to write parallel algorithms for maximum throughput on the GPU is where things become too specific. :)
cratermoon 3y ago
I dunno. The section "Latency Tolerance, High Throughput and Little’s Law" has many applications in programming. Ever need to scale your cache, or size a connection pool? Little's Law.
- abhi9u 3y ago
  Thank you for reading! :)
  - cratermoon 3y ago
    Thanks for writing this. I note you mention that GPUs are massively parallel, and I think it would strengthen the article to add a paragraph or two discussing what kinds of loads lend themselves to GPU computing. The sort of "embarrassingly parallel" things like linear algebra vs inherently serial example of numerical analysis.
    - abhi9u 3y ago
      Yes, that's a good feedback. Few other people also mentioned this. I was trying to save space, to spend time more on the "how things work" part. But I see your point that some motivation behind parallelization would make it a more interesting read.
yodsanklai 3y ago
> It can feel like a paintful sacrifice to admit that you'll never (have time to) learn And even if you do have time, you'll probably forget most of it if you don't make use of it in your daily job. Also if you do want to learn about a new topic, it'll take commitment. I remember going through an MIT OS project, it took me something like 50 hours of hours to complete the project. Pretty much impossible when you have a full time job (I didn't at the time). And despite that, I still consider myself a newbie in OS development. That being said, a little extra knowledge can come handy and make a difference in an interview for instance, or reduce ramp-up time when changing teams. This is also what school is for. Give you full time and a structured program to pick the fundamentals. There's only so much you can learn once you have a demanding job. It's actually pretty sad we don't get to go back to school in the middle of our career. Edit: still an interesting article ;)
- abhi9u 3y ago
  Thank you :)
- owl57 3y ago
  > I remember going through an MIT OS project, it took me something like 50 hours of hours to complete the project. Pretty much impossible when you have a full time job (I didn't at the time). And despite that, I still consider myself a newbie in OS development. I took an OS course loosely based on MIT one, and of course wouldn't consider myself an experienced OS developer as well, but I think the residual knowledge is sometimes really helpful when debugging bad application performance at work.
pjmlp 3y ago
Indeed, one thing that most seniors learn is humility and being able to say I don't know, without caring about consequences.
- bobwaycott 3y ago
  An invaluable life skill, as well.
loeg 3y ago
At this point, I think the running shtick / inside joke of "Every Developer Should Know ..." headlines is that of course every developer doesn't need to know the contents of the article that follows.
- bachmeier 3y ago
  Not true. Anyone experienced in programming discussions knows that "software development" overlaps exactly with the programming done by the commenter, and everything else is solving toy problems with tools that don't scale.
- wongarsu 3y ago
  > of course every developer doesn't need to know the contents of the article that follows. Things every developer should know about English, part 1: why word order matters in negation /s Obviously you meant "not every developer needs to know" instead of "every developer doesn't need to know", but I see this switch so often lately that I'm beginning to wonder if it's a dialect thing (similar to double negative implying a positive to some and an emphasis of the negative to others)
  - deleted 3y ago
    [deleted]
  - a1369209993 3y ago
    Actually, that seems to be a legitimate precedence ambiguity. The probably more common way to parse/interpret it is: > [Foreach developer D, D] needs to know this. > [Foreach developer D, D] doesn't need to know this. (No developers.) But you could also have: > [The logical conjuction of all developers[0]] needs to know this. > [The logical conjuction of all developers] doesn't need to know this. (Not all developers.) The former convention is clearly better unless I've missed something rather significant, but inherently broken features in a number of programming languages (eg perl 6 'junctive' operators) suggest the latter is well-established as a thing that exists. 0: ie, a linguistic fiction that has only those properties / needs only those things that are common to every developer.
- 3y ago
cj 3y ago
One thing I find interesting about the software industry is our lack of descriptive job titles. At big companies you have SWE L1, L2, L3, senior, staff, principal, etc. You also have SRE and maybe some devops or architect roles. Smaller companies you have lots of people generic “engineer” titles or “full stack engineers”, etc. Why don’t we encode people’s specialties in their titles if most engineers are working on narrow sections of software? E.g. “React & Node Developer” instead of “Full stack engineer” … etc I suppose the easiest rationale is generic titles allow for easier mobility between disciplines.
crabbone 3y ago
You are interpreting the title literally, while all this article is trying to do is to give an introduction to anyone who wants to get into GPU programming. Now, about every developer. Ideally, developers should know something, in general, about all fields related to programming. Similar to how in medicine you need to learn about different branches of medicine, even though you'll specialize in one, and so it is in mathematics, physics and so on. Presently, the demand for quality professionals in programming is very low. There aren't any good testing or certification programs that can tell a good programmer from a bad one. The industry is generally happy with "specialists" who perhaps only know to do one thing, somewhat. So, presently you don't need to know anything about GPU or any other field that's not directly related to your job description. ---- Now, about the article itself. While it gives a lot of valuable factual information, it's missing the forest for the trees. It's very dedicated to how CUDA works or some other particular aspects of NVidia's GPUs. The part that's missing is the part that could make it, potentially, a candidate for the kind of introduction to GPU programming that would make it worth reading to expand your general understanding of how computers work. If you ever paid attention to how encyclopedic articles are written: the structure of a definition given by encyclopedia often has two components. First puts the object of the definition into the more general category, second explains how the object of the definition is different from other elements of the category. What the GPU article is missing is the first component: putting GPU programming into the more general category. This, in practical terms, means that questions like "is DPU programming anything like GPU programming?" or "can `smart' SSDs (with FPGAs) be treated similar to GPUs?" unanswered.
deleted 3y ago
[deleted]
nativeit 3y ago
I can appreciate what you’re getting at in mourning the absence of greater opportunities for in-depth learning, but I personally value and appreciate the learning process such that I am overwhelmed by gratitude that I will likely never be without something to hold my interest. I have a deep understanding of the things I use in my daily work, but I think holding a breadth of knowledge is also useful in that you have a higher appreciation for what other specialists know and do, and in the event you need something novel, you may have a head start in getting to the knowledge you need at the time. I view it as unreservedly positive to audit many subjects even if you cannot engage with them further.
Agingcoder 3y ago
I’d say very few actually ( and I say this from the perspective of someone who used to work in hpc ). If they need to know about hw, most devs need to know about their primary platform, ie cpu. Gpus for general purpose computing ( I’m deliberately excluding games here, and even then it’s not obvious ) and programmed by people who don’t write ml/hpc libraries are far from ubiquitous. Yes, you want to know as much as possible ( helps debugging/zooming in on issues since you don’t need to introduce an outsider to your problem, helps avoid errors), yes you need to specialize somewhere, no you can’t know everything and often don’t need to

cebert 3y ago

> Most programmers have an intimate understanding of CPUs and sequential programming because they grow up writing code for the CPU Maybe it’s just where I work, but I feel like even this isn’t true. A lot of the newer/young employees don’t even seem to have a lot of OS/system understanding. A lot of the higher-level languages abstract the immediate need of having any “intimate understanding” of CPUs.

Rizu 3y ago
Do you have any recommendations to learn CPUs and Systems well ?, thanks
- mackrevinack 3y ago
  not sure if this is what you have in mind but Code by Charles Petzold might be worth checking out
- United857 3y ago
  The Patterson Hennessy computer architecture textbook is the classic and still the best IMO.
- mavelikara 3y ago
  Not the GP, but I recommend reading (or watching lectures based on) the book “Computer Systems: A Programer’s Perspective.”
- jocaal 3y ago
  Onur Mutlu puts his computer architecture lectures on youtube, their really good.
dr_kiszonka 3y ago
It's not just where you work, in my experience.
abhi9u 3y ago
True. It's becoming a lost art. Although, I think anyone who takes a CS program probably learns enough about CPUs in a Computer Organization or Computer Architecture course. Even if they never have to use that knowledge ever again. :) On the other hand I agree that people who take the non-CS route may not be that intimately familiar with all the details.

mannyv 3y ago

Now I understand why ML uses floats for precision. It wasn't a choice, it was because graphics code uses them. Another piece in the "why is ML so inefficient" puzzle! I wonder what that memory copying overhead is IRL. If it's like normal stuff it'll be brutal. I mean, they offload tcp processing into hardware to avoid that. This is way more data, though it is done in bigger chunks.

choppaface 3y ago
For a lot of larger modern networks the GPU compute time in computing gradients and doing the backwards pass is so slow that copying float data over the pcie bus is no bottleneck. I.e. copying a minibatch of float images is still plenty fast because the gradients / SGD iteration is so slow and requires so much compute (even with mixed precision). For shallower networks, there can be an advantage to copying just the original compressed data to GPU memory and then doing decompression etc there. But modern GPUs haven’t adopted pcie 5 yet because the raw compute is more important. Lastly tensor cores have had a big impact, depending on the network they can be fast enough to be very under-utilized.
- touisteur 3y ago
  I would say that some one modern GPU has adopted PCIe 5.0, the NVIDIA H100 - if you want more (FP32) 'compute' (and no HBM...) in one GPU your sourceable choice is the L40 which didn't get PCIe 5.0. This feels like another market/product segmentation (that and L40 didn't get nvlink).
oivey 3y ago
I don’t think the choice of using floating point numbers is particularly inefficient. If frameworks were fixed point by default it would be tricky to get the dynamic ranges right all the way through a network. The math in the training assumes the numbers are continuous, too.
Dylan16807 3y ago
What's inefficient about floats? ML seems to get a big benefit out of having access to multiple/many orders of magnitude.

Const-me 3y ago

> During execution the registers allocated to a thread are private to it, i.e., other threads cannot read/write those registers. Wave intrinsics in HLSL, and similar CUDA things can read registers from different threads within the current wavefront. Also, in the paragraph about memory architecture, I would mention the caches provide no coherency guarantees across threads of the same dispatch/grid, but there’s a special functional block global to the complete chips which implements atomics on global memory.

toppy 3y ago

Why are they still called GPU? PPU (Parallel Processing Unit) sounds like a better name.

croes 3y ago
CPUs are PPUs too
- rational_indian 3y ago
  A better name for GPUs would then be "Massively parallel processing units" or MPPUs.
- toppy 3y ago
  Touche!
speed_spread 3y ago
Vector Processing Unit would be more appropriate.
- KeplerBoy 3y ago
  Why stop there? Matrix Processing Unit or Tensor Processing Unit if you want to trigger the physicists or are a fan of the google cloud.
  - nologic01 3y ago
    Differential Geometric Processing Unit would do it for me.
  - speed_spread 3y ago
    Those things actually exists and are called NPU (Neural Processing Unit). Many recent ARM CPU have them (Apple, Qualcomm, etc.) and even new RISC-V CPU. They're different from GPU in being even more constrained in programming requiring usage of a fixed function vendor-supplied library.
  - 3y ago

01100011 3y ago

> Copying Data from Host to Device Surprised there's no mention of async copies here. If you want to get the most out of the GPU, you don't want it idle when copying data between the host and the GPU. Many frameworks provide for a mechanism to schedule async copies which can execute along side async work submission. The post is sort of GPU 101 but there's a whole world of tricks and techniques beyond that once you start doing real-world GPU programming where you want to squeeze as much out of the expensive GPU as possible. Profiling tools help a lot here because, like much of optimizing now, there are hidden cliffs and non-linearities all over that you have to be aware of.

nine_k 3y ago
Since you likely use 64-bit (double) floats, not every GPU would help much, especially compared to a beefy CPU. But if you use a GPU with a large number of FP64 units, it may speed things up a lot. These are generally not gaming GPUs, but if you have a 4060 sitting around anyway, it has about 300 GFLOPS FP64 performance, likely more than your CPU. Modern CPUs are mighty in this regard though, able to issue many FP64 operations per clock per core.
- 01100011 3y ago
  Did you reply to the wrong comment?

permo-w 3y ago

>Most programmers have an intimate understanding of CPUs maybe this article is brilliant, but when the first line is something so blatantly untrue it really makes it hard to take the rest seriously

anonporridge 3y ago
Definitely not true about most programmers, but maybe the author meant CS educated engineers. Going through a formal CS program will give you an intimate understanding of CPUs, especially when compared to the very light coverage of GPUs.
- croes 3y ago
  Maybe a generation thing, I remember reading Michael Abrash's Graphics Programming Black Book and learning a lot about cycle eaters in the CPU.
- usea 3y ago
  > Going through a formal CS program will give you an intimate understanding of CPUs I did an undergrad in CS, where I did well. I don't feel like I understand CPUs very well. Certainly not anywhere in the realm of "intimate."
- galangalalgol 3y ago
  Are most programmers self trained now? I can see if someone self trains for fe or even full stack with an eye on compensation, they wouldn't understand program counters and the like. But so many people seem motivated by video games to get into the industry that I'd expect them to be reading about things like the fast inverse square root or similar.
  - crooked-v 3y ago
    A lot of game dev now happens in high-level engines, often with their own extra layer of scripting engine. Even working on the nitty-gritty of spatial logic rather than using engine-based colliders/detection would be pretty rare.

zackmorris 3y ago

This is a great writeup. And GPUs are more advanced/performant for what they do than anything I could ever come up with. But I put SIMD in the category of something that isn't necessary once one has learned other (more flexible) paradigms. I prefer MIMD and clusters/transputers, which seem to have died out by the 2000s. Today's status quo puts the onus on developers to move data manually, write shaders under arbitrary limitations on how many memory locations can be accessed simultaneously, duplicate their work with separate languages for GPU/CPU, know if various hardware is available for stuff like ray tracing, get locked into opinionated frameworks like OpenGL/Metal/Vulkan, etc etc etc. GPUs are on a side tangent that can never get me to where I want to go, so my experience over the last 25 years has been of a person living on the wrong timeline. I've commented about it extensively but it just feels like yelling into the void now. Loosely, a scalable general purpose CPU working within the limitations of the end of Moore's law is multicore with local memories, sharing data through a copy-on-write content-addressable memory or other caching scheme which presents a single unified address space to allow the user to freely explore all methods of computation in a desktop computing setting. It uses standard assembly language but is usually programmed with something higher level like Erlang/Go, Octave/MATLAB or ideally a functional programming language like Julia. 3D rendering and AI libraries are written as a layer above that, they aren't fundamental. It's interesting that GPUs have arrived at roughly the multicore configuration that I spoke of, but with drivers that separate the user from the bare-metal access needed to do general purpose MIMD. I had thought that FPGAs were the only way to topple GPU dominance, but maybe there is an opportunity here to write a driver that presents GPU hardware as MIMD with a unified memory. I don't know how well GPU cores handle integer math, but that could be approximated with the 32 bit int portion of a 64 bit float. Those sorts of tradeoffs may result in a MIMD machine running 10-100 times slower than a GPU, but still 10-100 times faster than a CPU. But scalable without the over-reliance on large caches and fast busses which stagnated CPUs since around 2007 when affordability and power efficiency took priority over performance due to the mobile market taking over. And MIMD machines can be clustered and form distributed compute networks like SETI@home with no changes to the code. To get a sense of how empowering that could be to the average user: it's like comparing BitTorrent to FTP, but for compute instead of data.

paulddraper 3y ago

SIMD programming is f---ing wild. Want to run a calculation for every pixel on your screen? No problem. Want to have a branching condition? Ouchie.

eimrine 3y ago
Want to have eval? Stop everything.
- touisteur 3y ago
  Vectorized emulation is very interesting and fun, Brandon Falk (gamozolabs) has this series of hours-long streams and rust projects, you can start here: https://news.ycombinator.com/item?id=18222729 https://news.ycombinator.com/item?id=18222729
winwang 3y ago
To be fair, this makes sense: making a smart decision is "harder" than scaling a simple calculation out to a bunch of workers.

jokoon 3y ago

I wish it was easier to program a GPU... I've already refrained myself to learn vulkan because it scares me, but similarly, opengl and cuda are a bit mysterious to me, and I don't really know how I could take advantage of it, since most computing tasks cannot be made parallel. I've read there are data structures that are somehow able to take advantage of a GPU as an alternative to the CPU (for example a database running on a GPU), but it seems a very new domain, and I don't have the skill to explore it.

HalfCrimp 3y ago
If you are comfortable with C++ already then look at Thrust. It's nvidia's analogue to the standard library arrived at GPU computing. Writing and launching raw cuda kernels is too low level for me, but writing with Thrust makes it feel pretty similar to writing regular C++ code. You still need to deal with moving data from host to device and back, but that's as simple as assigning a `thrust::device_vector` to a `thrust::host_vector`
- rnrn 3y ago
  or look at using standard C++ with an implementation that uses the GPU: https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algorithms/index.html https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algor...

nologic01 3y ago

It is anybody's guess what the future will bring, but on past form GPU programming will remain a niche for specific highly-tuned (HPC) applications and mere mortals can focus on somewhat easier multi-core CPU programming instead. The main reason GPU gets so much attention these days is that CPU manufacturers (Intel in particular) simply cannot get their act together. Intel had promised significant breakthroughs with Xeon Phi like a decade ago. In the meantime people have invented more and more applications that need significant computational power. But it will eventually get there. E.g., AMD' latest epyc features 96 cores. Importantly, that computational power is available in principle with simpler / more familiar programming models.

natanye 3y ago

[flagged]

m3kw9 3y ago

Never really needed to know this stuff because compiler does all the translation, but low level programming like training AI would be useful. Is that a correct assumption?

MrRolleyes 3y ago

TL;DR most developers don't need to know any of this.

osigurdson 3y ago

>> Most programmers have an intimate understanding of CPUs I'd say the mental model for most programmers is: lines of text, in their language of choice, zipping by really fast.

quickthrower2 3y ago
Like an interpreted language I suppose
jacquesm 3y ago
By the time you're looking at using GPUs my assumption would be that you've left that particular mental model behind long ago.

qwertox 3y ago

Let's assume I have an array of 10.000 lat/lng-pairs. I want to compute the length of the track. I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. Then I use a vectorized haversine algorithm on these arrays to obtain a third one with the distances between each "row" of the two arrays. With NumPy this is fast, but I guess that a GPU could also perform this and likely do it much faster. Would it be worth it if one has to consider that the data needs to be moved to the GPU and the result be retrieved?

rational_indian 3y ago
You need to try it and see if it is any faster. NVIDIA has a drop-in replacement for numpy: https://developer.nvidia.com/cunumeric https://developer.nvidia.com/cunumeric.
dataflow 3y ago
Not the answer to your question, but: > I duplicate the array and remove the first item in the duplicated array, append the last entry of the original array to the duplicate in order for them to be equal in length. I assume/hope this is only what you're doing logically, not physically? Otherwise you might as well just compute the length using n - 1 points from the existing array, then do the remaining portion manually and add it to the existing sum. That would avoid the copying of the whole array.
dist-epoch 3y ago
> Would it be worth it if one has to consider that the data needs to be moved to the GPU and the result be retrieved Depends on your batch size. If the computation on the CPU is less than lets say 200 ms it's probably not worth it. Also consider that integrated GPUs don't have separate memory, I'm not sure, but they might not have a high cost of moving data to memory.
tlb 3y ago
Probably not. The computation is only a few trig instructions per array element, so most of the time is moving data on either CPU or GPU.

godelski 3y ago

I thought I'd share something with my experience with HPC that applies to many areas, especially in the rise of GPUs. The main bottleneck isn't compute, it is memory. If you go to talks you're gonna see lots of figures like this one[0] (typically also showing disk speeds, which are crazy small). Compute is increasing so fast that at this point we finish our operations long faster than it takes to save those simulations or even create the visualizations and put on disk. There's a lot of research going into this, with a lot of things like in situ computing (asynchronous operations, often pushing to a different machine, but needing many things like flash buffers. See ADIOS[1] as an example software). What I'm getting at here is that we're at a point where we have to think about that IO bottleneck, even for non-high performance systems. I work in ML now, which we typically think of as compute bound, but being in the generative space there are still many things where the IO bottlenecks. This can be loading batches into memory, writing results to disk, or communication between distributed processes. It's one beg reason we typically want to maximize memory usage (large batches). There's a lot of low hanging fruit in these areas that aren't going to be generally publishable works but are going to have lots of high impact. Just look at things like LLaMA CPP[2], where in the process they've really decreased the compute time and memory load. There's also projects like TinyLLaMa[3] who are exploring training a 1B model and doing so on limited compute, and are getting pretty good results. But I'll tell you from personal experience, small models and limited compute experience doesn't make for good papers (my most cited work did this and has never been published, gotten many rejections for not competing with models 100x it's size, but is also quite popular in the general scientific community who work with limited compute). Wfiw, companies that are working on applications do value these things, but it is also noise in the community that's hard to parse. Idk how we can do better as a community to not get trapped in these hype cycles, because real engineering has a lot of these aspects too, and they should be (but aren't) really good areas for academics to be working in. Scale isn't everything in research, and there's a lot of different problems out there that are extremely important but many are blind to. And one final comment, there's lots of code that is used over and over that are not remotely optimized and can be >100x faster. Just gotta slow down and write good code. The move fast and break things method is great for getting moving but the debt compounds. It's just debt is less visible, but there's so much money being wasted from writing bad code (and LLMs are only going to amplify this. They were trained on bad code after all). And no, pytorch isn't going to optimize everything for you automagically (no one can. Optimization is often situationally dependent). You're still gonna need to understand the distributed package and it is worth learning MPI. [0] https://drivenets.com/wp-content/uploads/2023/05/blog-networking-bottlenecks-part2-full-1536x803.png https://drivenets.com/wp-content/uploads/2023/05/blog-networ... [1] https://github.com/ornladios/ADIOS2 https://github.com/ornladios/ADIOS2 [2] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp [3] https://github.com/jzhang38/TinyLlama https://github.com/jzhang38/TinyLlama

einpoklum 3y ago

> Most programmers have an intimate understanding of CPUs Not at all. Very few programmers have an intimate understanding of CPUs. I'm not one of them, for example. And that's without mentioning how most programmers actually write Excel scripts, and most of the rest write interpreted code like Javascript or Python which is somewhat removed from the CPU. > because they grow up writing code for the CPU There is not really such thing as "The CPU". The design space of CPUs is large enough for it to contain quite a variation of beasts. For example, > CPUs were designed to execute sequential instructions Originally, yes, but - one cannot really make this argument about CPUs since Dennard scaling hit the wall, and scale-out began in earnest. ----- While the mental model is indeed different, hardware-wise, you should think of GPUs as CPUs, except that... * You only ever use SIMD registers: 32 lanes x 32bit * Large register file (up to 256 SIMD registers) * No branch prediction * Predicated execution supported, but no if-then-else (sort of). * Very little stack to speak of * ~10x L1 cache latency * A piece of L1 "cache" which doesn't cache anything and you can just write to it like regular memory. * ~100x L2 cache latency, no coherence, and it's smaller * ~100x main memory latency (and no L3) * A bunch of special registers for implementing geometric thread & lane indexing. * ~10x memory bandwidth * ~5x-10x cores than on a CPU (numbers are for NVIDIA cards, AMD are slightly different; and of course individual cards differ etc.) ----- Now take all of those hardware differences, and conceive a programming model. What you get is the fanciful tale of a "not-like-a-CPU" processor. Programming for the CPU, we often do the opposite, i.e. ignore the cores we have, ignore things like latency hiding, careful management of what's in registers and what's in the cache, and just write naive serial programs.

fatih-erikli 3y ago

I think GPU computing should not be done in application layer. It's way too low-level.

jacquesm 3y ago
That's the only layer where it makes sense because that's where you know what it is that you are trying to achieve. The overhead in GPU programming is such that if you make one small assumption that doesn't hold true in practice you may end up sinking your performance in a terrible way. So you need a lot of control over where and how things are laid out. For more generic stuff there are libraries, but those too run at the behest of your application. As this technology matures you'll see more and more abstraction and automation of the parts that squeeze out the most performance. But for now that's where you can make the biggest gains, just like any other kind of special purpose co-processor.
- deleted 3y ago
  [deleted]
- fatih-erikli 3y ago
  Makes sense.

fancyfredbot 3y ago

This article is pretty good but, looking at comments so far, nobody seems to have made the obvious/predictable point that it's terribly Nvidia specific. That would be understandable perhaps a few years ago. But the era where there was no reasonable alternative is over. Sycl is a good language which performs well across multiple hardware vendors. Sapphire rapids is very good hardware. AMD's MI300 looks amazing. Nvidia has run out of GPUs. It's finally time for GPU development to stop meaning CUDA, but I guess it's going to take the world a while to realise this.

jacquesm 3y ago
At this point the massive investment in software is what drives this, hardware differences may no longer be dominant but the only way to unseat NV at this point is drop in replacements and/or drop in replacement libraries. And that's getting there. Which is good because GPUs are too expensive and have too little memory, some competition might help move things along rather than to give NV more time to milk their precious digital cows.
- tormeh 3y ago
  I really wish the ML researchers would have stayed with Vulkan or OpenCL instead of standardising on CUDA. Everyone must have known how it would end.
  - behnamoh 3y ago
    Those ml researchers wanted to get hired…
    - p1esk 3y ago
      Those ML researchers (myself included) wanted to get the job done.
  - jacquesm 3y ago
    Sometimes people just have a job to do rather than to play politics and in this case doing it with OpenCL or Vulkan would have ceded the market for their application to the competition due the difference in speed and money spent for a given amount of compute. That's starting to change but the competition has reacted much too late. NVidia figured out that software is almost as important as the hardware early enough that betting on that with a sizeable dedicated team gave them a tremendous lead. I've used CUDA and have tried to use OpenCL but the number of hurdles I had to jump through with OpenCL made it a non-starter and that was before factoring in the cost. NV made it easy and made it work right out of the box. Quite often the most visible constraint is developer time and reducing developer friction is a great way to capture a larger slice of the market as well as to actually grow that market.

diimdeep 3y ago

Also check out this talk and slides from few years ago about CPU and GPU nitpicks Alexander Titov — Know your hardware: CPU memory hierarchy https://youtu.be/QOJ2hsop6hM https://youtu.be/QOJ2hsop6hM https://github.com/alexander-titov/public/blob/master/conferences/cpp-meetup-march-2019/ https://github.com/alexander-titov/public/blob/master/confer... Know Your Hardware - CPU Memory Hierarchy -- Alexander Titov -- C%2B%2B Moscow Meetup March 2019.pdf https://github.com/alexander-titov/public/blob/master/conferences/corehard-2019/ https://github.com/alexander-titov/public/blob/master/confer... GPGPU - what it is and why you should care -- Alexander Titov -- CoreHard 2019.pdf

RevEng 3y ago

This is one of the better explanations I've seen of how GPU programming works. I'll be using this for my mentees in the future. Well done!

kdwikzncba 3y ago

> We can understand this with the help of Little’s law from queuing theory. It states that the average number of requests in the system (Qd for queue depth) is equal to the average arrival rate of requests (throughput T) multiplied by the average amount of time to serve a request (latency L). First off, this is obviously false. If you can serve 9req/s and you're getting 10req/s the size of the queue depth is growing at a rate of 1req/s. It's not stationary. Second, what's the connection between this and gpus? What's the queue? What's the queue depth? What are the requests? Seems to me that the article focuses more on being smart than actually learning.

minitoar 3y ago
The scenario is that you’re calculating Qd given a static average latency. Absent that, this formula doesn’t give you a way to compute Qd. What is the average amount of time to service a request in a system where the queue depth is growing without bound?
ssivark 3y ago
> First off, this is obviously false. If you can serve 9req/s and you're getting 10req/s the size of the queue depth is growing at a rate of 1req/s. It's not stationary. I haven’t formally studied any queuing theory, but I think: 1. The rule assumes you have enough processing power to service the average load (otherwise it fails catastrophically like you mentioned) 2. The rule is trying to model the fluctuations in the pending load (which might determine wait time or whatever else).

rakkhi 3y ago

Imagine with NVIDIA banned in china, how well the Chinese local companies will do in GPU's for AI: https://x.com/BeijingDai/status/1715861773495279743?s=20 https://x.com/BeijingDai/status/1715861773495279743?s=20

kevingadd 3y ago
It's going to take them a while to catch up to CUDA, though. Even with stolen IP it's going to be tough to make 1:1 drop-in replacements. The amount of existing investment into AMD and NVIDIA architectures is huge, as evidenced by how bad the Moore Threads GPUs are - it simply isn't easy to enter the market as a serious competitor. Even Intel is struggling.

guidedlight 3y ago

One thing I don’t understand is how the architecture of Apple Silicon is different from NVidia’s. Looking at this quote: > the Nvidia H100 GPU has 132 SMs with 64 cores per SM, totalling a whopping 8448 cores. 8448 cores sure sounds impressive. But the Apple M2 Ultra only has 76 cores?! How can the NVidia H100 GPU have over 110x more cores? Clearly it doesn’t have 110x more performance over the M2 Ultra, so what is going on here?

kevingadd 3y ago
NVIDIA's SMs are most comparable to the 'CUs' on AMD GPUs or cores on Apple GPUs, generally speaking. The "cores" are subsets of the SM that perform individual operations, IIRC. See this diagram from an nvidia blog post: https://developer-blogs.nvidia.com/wp-content/uploads/2021/guc/raD52-V3yZtQ3WzOE0Cvzvt8icgGHKXPpN2PS_5MMyZLJrVxgMtLN4r2S2kp5jYI9zrA2e0Y8vAfpZia669pbIog2U9ZKdJmQ8oSBjof6gc4IrhmorT2Rr-YopMlOf1aoU3tbn5Q.png https://developer-blogs.nvidia.com/wp-content/uploads/2021/g... ( https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/ https://developer.nvidia.com/blog/nvidia-ampere-architecture... )
subharmonicon 3y ago
NVIDIA is intentionally being obtuse and frankly dishonest calling what’s effectively a vector lane a “core” and similarly using “thread” in “SIMT” to mean the execution of one of those vector lanes. Yes, their architecture is different from many in that they support a separate program counter per lane (which is why they feel justified in calling this a “thread”), but ultimately it’s the rate and throughput of ALUs that matter.
- freeone3000 3y ago
  It’s not even a seperate PC per lane, you only get that per block — lane level execution goes to an execution mask lut per insn. Branchy code that’s not branchy uniformly in the block executes a lot of noops.
  - sakras 3y ago
    As of Volta, they have independent PCs with a warp optimizer that dynamically groups threads with the same program counter, so branches aren’t nearly as bad as they used to be.

markhahn 3y ago

it's not just nvidia-specific, but nvidia-biased. the misuse of "core", pretending that CPUs are backwards, etc. every developer should know about more than nvidia's spin.

dang 3y ago

Someone emailed to complain about this: https://twitter.com/abhi9u/status/1715753871564476597 https://twitter.com/abhi9u/status/1715753871564476597 That is against HN's rules. In fact, it's the one thing that's important enough to be in both the site guidelines and FAQ. HN users feel extremely strongly about this. Q: Can I ask people to upvote my submission? A: No. Users should vote for a story because they personally find it intellectually interesting, not because someone has content to promote. We penalize or ban submissions, accounts, and sites that break this rule, so please don't. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html Don't solicit upvotes, comments, or submissions. Users should vote and comment when they run across something they personally find interesting—not for promotion. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

abhi9u 3y ago
Sorry, I was not aware of this rule. Although it was submitted by someone I don't know. But I won't do it again now that I know.