9 ms·
Gentle introduction to GPUs inner workings
- pixelpoet 5y agoThis article says that AMD GPUs are vector in nature, but I think that stopped being the case with GCN; before that they had some weird vector stuff with 5 elements or something.
- dragontamer 5y agoWrong. GPUs are definitely vector (GCN in particular being 64 x 32-bit wide vectors). What you're confusing is "Terascale" (aka: the 6xxx series from the 00s), which was VLIW _AND_ SIMD. Which was... a little bit too much going on and very difficult to optimize for. GCN made things way easier and more general purpose. Terascale was theoretically more GFLOPs than the first GCN processors (after all, VLIW unlocks a good amount of performance), but actually utilizing all those VLIW units per clock tick was a nightmare. Its hard enough to write good SIMD code as it is.
- Jasper_ 5y agoEh, it's sort of a naming conflict. There were two dimensions of "vector" -- whether you used XMM-style registers, requiring the compiler to auto-vectorize different operations. The old vector ISAs had things like dot product instructions. Those things are now all gone, because it turns out it's hard to optimize. We in the industry call scalar-for-each-thread ISAs as "scalar ISAs" these days. For instance, Mali describes their transition from a vec4-for-each-thread-based ISA to a scalar-for-each-thread-based ISA as transitioning from "vector to scalar" [0]. [0] Compare 4:26 and 6:29 in the "Mali GPU Family" video on this page https://developer.arm.com/solutions/graphics-and-gaming/arm-mali-gpu-training https://developer.arm.com/solutions/graphics-and-gaming/arm-...
- dragontamer 5y agoConnection Machine / Thinking Machines Corporation use of "scalar-like SIMD" was around in the late 1980s and early 1990s. Look up *Lisp, the parallel language that Thinking Machines Corporation used. You can still find *Lisp manuals today. https://en.wikipedia.org/wiki/*Lisp https://en.wikipedia.org/wiki/*Lisp Intel implemented SIMD using a technique they called SWAR: SIMD within a Register (aka: the 64-bit MMX registers), which eventually evolved into XMM (SSE), YMM (AVX), and ZMM (AVX512). Today's GPUs are programmed using the old 1980s style / Connection Machine *Lisp, which is clearly the source of inspiration for HLSL / GLSL / OpenCL / CUDA / etc. etc. Granted, today's GPUs are still SWAR (GCN's vGPR really is just a 64-wide x 32-bit register). We can see with languages like ispc (Intel's SPMD program compiler), that we can indeed implement a CUDA-like language on top of AVX / XMM registers.*
- Jasper_ 5y agoI don't think any of it was based on *Lisp. Graphics mostly developed independently. And as such, we use different words and different terminology sometimes, like saying "scalar ISA" when we talk about designing ISAs that don't mandate cross-lane interaction in one thread. Sorry! As far as I know, the first paper covering using SIMD for graphics was Pixar's "Channel Processor", or Chap [0], in 1984. This later became one of the core implementation details of their REYES algorithm [1]. By 1989, they had their own RenderMan Shading Language [2], an improved version of Chap, and you can see the similarities from just the snippet at the start of the code. This is where Microsoft took major inspiration from when designing HLSL, and which NVIDIA then started to extend with their own Cg compiler. 3dlabs then copy/pasted this for GLSL. [0] http://www.cs.cmu.edu/afs/cs/academic/class/15869-f11/www/readings/levinthal84_chap.pdf http://www.cs.cmu.edu/afs/cs/academic/class/15869-f11/www/re... [1] https://graphics.pixar.com/library/Reyes/paper.pdf https://graphics.pixar.com/library/Reyes/paper.pdf [2] https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.210.5592&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.21...
- deleted 5y ago[deleted]
- atq2119 5y agoThe terms are definitely overloaded and can differ subtly between vendors. Those "scalar-for-each-thread" instructions are prefixed v_ on AMD's GCN. The "v" doesn't stand for "scalar" ;)
- Jasper_ 5y agoYeah, certainly having "V"GPRs in a "scalar ISA" is a bit off. In graphics parlance, the term really just refers to the lack of things like dot products and matrix multiply instructions where the compiler has to line numbers into "per-thread" vectors for the instructions to work out. Turns out you can save a lot of work by not having that kind of ISA, and just moving to ones where you operate on number registers (which are then "vector"'d to be thread-count wide). Wouldn't be graphics unless we overloaded a piece of terminology 20 times.
- sanketsarang 5y agoOn the same basis, it would also help if you could provide a comparison between GPUs commonly used for ML. Tesla k80, P100, T4, V100 and A100. How has the architecture evolved to make the A100 significantly faster? Is it just the 80GB RAM, or there is more to it from an architecture standpoint?
- M277 5y agoYou may find this[0] helpful (note -- download link to a .PDF). It's the GA100 whitepaper. [0]: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...
- kkielhofner 5y agoAs a starter T4 is heavily optimized for low power consumption on inference tasks. IIRC it doesn’t even require additional power beyond what the PCIe bus can provide but basically useless for training unlike the others.
- einpoklum 5y ago> How has the architecture evolved to make the A100 significantly faster? Oh, very much so. By way more than an order of magnitude. For a deeper read, have a look at the "architecture white papers" for Kepler, Pascal, Volta/Turing, and Ampere: https://duckduckgo.com/?t=ffab&q=NVIDIA+architecture+white+papers&ia=web https://duckduckgo.com/?t=ffab&q=NVIDIA+architecture+white+p... or check out the archive of NVIDIA's parallel4all blog ... hmm, that's weird, it seems like they've retired it. They used to have really good blog posts explaining what's new in each architecture. You could also have a look here: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#features-and-technical-specifications https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.... for the table of various numeric sizes and limits which change with different architectures. But that's not a very useful resource in and of itself.
- touisteur 5y agoOne day I'll get my hands on both an A40 and an A100 and I'll maybe get an answer to the question: does the 5120bits memory bus help that much? The A100 has less cuda cores, around 1/4 more tensor cores but seems to be the preferred 'compute' and 'ai training' option all around. What gives?
- tppiotrowski 5y agoI know there's a lot of Javascript developers on this forum. If you want to get into GPU programming, I highly recommend gpu.js [1] library as a jumping off point. It's amazing how powerful computers are and how we squander most our cycles. [1] https://gpu.rocks/#/ https://gpu.rocks/#/ Disclaimer: I have one un-merged PR in the gpu.js repo
- tenaciousDaniel 5y agoThanks! I’m a JS dev who happens to be very interested in getting into graphics.
- Jasper_ 5y agoNote that GPU.js is not for graphics, it's primarily for doing GPU Compute. I'd recommend looking at a 3D scene library to get your feet wet (and there are plenty of those in JS), or if you're interested in the underlying workings, to look at either WebGL or WebGPU.
- corysama 5y agoWhenever I experience a slow, stuttering UI, I am reminded that 11 years ago, and iPhone 3GS could do this https://m.youtube.com/watch?v=kqz5ehun-o0 https://m.youtube.com/watch?v=kqz5ehun-o0
- floatboth 5y ago> Mesa 3D - driver library that provides open source driver (mostly copy of AMD-VLK) Wrong, wrong, very wrong. No copy here. Mesa's RADV was developed completely independently from AMD, in fact it predates the public release of AMDVLK. It's also possibly the best Vulkan implementation out there. Valve invested heavily in the ACO compiler backend, so it compiles shaders both very well and very quickly.
- mhh__ 5y ago(Thank you valve!)
- MayeulC 5y agoValve is contributing quite a bit, with their ACO compiler for instance, but the RADV driver did not in fact originate with them. We owe a great deal to David Airlie for that. He wrote a complete vulkan driver years before AMD released the source code for AMDVLK. Valve started contributing a few years in, IIRC. To AMD's credit, they did help with documentation, questions, and much of the work was built on top of what was there for RadeonSI (OpenGL): shader compiler back-end, etc. https://www.phoronix.com/scan.php?page=news_item&px=RADV-Radeon-Vulkan-Driver https://www.phoronix.com/scan.php?page=news_item&px=RADV-Rad... -> https://airlied.livejournal.com/81460.html https://airlied.livejournal.com/81460.html
- MayeulC 5y agoNot to mention that there is much more to Mesa than RADV. Gallium3D and state trackers, RadeonSI and a few other drivers (Apple M1, Qualcomm Adreno, Broadcom, Mali) to only name these. The status of some drivers is tracked here: https://mesamatrix.net/ https://mesamatrix.net/
- Jasper_ 5y agoNote that a lot of the Gallium3D architecture is designed for the stateful APIs like OpenGL. Very little of it is used in the Vulkan drivers.
- wpietri 5y agoHave folks seen good Linux tools for actually monitoring/profiling the GPU's inner workings? Soon I'll need to scale running ML models. For CPUs, I have a whole bag of tricks for examining and monitoring performance. But for GPUs, I feel like a caveman.
- corysama 5y agohttps://developer.nvidia.com/nsight-compute https://developer.nvidia.com/nsight-compute
- not-elite 5y agoIn nvidia land, `nvidia-smi` is like `top` for your gpus. If you're running compiled CUDA, `nvprof` is very useful. But I'm not sure how much work it would take to profile something like a pytorch model. https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-handy-universal-gpu-profiler/ https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-h...
- arcanus 5y agoIn AMD land, `rocm-smi` is like `top` for your gpus. Very similar in capabilities and functionality to NVIDIA.
- zetazzed 5y agoFor NVIDIA GPUs, Nsight systems is wildly detailed and has both GUI and CLI options: https://developer.nvidia.com/nsight-systems https://developer.nvidia.com/nsight-systems For DL specifically, this article covers a couple of options that actually plug into the framework: https://developer.nvidia.com/blog/profiling-and-optimizing-deep-neural-networks-with-dlprof-and-pyprof/ https://developer.nvidia.com/blog/profiling-and-optimizing-d... nvidia-smi is the core tool most folks use for quick "top"-like output, but there is also an htop equivalent: https://github.com/shunk031/nvhtop https://github.com/shunk031/nvhtop A lot of other tools are build on top of the low-level NVML library (https://developer.nvidia.com/nvidia-management-library-nvml https://developer.nvidia.com/nvidia-management-library-nvml). There are also Python NVML bindings if you need to write your own monitoring tools.
- dragontamer 5y agoFor another gentle introduction to GPU architecture, I like "The Graphics Codex", specifically the chapter on Parallel Architectures: https://graphicscodex.courses.nvidia.com/app.html?page=_rn_parallel https://graphicscodex.courses.nvidia.com/app.html?page=_rn_p...
- corysama 5y agoAlso https://fgiesen.wordpress.com/2011/07/09/a-trip-through-the-graphics-pipeline-2011-index/ https://fgiesen.wordpress.com/2011/07/09/a-trip-through-the-...
- M277 5y agoThanks a lot, always enjoy your posts on here and r/hardware! Do you have a hardcore introduction with even more detail / perhaps even with examples of implementations? :) I find white papers quite good (although I admit there are many things I don't understand yet and constantly have to look up), but even these sometimes feel a bit general.
- dragontamer 5y ago> Do you have a hardcore introduction with even more detail / perhaps even with examples of implementations? Hard to beat the actual technical references when you want something hardcore! The actual CUDA documentation is good: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.... In particular, "Section 5: Performance Guidelines": https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#performance-guidelines https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.... gives a lot of micro-architectural details, including those nasty "bank conflicts" that people keep talking about. Honestly, the original documentation says it best with just a few paragraphs on that particular manner: > To achieve high bandwidth, shared memory is divided into equally-sized memory modules, called banks, which can be accessed simultaneously. Any memory read or write request made of n addresses that fall in n distinct memory banks can therefore be serviced simultaneously, yielding an overall bandwidth that is n times as high as the bandwidth of a single module. > > However, if two addresses of a memory request fall in the same memory bank, there is a bank conflict and the access has to be serialized. The hardware splits a memory request with bank conflicts into as many separate conflict-free requests as necessary, decreasing throughput by a factor equal to the number of separate memory requests. If the number of separate memory requests is n, the initial memory request is said to cause n-way bank conflicts. See? Its really not that hard or unapproachable. Just read the original docs, its all there. AMD's documentation is scattered to the winds, but the same information is around. I'd say that your #1 performance guidelines are from the ancient optimization guide from 2015. Its a bit dated, but its fine: http://developer.amd.com/wordpress/media/2013/12/AMD_OpenCL_Programming_Optimization_Guide2.pdf http://developer.amd.com/wordpress/media/2013/12/AMD_OpenCL_... Chapter 1 and Chapter 2 are relevant to today's architectures (even RDNA, even though some details have changed). Chapter 2: GCN, applies for all AMD GPUs from the 7xxx series, through the Rx 2xx series, Rx 3xx, 4xx, 5xx, Vega, and CDNA (aka: MI100) architectures. RDNA does not have as good of an architectural guide. Start with the OpenCL guide for optimization, and then "update" your knowledge with the rather short RDNA guide: https://gpuopen.com/performance/ https://gpuopen.com/performance/ For assembly language details: * CUDA's PTX is basically assembly language: https://docs.nvidia.com/cuda/parallel-thread-execution/index.html https://docs.nvidia.com/cuda/parallel-thread-execution/index... * PTX is a portable assembly language: NVidia continuously updates their GPUs. Volta was very well studied by this paper: https://arxiv.org/abs/1804.06826 https://arxiv.org/abs/1804.06826. I'd suggest reading this AFTER you learn the basics of PTX. * AMD publishes their assembly language for each generation. Vega is probably a good starting point: https://developer.amd.com/wp-content/resources/Vega_Shader_ISA_28July2017.pdf https://developer.amd.com/wp-content/resources/Vega_Shader_I...
- h2odragon 5y agoVery nice. This is gentle like movie dinosaurs are "cheeky lizards". I'd hate to see the "Turkish prison BDSM porn" version. I'm looking at graphics code, again, from a "I know enough C to shoot myself in the foot and want to draw a circle on the screen" perspective. It's hilarious how much "stack" there is in all the ways of doing that; I look at some of this shit and want to go back to Xlib for its simple grace.
- Jasper_ 5y ago2D graphics is very different and mostly doesn't require the GPU's assistance. If you want to plot a circle on an image and then display that to the screen, you don't require any of this stack. If you want a high-level library that will draw a circle for you, you can use something like Skia or Cairo which will wrap this for you into a C API. GPUs solve problems of much larger scale, and so the stack has evolved over time to meet the needs of those applications. All this power and corresponding complexity has been introduced for a reason, I assure you.
- zozbot234 5y ago> 2D graphics is very different and mostly doesn't require the GPU's assistance. There are interesting tasks in 2D graphics that may easily warrant GPU acceleration, such as smooth animation of composited surfaces (even something as simple as displaying a mouse pointer falls under this, and is accelerated in most systems) or rendering complex shapes including text.
- deleted 5y ago[deleted]
- ww520 5y agoParallelism with the computing units in GPU permeates the entire computing model. For the longest time I don't get how partial derivative instructions like dFdx/dFdy/ddx/ddy work. None of the doc helps. These instructions take in a number and return its partial derivative, just a generic number, nothing to do with graphic, geometry or function. The number could have been the dollar amount of a mortgage and its partial derivative is returned. It turns out these functions tied to how the GPU architecture runs the computing units in parallel. The computing units are arranged to run in parallel in a geometric grid according to the input data model. The computing units run the same program in LOCK STEP in parallel (well at least in lock step upon arriving at the dFdx/dFdy instructions). They also know their neighbors in the grid. When a computing unit encounters a dFdx instruction, it reaches across and grabs its neighbors' input value to their dFdx instructions. All the neighbors arrive at dFdx at the same time with their input value ready. With the neighbors' numbers and its own number as the mid point, it can compute the partial derivative using gradient slope.
- anonymous532 5y agoI don't sign in often, thank you for this amazing reveal.
- bigdict 5y agoSo the function F is implicitly defined to have value F(x,y) = v, where x and y are coordinates of the core, and v is the input value to the dFdx/dFdy instruction? Then the output of the instruction running on the x,y core (take dFdx for example) is supposedly equal to (F(x+1,y)-F(x-1,y))/2?
- ww520 5y agoYes. That's the idea. Though not exactly sure how the GPU uses the neighboring values to interpolate the partial derivative. It's probably GPU dependent.
- corysama 5y agoPretty much all of them work in 2x2 quads. DyDx is calculated within a quad. But, not across them. This is an imperfect approximation to the derivative that needs to be accounted for. But, it’s useful and very cheap and trivially easy.
- jimmyvalmer 5y ago> GTX version of Turing architecture (1660, 1650) has no Tensor cores, instead > it has freely available FP16 units! It's always a bad sign when an author's exclamation has all the surprise factor of a tax code. As a previous poster said, "gentle" is relative.
- sidewinder128 5y agoAwesome read, I get more how GPUs works thank you to the author!.
- ai_ja_nai 5y agoNot gentle at all :) Requires familiarity. Also, how about Mesa3D copying Vulkan? It predates it.
- devit 5y agoThat's very verbose but not super clear. A better explanation is that the main problems of processor design are that memory reads take 100-1000 times the time of an arithmetic operation and that hardware is faster when operation are run in parallel. CPUs handle the those issues by having large memory caches, and lots of circuitry to execute instructions "out of order", i.e. run other instructions that don't depend on the memory read result or the result of other operations. This is great to run sequential code as fast as possible, but quite inefficient overall. GPUs instead handle the memory problem by switching to another thread in hardware, and the parallelism problem by mainly using SIMD (with masking and scatter/gather memory accesses, so they look like multiple threads). This works well if you are doing mostly the same operations on thousands or millions of values, which is what graphics rendering and GPGPU is. Then there are also DSPs that solve the memory access issue by only having very little on-CPU memory and either explicit DMA or memory reads giving a delayed result, and parallelism by being VLIW. And finally the in-order/microcontroller CPUs that simply don't care about performance and do the cheapest and simplest thing.