12 ms·
How CUDA Programming Works
- cbrogrammer 4y agoWriting CUDA by hand is like writing x86 assembly by hand, why would anyone ever do that willingly and not for fun?
- pjmlp 4y agoSo now standard C++17 or modern Fortran is like writing Assembly by hand?
- cbrogrammer 4y ago> is **like** writing x86 assembly by hand yes absolutely, way too granular and low level
- w1nk 4y agoWhat? CUDA is intentionally granular and low level, why do you feel that is a negative thing at this level of abstraction? Are you suggesting the tools should be better so that doesn't have to be the case? I can't figure out what you're actually trying to express here.
- cbrogrammer 4y agoCUDA is absolutely fine just like x86 assembly is absolutely fine. They are well thought out. What I am saying is that from my perspective they are as granular and difficult, and programming directly in them for even small projects is not a good idea. GPU programming is, at the moment, stuck in 1960's way of thinking, as there are multiple people writing CUDA code by hand.
- w1nk 4y agoThere's nothing 1960s about it, that's just not well reasoned. These computation constructs/tools just don't currently have better abstractions while maintaining the desired computational performance. It's a strange intersection of needs where one wants or needs something like CUDA, but doesn't care to ensure their computation is actually running optimally. If you don't want to be bothered with control and granularity, why are you trying to write high performance CUDA code in the first place? Would you mind elaborating as to what your hobby project was?
- pjmlp 4y agoNow C++17 with NVidia parallel libraries is too granular and low level, what a joke. Yes one can make use of the Assembly like low level APIs, which unlike OpenCL, are opt-in, not forced upon everyone. Clever devs use the high level C++ and Fortran NVidia libraries.
- ykl 4y agoControversial opinion: I wish every GPU programming language and API did memory management like CUDA. In CUDA, you allocate and manage GPU memory using things like cudaMalloc and cudaMemcpy, which behave exactly like what their names suggest, and handles to GPU memory allocations are just plain old pointers just like regular pointers, except pointing to the GPU’s address space. Basically if you know how to deal with memory in C, you already know how to deal with GPU memory in CUDA. Contrast with Vulkan/GL/Metal/DirectX, where in order to manage GPU memory, you need to know about a whole diverse zoo of different buffer types and different rules for different buffers and there’s a billion different API calls to handle different cases… I just want a chunk of GPU memory that I can do whatever I want with!
- tehsauce 4y agoI agree. Maybe there’s a reason for all the extra complexity in vulkan, perhaps because such a wide variety of devices are targeted?
- my123 4y agoExplicit GPU pointers, instead of opaque structures from the programmer's view, only became part of core in Vulkan 1.2 (VK_KHR_buffer_device_address). And for Metal, getting the GPU address of a buffer only became exposed in Metal 3.0, which is going to be released this fall in iOS/iPadOS 16 and macOS Ventura.
- nmfisher 4y agoI've only touched OpenGL very briefly, but isn't this mostly explained by the fact that CUDA is for general purpose computing and not hardware rendering? If you're mostly dealing with vertex/texture buffers, then presumably you'll need to dress up CUDA with a bit of custom boilerplate, at which point it starts looking like the OpenGL approach anyway.
- upbeat_general 4y agoDoes anyone know of good tutorials for CUDA programming incl debugging? Especially one that also covers the recommended way to bind w/PyTorch? I've always found writing CUDA kernels to be a bit unapproachable.
- synergy20 4y agoI kind of feel there are two different levels of sw development for CUDA. 1. CUDA level programming for graphic processing etc, or writing a c/c++ library for Pytorch/tensorflow framework. 2. Pytorch/Tensorflow level coding(e.g. training a model), you just pick their CUDA-specific APIs, and the two frameworks handle the rest under the hood, no CUDA specific coding details(point 1 above) is required from the users as far as I can tell. if you're interested in 1, Nvidia has c++ guide to download, if you're interested in 2, then the focus is on the AI framework instead of CUDA.
- serial_dev 4y agoCUDA by Example helped me a lot learning about general purpose GPU programming. It doesn't cover Python, PyTorch at all though.
- arpankapoor 4y agoI found the introductory chapters (1-3) of this book[0] quite good. It is different from the NVIDIA CUDA C++ guide in that it uses modern C++ and has non-trivial real-world examples. I also wrote a blog post [1] exploring CUDA to write a simple CNN inference module which you might find useful. [0]: https://www.cambridge.org/core/books/programming-in-parallel-with-cuda/C43652A69033C25AD6933368CDBE084C https://www.cambridge.org/core/books/programming-in-parallel... [1]: https://arpankapoor.com/cuda https://arpankapoor.com/cuda
- zmgsabst 4y agoI don’t know a good tutorial, but numba’s CUDA is good for learning on your own projects. I made some automata engines with that framework. http://numba.pydata.org/numba-doc/latest/cuda/ http://numba.pydata.org/numba-doc/latest/cuda/
- m_mueller 4y agoThis right here is why Nvidia has such a massive lead in computational GPU sales. Investments into this ecosystem were made 15-17 years ago. I.e. there are long ways to go for AMD, Intel and others (Apple?).
- sennight 4y agoThey have a massive lead due to vendor locking any software written to depend on CUDA instead of an open standard... and that is a good thing? Okeydokey.
- snovv_crash 4y agoCUDA is easy to use and the open standard has an extremely high barrier to entry. This is what enabled Nvidia to even lock people in in the first place - their tech was just so much better.
- sennight 4y agoThat "barrier to entry" line works for things that saturate broad markets... and that is definitely not the case with GPGPU. So when you try to use that line of thinking, given the incredibly well funded and hyper niche use cases it sees, it sounds as if you're saying that opencl is too hard for those dummies writing nuclear weapon simulators at Oak Ridge National Laboratory. And before anybody swings in on a chandelier with "but the scientists are science-ing, they can't be expected to learn to code - something something python!": take a look at the software and documentation those labs make publicly available - they are definitely expected to be comfortable with things weirder than opencl.
- oytis 4y agoMaybe they can work with complex libraries, but if there is a better one available I would totally understand that they prefer it. You need to be in a certain software bubble not to understand how to work with OpenCL, but to care enough about whether something is an open standard, whether something is open source etc.
- torginus 4y agoIt's so strange, when I went to school almost a decade ago, I took a GPGPU course - I assumed that with GPU compute becoming widely available, it would become ubiquitous, why sort an array on the CPU in nlogn time, when you can do it on the GPU in logn with bitonic sort? For some reason this future has never come to pass, GPGPU remained a computational curiosity, with it largely relegated to computer graphics and machine learning (although in the latter case, people rarely write CUDA themselves). I wonder why that is the case, and if it will ever become mainstream.
- jahewson 4y agoFor most day-to-day (non-big) data the delay from copying the data to and from the GPU will outweigh the time saved. For large data, there’s usually an existing library for handing it - images, videos, ML weights, etc. It may take advantage of SIMD instructions, perhaps, or GPU, the end user doesn’t need to know. Another reason is that specialised devices beat general devices. Once everyone settles on H.264, then adding specialised decoding silicon to the die is a big win, especially where battery life and efficiency matter. Same for ray-tracing, neural/tensor cores.
- torginus 4y agoYeah, but we got integrated GPUs which share memory memory with the CPU - consoles, mobiles, have done this forever, and it seems that sharing memory between the CPU and GPU is the way to go, considering stuff like the M1.
- rocqua 4y agoM1 can pull off shared memory because it has ridiculous RAM speed. iGPU has to share memory. Thing is CPU is very memory latency sensitive, whilst GPU is much more memory throughout dependent. Hence they get memory with different tradeoffs.
- majke 4y agoIn my humble opinon there just isn't much point to use GPGPU for normal computing. I'll give you an example. https://blog.cloudflare.com/computing-euclidean-distance-on-144-dimensions/ https://blog.cloudflare.com/computing-euclidean-distance-on-... during this work I was able to achieve: - 7.5kqps on 40 Xeon CPU's (optimized avx code) - 35kqps on V100 gpu V100 is say $10k, Xeon 6162 is $3k. That means $0.4 per request on intel, and $0.28 per request on GPU. This really is not much difference. I'd prefer to own more general purpose servers than hard to program proprietary GPU's. Obviousluy, getting GPU's sometimes makes sense. There are tasks that are a better fit for CUDA/GPGPU, stuff that requires high memory bandwidth, doesn't do branches. Furthermore, while V100 costs a lot, 3080 is supposed to be only say $700, which changes the equation in favor of gpu. But overally, modern x86 AVX is surprisingly good, and apart from very specialistic tasks, porting generic algorithms to GPU is not worth the effort.
- halotrope 4y agoNVIDIA gets a lot of flak (and rightfully so) but CUDA is a really approachable API. Considering how much complexity it encapsulates, it could be a lot worse. The concurrency primitives, C(++) integration and memory management are quite good. The profiling and debugging toolkits that come with they CUDA stack are also world-class. I really wish the could become a better citizen in Linux and OSS at some point.
- pjmlp 4y agoNVidia saw what Khronos has decided to ignore for so long. Programming models beyond plain C, good IDE and graphical debugging experience, library ecosystem for GPGPU code. Proprietary APIs from Apple, Sony, Nintendo and Microsoft are similar in this regard. Khronos has waken up with SPIR, SPIR-V and C++ for OpenCL, how much it will matter remains to be seen. It is no wonder that SYCL is adopting a backend agnostic approach instead.
- einpoklum 4y agoFor writing kernels and profiling, yes, it's pretty nice. But the host-side API is rather unapproachable. There are, like, 20 API calls just for different variants of memory allocation, all inconsistently named. Also, the APIs are almost entirely (but not 100%) C'ish. In the API wrappers I've written: https://github.com/eyalroz/cuda-api-wrappers https://github.com/eyalroz/cuda-api-wrappers I try to address these and some other issues. We should also remember that NVIDIA artificially prevents its profiling tools from supporting OpenCL kernels - with no good reason.
- dahart 4y ago> I really wish the could become a better citizen in Linux and OSS at some point. Did you miss the good news a month ago? https://www.makeuseof.com/nvidia-open-sources-gpu-driver-for-linux/ https://www.makeuseof.com/nvidia-open-sources-gpu-driver-for...
- mike_hearn 4y agoThis is really a very good talk, kudos (cudos?) to the speaker. It really drives home the core point about the high level resource management issues you'll face when using a GPU and how you have to plan your algorithms so carefully to ensure they'll fit.
- flakiness 4y agoIt's fun to see how they refer to semi-classic supercomputers like Clay and Hitachi as the scale, although it doesn't help grasping the speed at all! To be useful (in price of non-cool), it can probably be replaced with today's cheap computers like lower end phones or chromebooks. Well, I know it isn't worth it as the deck is more about encouragement and excitement than getting detailed understanding :-) I love how NVIDIA folks love computers.
- Const-me 4y agoNice presentation. None of what was said is specific to CUDA. It’s the same thing in DirectCompute HLSL. Only the terminology is different: CUDA thread blocks are called thread groups in DirectCompute, warps are called waves, shared memory is called “group shared”, etc. Probably applies to others too (Vulkan, Metal, etc.) albeit I never used these for HPC-like stuff, not 100% sure.