13 ms·
GPU Survival Toolkit for the AI age
- aunty_helen 3y agoWe’re back to “every developer must know” clickbait articles?
- igh4st 3y agoit seems so... Should take this article's statements with a grain of salt.
- mhh__ 3y agoAlthough I think they'll be replaced by ChatGPT a good article in that style is actually quite valuable. I like attacking complexity head on, and have a good knowledge of both quantitative methods & qualitative details of (say) computer hardware so having an article that can tell me the nitty gritty details of a field is appreciated. Take "What every programmer should know about memory" — should every programmer know? Perhaps not, but every good programmer should at least have an appreciation of how a computer actually works. This pays dividends everywhere — locality (the main idea that you should take away from that article) is fast, easy to follow, and usually a result of good code that fits a problem well.
- lordwiz 3y agoIn this AI Age, It is crucial for developers to have a fundamental understanding of GPUs and their application to AI development.
- anonylizard 3y agoI think python is dominant in AI, because the python-C relationship mirrors the CPU-GPU relationship. GPUs are extremely performant, and also very hard to code in, so people just use highly abstracted API calls like pytorch to command the GPU. C is very performant, and hard to code in, so people just use python as a abstraction layer over C. Its not clear if people need to understand GPUs that much (Unless you are deep in AI training/ops land). In time, since moore's law has ended and multithreading becomes the dominant mode of speed increases, there'll probably be brand new languages dedicated to this new paradigm of parallel programming. Mojo is a start.
- sitkack 3y agoMoore’s law is far from over and multithreading is not the answer. Your opening sentence is spot on tho.
- bmc7505 3y agoCare to elaborate?
- guyomes 3y agoNo idea about the future of the Moore's law precisely. Yet recent research results show that there is still room for faster semiconductors, as discussed on HN [0]. [0]: https://news.ycombinator.com/item?id=38201624 https://news.ycombinator.com/item?id=38201624
- bmc7505 3y agoI mean why isn’t multithreading the answer?
- samus 3y agoMultithreading has the following disadvantages: * Overhead: the overhead to start and manage multiple threads is considerable in practice. Most multithreaded algorithms are in fact slower than optimized serial implementations when n_threads=1 * Communication: threads have to communicate with each other and synchronize access to shared resources. "Embarrassingly parallel" problems don't require synchronization, but many interesting problems are not of that kind. * Amdahl's law: there is a point of diminishing returns on parallelizing an application since it quite likely contains parts that are not easily parallelized.
- the__alchemist 3y agoAnother tip that took me longer than I wished to figure out. Use CUDA, vice graphics APIs+compute. The latter (Vulkan compute etc) is high friction. CUDA is far easier to write in, and the resulting code is easier to maintain and update.
- behnamoh 3y agoYes, and in the process, contribute to Nvidia’s monopoly.
- bogwog 3y agoIt isn't the consumer's responsibility to regulate the market.
- danielmarkbruce 3y agoAnd, in this case NVIDIA earned it. They built a very useful software layer around their chips.
- dzikimarian 3y agoIf consumers don't care about their money, then who would?
- calamari4065 3y agoThen whose responsibility is it? Corporations? The government? Or maybe the tooth fairy?
- tjoff 3y agoThat is a weak argument that could be used to justify tons of behavior, very convenient. Vote with your feet. Maybe you can't or can't afford it, then at least admit the problem to yourself and maybe don't try to persuade others in order to feel better for your own decision.
- tovej 3y agoEh, CUDA can mostly be transformed to HIP, unless you use specialized NVIDIA stuff.
- password4321 3y agoI need a buyers guide: what's the minimum to spend, and best at a few budget tiers? Unfortunately that info changes occasionally and I'm not sure if there's any resource that keeps on top of things.
- alsodumb 3y agohttps://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... This is the best one imo.
- deleted 3y ago[deleted]
- coffeebeqn 3y agoYou can also rent compute online if you don’t want to immediate plop down 1-2k
- fulafel 3y agoFor learning basics of GPU programming your iGPU will do fine. Actual real-world applications are very varied of course.
- johndough 3y agoGoogle Colab, Kaggle Notebooks and Paperspace Notebooks all offer free GPU usage (within limits), so you do not need to spent anything to learn GPU programming. https://colab.google/ https://colab.google/ https://www.kaggle.com/docs/notebooks https://www.kaggle.com/docs/notebooks https://www.paperspace.com/gradient/free-gpu https://www.paperspace.com/gradient/free-gpu
- alberth 3y agoNx / Axon Given that most programming languages are designed for sequential processing (like CPUs), but Erlang/Elixir is designed for parallelism (like GPUs) … I really wonder if Nx / Axon (Elixir) will take off. https://github.com/elixir-nx/ https://github.com/elixir-nx/
- oytis 3y agoErlang was designed for distributed systems with a lot of concurrency not for computation-heavy parallelism
- coffeebeqn 3y agoWould that run on a GPU? I think the future is having both. Sequential programming is still the best abstraction for most tasks that don’t require immense parallel execution
- dartos 3y agoAxon runs compute graphs on gpu, but elixirs parallelism abstractions run on cpu
- matrss 3y agoI am really wondering how well Elixir with Nx would perform for computation heavy workloads on a HPC cluster. Architecturally, it isn't that dissimilar to MPI, which is often used in that field. It should be a lot more accessible though, like numpy and the entire scientific python stack.
- zoogeny 3y agoI've been investigating this and I wonder if the combination of Elixir and Nx/Axon might be a good fit for architectures like NVIDIA Grace Hopper where there is a mix of CPU and GPU.
- arriu 3y agoAre amd GPUs still to be avoided or are they workable at this point?
- JonChesterfield 3y agoThe cuda happy path is very polished and works reliably. The amdgpu happy path fights you a little but basically works. I think the amd libraries starting to be packaged under Linux is a big deal. If you don't want to follow the happy path, on Nvidia you get to beg them to maybe support your use case in future. On amdgpu, you get the option to build it yourself, where almost all the pieces are open source and pliable. The driver ships in Linux. The userspace is on GitHub. It's only GPU firmware which is an opaque blob at present, and that's arguably equivalent to not being able to easily modify the silicon.
- latchkey 3y agoAMD GPUs work great, the issue is that people don't want to mess with ROCm/HIP when CUDA is kind of the documented workflow. Along with the fact that ROCm was stagnant for a long time. AMD missed the first AI wave, but are now committed to making ROCm into the best it can be. The other problem is that there aren't any places to rent the high end AMD AI/ML GPUs, like the MI250's and soon to be released MI300's. They are only available on things like the Frontier super computer, which few developers have access to. "regular" developers are stuck without easy access to this equipment. I'm working on the later problem. I'd like to create more of a flywheel effect. Get more developers interested in AMD by enabling them to inexpensively rent and do development on them, which will create more demand. @gmail if you'd like to be an early adopter.
- k1ns 3y agoI am very new to GPU programming in general and this article was a fun read. It's amazing how far we've come, prime example being able to train a simple "dog or cat" NN that easily.
- z5h 3y agoWe have compilers (languages) like Futhark that aim to optimize explicitly parallel operations. And universal models of computation like interaction nets that are inherently parallel. Am I lazy to expect we’ll be getting a lot more “parallel-on-the-GPU-for-free” in the future?
- zozbot234 3y agoYou can already convert a compute graph to GPU-optimized code using something like Aesara (formerly known as Theano) or TensorFlow. There are also efforts in the systems space that ought to make this kind of thing more widespread in the future, such as the MLIR backend for LLVM.
- rushingcreek 3y agoGood read. However, the AWS P5 instance (along with P4d and P4de) is most certainly oriented towards training, not inference. The most inference-friendly instance types are the G4dn and the G5, which feature T4 and A10G GPUs, respectively.
- axpy906 3y agoCame here to say this author forgot G5.
- bilsbie 3y agoCan I do this in my GeForce GTX 970 4GB?
- dartos 3y agoI don’t think transformer models generate multiple tokens in parallel (how could they?) They just leverage parallelism in making a single prediction
- atomicnature 3y agoTransformers tend to be trained in parallel. BERT = 512 tokens per context, in parallel. GPT too is trained while feeding in multiple words in parallel. This enables us to build larger models. Older models, such as RNNs couldn't be trained this way, limiting their power/quality.
- dartos 3y agoAhh, that makes a lot of sense
- zozbot234 3y agoThis is only sort of true, since you can still train RNNs (including LSTM, etc.) in big batches-- which is usually plenty enough to make use of your GPU's parallel capabilities. The inherently serial part only applies to the length of your context. Transformer architectures thus happen to be helpful if you have lots of idle GPU's such that you're actually constrained by not being able to parallelize along the context dimension.
- atomicnature 3y agoIn RNN, hidden states are to be sequential; in transformers with attention mechanism, we break free of the sequential requirement. Transformers are more amenable to parallelism, and make use of GPUs the most (within the context axis, and outside).
- DougBTX 3y agoA quick search uncovers [0] with a hint towards an answer: just train the model to output multiple tokens at once. [0] https://arxiv.org/abs/2111.12701 https://arxiv.org/abs/2111.12701
- JonChesterfield 3y agoThe CPUs are good at serial code and GPUs are good at parallel code is kind of true but something of an approximation. Assume equivalent power budget in the roughly hundreds of watts range, then: A CPU has ~100 "cores" each running one (and-a-hyperthread). independent things, and it hides memory latency by branch prediction and pipelining. A GPU has ~100 "compute units", each running ~80 independent things interleaved, and it hides memory latency by executing the next instruction from one of the other 80 things. Terminology is a bit of a mess, and the CPU probably has a 256bit wide vector unit while the GPU probably has a 2048bit wide vector unit, but from a short distance the two architectures look rather similar.
- bee_rider 3y agoI’m always surprised there isn’t a movement toward pairing a few low latency cores with a large number of high throughput cores. Surround single Intel P core with a bunch of E cores. Then, hanging off the E cores, stick a bunch of iGPU cores and/or AVX-512 units. Call it Xeon Chi.
- softfalcon 3y agoNeat idea, probably even viable! I think they may have a hurdle of getting folks to buy into the concept though. I imagine it would be analogous to how Arria FPGA’s were included with certain Xeon CPU’s. Which further backs up your point that this could happen in the near future!
- Const-me 3y agoI think one possible reason for that, ideally these things need different memory. If you use high-bandwidth high-latency GDDR memory, CPU cores will underperform due to high latency, like there: https://www.tomshardware.com/reviews/amd-4700s-desktop-kit-review-ps5-cpu https://www.tomshardware.com/reviews/amd-4700s-desktop-kit-r... If you use low-latency memory, GPU cores will underperform due to low bandwidth, see modern AMD APUs with many RDNA3 cores connected to DDR5 memory. On paper, Radeon 780M delivers up to 9 FP32 TFLOPS, the figure is close to desktop version of Radeon RX 6700 which is substantially faster in gaming.
- Matumio 3y ago> When faced with multiple tasks, a CPU allocates its resources to address each task one after the other Ha! I wish CPUs were still that simple. Granted, it is legitimate for the article to focus on the programming model. But "CPUs execute instructions sequentially" is basically wrong if you talk about performance. (There are pipelines executing instructions in parallel, there is SIMD, and multiple cores can work on the same problem.)
- pclmulqdq 3y agoI think this post focused on the wrong things here. CPUs with AVX-512 also have massive data parallelism, and CPUs can execute many instructions at the same time. The big difference is that CPUs spend a lot of their silicon and power handling control flow to execute one thread efficiently, while GPUs spend that silicon on more compute units and hide control flow and memory latency by executing a lot of threads.
- mhh__ 3y agoIt will do multipleS SIMD instructions at the same time, too.
- shortrounddev2 3y agoThis article claims to be something every developer must know, but it's a discussion of how GPUs are used in AI. Most developers are not AI developers, nor do they interact with AI or use GPUs directly. Not to mention the fact that this articles barely mentions 3d graphics at all, the reason gpus exist
- lucb1e 3y agoOne can benefit from knowing fundamentals of an adjacent field, especially something as broadly applicable as machine learning. - You might want to use some ML in the project you are assigned next month - It can help collaborating with someone who tackles that aspect of a project - Fundamental knowledge helps you understand the "AI" stuff being marketed to your manager The "I don't need this adjacent field" mentality feels familiar from schools I went to: first I did system administration where my classmates didn't care about programming because they felt like they didn't understand it anyway and they would never need it (scripting, anyone?); then I switched to a software development school where, guess what, the kids couldn't care about networking and they'd never need it anyway. I don't understand it, to me it's both interesting, but more practically: fast-forward five years and the term devops became popular in job ads. The article is 1500 words at a rough count. Average reading speed is 250wpm, but for studying something, let's assume half of that: 1500/125 = 12 minutes of your time. Perhaps you toy around with it a little, run the code samples, and spend two hours learning. That's not a huge time investment. Assuming this is a good starting guide in the first place.
- mrec 3y agoThe objection isn't to the notion that "One can benefit from knowing fundamentals of an adjacent field". It's that this is "The bare minimum every developer must know". That's a much, much stronger claim. I've come to see this sort of clickbait headline as playing on the prevalence of imposter-syndrome insecurity among devs, and try to ignore them on general principle.
- lucb1e 3y agoFair enough! I can kind of see the point that, if every developer knew some basics, it would help them make good decisions about their own projects, even if the answer is "no, this doesn't need ML". On the other hand, you're of course right that if you don't use ML, then it's clearly not something you "must" know to do your job well.
- lucb1e 3y ago> AWS GPU Instances: A Beginner's Guide [...] Here are the different types of AWS GPU instances and their use cases The section goes on to teach Amazon-specific terminology and products. A "bare minimum everyone must know" guide should not include vendor-specific guidance. I had this in school with Microsoft already, with never a mention of Linux because they already paid for Windows Server licenses for all of us... Edit: and speaking of inclusivity, the screenshots-of-text have their alt text set to "Alt text". Very useful. It doesn't need to be verbatim copies, but it could at least summarize in a few words what you're meant to get from the terminal screenshot to help people that use screen readers. Since this comment floated to the top, I want to also say that I didn't mean for this to dominate the conversation! The guide may not be perfect, but it helped me by showing how to run arbitrary code on my GPU. A few years ago I also looked into it, but came away thinking it's dark magic that I can't make use of. The practical examples in both high- and low-level languages are useful Another edit: cool, this comment went from all the way at the top to all the way at the bottom, without losing a single vote. I agree it shouldn't be the very top thing, but this moderation also feels weird
- mikehollinger 3y agoAgreed. This isn’t actually that useful of a guide in the first place. Tbh the most basic question is: “are you innovating inside the AI box or outside the AI box?” If inside - this guide doesn’t really share anything practical. Like if you’re going to be tinkering with a core algorithm and trying to optimize it, understanding BLAS and cuBLAS or whatever AMD / Apple / Google equivalent, then understanding what pandas, torch, numpy and a variety of other tools are doing for you, then being able to wield these effectively makes more sense. If outside the box - understanding how to spot the signs of inefficient use of resource - whether that’s network, storage, accelerator, cpu, or memory, and then reasoning through how to reduce that bottleneck. Like - I’m certain we will see this in the near future, but off the top of my head the innocent but incorrect things people do: 1. Sending single requests, instead of batching 2. Using a synchronous programming model when asynchronous is probably better 3. Sending data across a compute boundary unnecessarily 4. Sending too much data 5. Assuming all accelerators are the same. That T4 gpu is cheaper than an H100 for a reason. 6. Ignoring bandwidth limitations 7. Ignoring access patterns
- Matumio 3y agoThe Mandelbrot example seems to make interpreted Python stand in for "the CPU performance"? If that's true, then I'm surprised they only see a 10x speed-up. I would expect more from only compiling that loop for the CPU. (Comparing to interpreted Python without numpy.) Given they already have a numba version, why not compile it for the CPU and compare? Also, they say consumer CPUs have 2-16 cores. (Who has 2 cores these days?) They go on suggest to rent an AWS GPU for $3 per hour. You're more likely to get 128 cores for that price, still on a single VM. Not saying it will be easy to write multi-threaded code for the CPU. But if you're lucky, the Python library you're using already does it.
- lucb1e 3y ago> Also, they say consumer CPUs have 2-16 cores. (Who has 2 cores these days?) Pretty sure my mom's laptop has 2 cores; I can't think of anyone whose daily driver has 16 cores. Real cores, not hyperthread stuff running at 0.3× the performance of a real core. As for the 128-core server system, note that those cores are typically about as powerful as a 2008 notebook. My decade-old laptop CPU outperforms what you get at DigitalOcean today, and storage performance is a similar story. The sheer number makes up for it, of course, but "number of cores" is not a 1:1 comparable metric. Agree, though, that the 10x speedup seems low. Perhaps, at 0.4s, a relatively large fraction of that time is spent on initializing the Python runtime (`time python3 -c 'print("1337")'` = 60ms), the module they import that needs to do device discovery, etc.? Hashcat, for example, takes like 15 seconds to get started even if it then runs very fast after that.
- 65a 3y agoMy 2016 desktop had 22 cores and 44 threads. You can have the same processor for < $200 on ebay right now.
- lucb1e 3y agoCool, now I know one person :D I don't think the price matters btw. If 4 cores is all I need (I can't think of a desktop application that I use which benefits from more than 4 cores more than from faster cores, which is typically the trade-off at >4 real cores) then that's what I'd get because that's the optimum for my wallet and performance profile.
- shmerl 3y agoShouldn't the article mention SIMD? I haven't seen it even being brought up.
- johndough 3y agoThe code in this article is incorrect. The CUDA kernel is never called: https://github.com/RijulTP/GPUToolkit/blob/f17fec12e008d0d373e6e8978e6d26f312e2f4ce/mandelbrot/mandelGPU.py#L47 https://github.com/RijulTP/GPUToolkit/blob/f17fec12e008d0d37... I'd also like to point out that 90 % of the time spent to "compute" the Mandelbrot set with the JIT-compiled code is spent on compiling the function, not on computation. If you actually want to learn something about CUDA, implementing matrix multiplication is a great exercise. Here are two tutorials: https://cnugteren.github.io/tutorial/pages/page1.html https://cnugteren.github.io/tutorial/pages/page1.html https://siboehm.com/articles/22/CUDA-MMM https://siboehm.com/articles/22/CUDA-MMM
- Handprint4469 3y agoThank you for this, comments like yours is exactly why I keep coming back to HN.
- sevagh 3y ago>If you actually want to learn something about CUDA, implementing matrix multiplication is a great exercise. There is SAXPY (matrix math A*X+Y), purportedly ([1]) the hello world of parallel math code. >SAXPY stands for “Single-Precision A·X Plus Y”. It is a function in the standard Basic Linear Algebra Subroutines (BLAS)library. SAXPY is a combination of scalar multiplication and vector addition, and it’s very simple: it takes as input two vectors of 32-bit floats X and Y with N elements each, and a scalar value A. It multiplies each element X[i] by A and adds the result to Y[i]. [1]: https://developer.nvidia.com/blog/six-ways-saxpy/ https://developer.nvidia.com/blog/six-ways-saxpy/
- varelse 3y ago[dead]
- lordwiz 3y agoThanks a lot for pointing it out, I have fixed the code and updated the blog.
- convexstrictly 3y agoA great beginner guide to GPU programming concepts: https://github.com/srush/GPU-Puzzles https://github.com/srush/GPU-Puzzles