8 ms·

A Taste of GPU Compute [video]

280 points by raphlinus 7y ago

raphlinus 7y ago
This is a talk I gave at Jane Street late February on GPU compute, especially using graphics API's such as Vulkan and the upcoming WGPU. Feel free to scan through the slides [1], and to ask me anything. [1]: https://docs.google.com/presentation/d/1FRH81IW9RffkJjm6ILFZ7raCgFAUPXYYFXfiyKmhkx8/edit?usp=sharing https://docs.google.com/presentation/d/1FRH81IW9RffkJjm6ILFZ...
- Jhsto 7y ago
  Thanks for the video and slides, super interesting! One question: in one of the slides, you mention the Vulkan 1.2 / SPIR-V 1.5 Vulkan Memory model. What exactly is this and why do you think this is meaningful? Would you have any links to learn more? When I read the SPIR-V specification, it mentions that this kind of memory model exists, but it does not put any effort into explaining why it's worthwhile or how it works on a high level.
  - jabl 7y ago
    Perhaps in a few years when we'll start to see things like CXL providing cache-coherency between main memory and GPU memory this will become important?
  - raphlinus 7y ago
    That could be the subject of a whole other talk :) The memory model is needed to do more sophisticated communication between the parallel workgroups. The classic example is to make a queue, even a simple one such as a simple single-producer single-consumer ring buffer (for communication between a pair of workgroups). The way this works, the producer fills an element in the buffer, then bumps an index. The consumer observes the index, then reads the element from the buffer. Without a memory model, there's no way to guarantee that this ordering is preserved, and (to map this to hardware), the write of the index bump could flow through the caches, while the write of the element could still be "dirty" in the cache on the producer, so the consumer reads stale data. What the memory model does is provide the programmer an explicit way to express ordering constraints. So the write of the index bump is with "release" semantics and the read of that by the consumer is with "acquire" semantics, which guarantees that producer writes before the release are visible to consumer reads after the acquire. This unlocks a whole range of sophisticated concurrent data structures, such as concurrent hash maps, fancier memory allocation (even without a memory model, a simple allocate-only bump allocator is feasible), and so on. Thanks for the question and kind words, I hope this illuminates.
    - Jhsto 7y ago
      Thanks, the answer with some examples was exactly what I was hoping to get answered! Thank you for taking the time to respond.
- throwlaplace 7y ago
  of the links you put on the last page (or other gpu languages you might not have included) what's the most mature way to write decent gpu code right now (even if not necessarily optimal). i'm right this moment working on a project where i'm trying to speed up a blob detection algorithm by porting it to gpu. i'm using pytorch because the primitives fit naturally but if that weren't the case i think i would be kind of at a loss for how to do it (i do not know, as of yet, how to write cuda kernels). btw the Julia link is dead
  - raphlinus 7y ago
    It depends on what you're trying to do. If you want to get stuff done now then just get an Nvidia card and write CUDA. The tools and knowledge community are vastly ahead of other approaches. If you're doing image-like stuff, Halide is good, there's lots of stuff shipping on it. One other project I'd add is emu, which is built on wgpu. I think that's closest to the future I have in mind, but still in early stages. https://github.com/calebwin/emu https://github.com/calebwin/emu
    - throwlaplace 7y ago
      okay in that case can you recommend a good tut for writing cuda code?
      - alexhutcheson 7y ago
        The official docs are pretty good. This is probably a good starting point: https://devblogs.nvidia.com/even-easier-introduction-cuda/ https://devblogs.nvidia.com/even-easier-introduction-cuda/ Nvidia also sponsored a Udacity course, but I don't have any direct experience with it: https://developer.nvidia.com/udacity-cs344-intro-parallel-programming https://developer.nvidia.com/udacity-cs344-intro-parallel-pr...
  - ZeroCool2u 7y ago
    If you're interested in getting started writing GPU code and you're currently dealing with PyTorch a good place to start might be using NUMBA for CUDA programming. It lets you do in Python what you'd normally do in C/C++. It's great, because setup is minimal and it has some good tutorial materials. Plus, you're mostly emulating what you'd end up doing in C/C++, so conceptually the code is very similar. When I was at GTC 2018 this is one of the training sessions I went to and you can now access that here[1]. The official NUMBA for CUDA docs are great too[2]. Also, you can avoid a lot of installation/dependencies headaches by using anaconda/miniconda if you're doing this stuff locally. [1]: https://github.com/ContinuumIO/gtc2018-numba https://github.com/ContinuumIO/gtc2018-numba [2]:https://numba.pydata.org/numba-doc/latest/cuda/overview.html https://numba.pydata.org/numba-doc/latest/cuda/overview.html
- sorenjan 7y ago
  Do you think other alternatives to Cuda will have the same drawback as OpenCL, that Nvidia won't support it properly and therefore not be a viable option on their hardware which is dominating the market at the moment?
  - gmueckl 7y ago
    Well, what are the chances that less vendor lock in is in nVidia's interest?
    - angry_octet 7y ago
      Is it vendor lock-in if the clients want to be there? It's more like you can be in the club with the good music and hip people, or the club with the average unknown DJ and an empty dance floor.
      - bl0b 7y ago
        That makes sense for GPU-compute-in-the-cloud applications where you can pick the hardware and don't care if other hardware doesn't work, but some people want to write programs that use GPGPU that can run on anyone's machine.
  - raphlinus 7y ago
    No, precisely because I see the emerging alternatives as being based on Vulkan and SPIR-V, which I believe have momentum far greater than OpenCL. The only question is whether a compute platform based on these standards and open source tools could be anywhere nearly as good as CUDA. I believe we're far from that today, but it is possible and there's a lot of good work in that direction.
    - muro 7y ago
      Doesn't that exclude Apple?
      - 7y ago
- sprash 7y ago
  Halfway through you were bringing up hopes for a great new way to do 2D rendering on GPUs but then there is no show of actual implementation. Rather anticlimactic! Even decades old Cairo is beating VulKan/GPU based renderers like Skia in some tasks. Can it be that hardware intended for ML or 3D-graphics is not suited at all for 2D rendering (parallelism vs. pipelining)? How will WebGPU solve those problems? To me this talk raises more questions than it answers...
  - raphlinus 7y ago
    It's work in progress. I was hoping to get more done on the renderer but various things have been slowing me down. But lately I've been feeling a new burst of energy, so hope to have something more to show before long. I agree that the talk raises a lot of questions. Hopefully that's a good thing, as I think they deserve the attention of curious people! Regarding the mismatch between 3D hardware and 2D rendering, we'll need hard empirical numbers, but the approach I'm taking is basically to write a software renderer (not unlike Cairo or even libart which predates it), just one that's designed for massive parallelism so it can run on the GPU. This will have performance characteristics more similar to Cairo (just faster as so much more throughput is available) than rasterization-based approaches. The latter really fall down when there's soft clipping and lots of blending, as that requires generation of intermediate texture buffers. I do address this in the talk :) I hope I will have more solid answers for some of these questions before long.
    - sprash 7y ago
      I'm experimenting with GPU based 2D rendering using signed distance functions as drawing primitives for quiet a long time now. My experience is that as soon as the object tree exceeds a certain size the CPU will always beat the GPU. The only way the GPU can win is when you have high complexity drawing primitives and a low amount of objects which is exactly the opposite one usually encounters in the real world. Maybe the partitioning approach you are proposing will help but only real world implementations and benchmarks will tell.
      - 7y ago
- zozbot234 7y ago
  Is Vulkan really becoming viable for compute? It seems like something that's obviously worth exploring given the overlap with more traditional graphics workloads, but I have seen some claims that it's still lagging behind OpenCL wrt. some features, and that it could be good for some things but is not obviously competitive across the board. It's possible that this impression is outdated though.
  - raphlinus 7y ago
    I think the answer to this is emerging, but I would say yes. Vulkan and SPIR-V are rapidly gaining features, while OpenCL is stagnating. Some of these features are very recent, like pointers and a memory model. Another area I'm expecting Vulkan to pull ahead is support for efficient float16 computation, which is important for ML workloads, and is increasingly well supported in hardware. One thing to keep in mind though, programming directly in Vulkan is too low-level for most users, while OpenCL is a fairly friendly API. We need tools that can compile to SPIR-V and deploy on Vulkan, but high enough level you can express your problem rather than (for example) manually placing pipeline barriers between dispatches. The lack of such tools, and having to build some of my own, is one of the reasons for the slowdown in progress on my renderer project.
    - Jasper_ 7y ago
      CUDA is unfortunately very, very far ahead of Vulkan. "Graphics Compute" is quite different from CUDA at the advanced levels if you've never used it.
    - pjmlp 7y ago
      In CUDA I can just use ISO C++17 (C++20 around the corner) plus CUDA extensions, or any other language that targets PTX, with very nice graphical debugging from what is going on the GPGPU. OpenCL is playing catch up with CUDA, but it requires at least version 2.0 for similar capabilities, while many vendors are still shipping 1.x drivers. Vulkan, well it seems to be the same story as OpenCL regarding compute tooling, but lets see how much love it will get from NVidia. I advise watching the GTC 2020 talks regarding CUDA and Vulkan compute.
- Symmetry 7y ago
  Any opinion on ROCm?
  - mpreda 7y ago
    ROCm provides a good OpenCL 2.x implementation targeting AMD GCN. The compiler/optimizer is clang-based. It is open-source. One great thing is that ROCm OpenCL supports GCN assembly in the OpenCL source, which is handy in some corner-cases that are not properly handled by the optimizer.
  - VHRanger 7y ago
    Doesn't support many platforms
- Strilanc 7y ago
  What is a reasonable lower limit on latency that I could expect from a GPU pipeline? For example, I know that if I use WebGL and issue a `readPixels` call then that seems to take on the order of 1 to 10 milliseconds no matter what I do. This suggests that if I needed to react to something in under 100 microseconds, I probably shouldn't have the computation of the reaction involve WebGL. But is that true of GPU compute in general, or just an artifact of the abstraction exposed by WebGL?
  - jdashg 7y ago
    There's a bunch of intractable latency, but there are patterns that can help maintain pipelining, if you don't need the data immediately: https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/WebGL_best_practices#Non-blocking_async_data_downloadreadback https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/W...
    - Strilanc 7y ago
      Right, but if I go closer to the metal how much of the latency is really truly intractable? 100 microseconds? 1 millisecond? 10 milliseconds?
      - bl0b 7y ago
        Can't speak for WebGL APIs, but in OpenGL (and other C++ graphics/GPGPU platforms), it is certainly possible (and necessary) to write fully interactive real-time apps that rely on a few rounds of communication between GPU and CPU code every frame. At 60 fps, you have 16 ms per frame. As long as you aren't copying huge arrays back and forth, I'd say you can budget in the 1 ms order-of-magnitude per data transfer. I don't know much of that would be avoidable if you were actually closer to the metal than the raw OpenGL C++ API call
  - kllrnohj 7y ago
    You do have to design around the GPU typically being almost like a server. It's very fast, but also "very far" away - both in latency & in bandwidth. PCI-E 3.0 x16 is 'only' 16GB/s. That's fast for I/O, but still in the ballpark of being like I/O & not like RAM. And that's if you even get x16 electrical - x8 for GPUs is pretty common in eg. laptops where the other lanes go to things like thunderbolt 3. Integrated graphics are a different story, they often just have a direct connection to DRAM and CPU<->GPU communication is therefore super fast & low latency. In theory, anyway, if the API abstractions even let you share memory between the two. So that means even for things where GPUs should do really well at it, like summing two arrays together, doing it on the CPU can still be a lot faster if the inputs & outputs are local to the CPU.
- 2008guy 7y ago
  What were you talking about with the socialist revolution?
  - mkl 7y ago
    He was giving a talk to people from Jane Street [1], who make money on the stock market using OCaml (mentioned in the same bit). [1] https://en.wikipedia.org/wiki/Jane_Street_Capital https://en.wikipedia.org/wiki/Jane_Street_Capital
    - 2008guy 7y ago
      Right but he basically asserted to everyone that a socialist revolution is coming to wipe them out. Is there something I don’t know? Should I be getting ready for this?
      - mkl 7y ago
        Huh? It was a joke. Explaining the joke will kill it, but here goes: He was saying that their OCaml skills would still be useful even if they couldn't make money in finance because capitalism fell over. The audience got the joke and laughed. It's nothing like what you're saying.
        2008guy 7y ago
        What I saw was this guy flatly asserting that a socialist revolution is coming, I thought at first he was talking about Bernie Sanders, and where there should be a punchline there is no punchline. Then everyone gives an awkward and half-cocked chuckle as one does when presented with something so bizarre.
- fluffything 7y ago
  I write a lot of compute kernels in CUDA, and my litmus test is prefix sum, for two reasons. First, you can implement it in pure CUDA C++, and max out the memory bandwidth of any nvidia or AMD GPU. The CUB library provides a state of the art implementation (using decoupled-lookback) that one can compare against new programming languages. Second, it is one of the most basic parallel algorithm building blocks. Many parallel algorithms use it, and many parallel algorithms are "prefix-sum-like". If I am not able to write prefix sum from scratch efficiently in your programming language / library, I can't use it. Every time someone shows a new programming language for compute, the examples provided are super basic (e.g. `map(...).fold(...)`), but I have yet to see a new programming language that can be used to implement the 2-3 most fundamental parallel algorithms from any parallel algorithms graduate course. For example, Futhark provides a prefix sum intrinsic, that just calls CUB - if you want to implement a prefix-sum-like algorithm, you are out of luck. In WGPU, it appears that prefix-sum will be an intrinsic of WHSL, which sounds like you would be out-of-luck too. You mentioned WGPU and Vulkan. Do you know how to implement prefix-sum from scratch on these? If so, do you know how the performance compare against CUB?
  - raphlinus 7y ago
    That sounds like a great project, one I heartily recommend to people wanting to get their feet wet. First, the basic principles of how to write an efficient prefix sum are the same as before. Definitely read [1]. Also see [2] for an exploration into how much subgroups help over threadgroup shared memory. In fact, [2] has much of the answer you seek, as it's already written in Vulkan and GLSL (compiling to SPIR-V), though it does miss a direct performance comparison to CUB. Second, doing this depends on WebGPU exposing threadgroup shared memory and subgroups operations. I'm not sure where the standard and implementations are on this; there are significant challenges. For example, subgroup shuffle is not available in DX12, though subgroup reduce operations such as WavePrefixSum are. So in general an implementation will need to do runtime detection of capabilities and choose a most-optimum kernel based on that. In theory, it should be possible to achieve comparable performance. But it's likely that WebGPU implementations haven't really been tuned for performance yet, while CUB has seen a ton of work. So, as always, the thing to do is implement and measure. [1]: https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-co... [2]: https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-GPU https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-G...
- m463 7y ago
  When I heard the definition SIMT, it helped me understand the if / do_a / do_b concept. But your subsequent explanation helped understand what went beyond that. Thank you for the talk.
- unixhero 7y ago
  Really good talk, thanks a lot.
moritonal 7y ago
Great technical talk, but an hour long video of one of the more visually appealing techs and not a single demo :( Demo's like this (https://paveldogreat.github.io/WebGL-Fluid-Simulation/ https://paveldogreat.github.io/WebGL-Fluid-Simulation/) help show people why many small processors can beat a few large ones.
- raphlinus 7y ago
  Noted, and I'll do my best to raise up demos for future presentations.
  - rudedogg 6y ago
    I thought you did a really nice job with the slides. The diagrams made it easier to follow along.
  - moritonal 6y ago
    Sorry, the other replies to this are completely correct that you should know your audience and flashy demos might also be a turn-off. This was a fantastic technical demo, I simply work in graphics so am biased towards that.
  - doublesCs 7y ago
    As a counter-point to the above commenter, I'm someone who's in an adjacent industry (HFT) and to me personally spending time on pretty pictures is a turn off. So I guess the take away is: know your audience. I really really enjoyed your presentation, thanks for sharing.
streb-lo 7y ago
I'm not well versed in this area at all -- but something I find fascinating is what I have heard of the early days of GPU compute. Before CUDA and the like, you would apparently take your data you wanted to compute on, structure it as an image/texture and use the graphics API to ham fist your way to the result you wanted. No idea if that's true, but pretty neat if it is.
- gh123man 7y ago
  This is actually still true in a lot of cases where proper compute shaders are not available - notably webGL. See: https://threejs.org/examples/webgl_gpgpu_birds.html https://threejs.org/examples/webgl_gpgpu_birds.html and: https://github.com/yomboprime/GPGPU-threejs-demos https://github.com/yomboprime/GPGPU-threejs-demos Though WebGPU is coming which I believe will mitigate this.
- Qasaur 7y ago
  As the other comment mentioned, this is still true if you want to support legacy graphics APIs. A non-negligible amount (~20%) of Android phones cannot execute compute pipelines as they only support OpenGL ES 3.0 - compute shaders were introduced in GLES 3.1. While it is an headache if you want to support compute on legacy devices, I do think that writing regular vertex/fragment shaders for general purpose GPU computation is an underrated pleasure as you really need to break out the coding golf toolkit to squeeze out maximum performance.
  - gh123man 7y ago
    To add to this - it's not just legacy/Android devices. Since Apple dropped support for OpenGL in favor of metal, cross platform OpenGL compute shaders are now impossible since iOS will never get OpenGL ES 3.1. This has caused headaches for me as I wrote a cross platform game and am now pinned to GLES 3.0 forever.
- raphlinus 7y ago
  That works well for certain kinds of computations, basically where each thread can compute a "pixel" independently. Modern GPU compute has threadgroup local storage, subgroups, atomic operations, etc. These unlock huge categories of workloads that would not fit nicely in the rasterization pipeline.
londons_explore 7y ago
When someone finds a way to use a GPU to accelerate all the code in massive react webapps, that will be the day GPU's turn from a specialists-only device to mainstream...
- SQueeeeeL 7y ago
  Are GPUs specialist devices now?
  - londons_explore 7y ago
    Programming them is a specialist task... I'd take a guess <10% of programmers have ever written CUDA or something equivalent.
    - vmchale 7y ago
      You can use something like Futhark and link it with your program.
maleadt 7y ago
Great talk! Any thoughts on Intel's oneAPI?
- raphlinus 7y ago
  Thanks. First I've heard of it, in fact, so I don't have any thoughts on it. It's interesting though, and would definitely be worth adding to a list of attempts to build portable layers.
  - maleadt 7y ago
    I'm looking at targeting it from Julia, and the lower-level (Level Zero) API seems rather nice, resembling the CUDA driver API but building on SPIR-V. It's also nice how the API is decoupled from the implementation, so let's hope more vendors implement it (apparently Codeplay is working on a CUDA-based implementation).
vmchale 7y ago
My experience with Futhark (and to some extent accelerate) is that they're not terribly hard to program in applicable cases. I'm not sure Futhark-generated code is as fast as specialist code but it's definitely a speedup compared to even skilled CPU implementations.
vmchale 7y ago
Question: what is the state of GPU compute w.r.t. Rust? Are the libraries on par with accelerate or (not exactly comparable) Futhark?
- raphlinus 7y ago
  There's a ton of low and mid level infrastructure for GPU being written in Rust. That includes the gfx-hal work, the wgpu implementation, and more experimental languages such as emu. But it's still early days, and I don't think there's a complete solution comparable to either of those yet.
- VHRanger 7y ago
  Expect a lot of rough patches
jklinger410 7y ago
https://www.google.com/search?q=how+to+pronounce+arbiter&oq=how+to+pronounce+arbiter&aqs=chrome..69i57j0l4.2632j0j1&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=how+to+pronounce+arbiter&oq=...
seanalltogether 7y ago
Is there a meaningful speed difference between creating a 2d drawing engine using "gpu compute" vs going through normal libraries like directx or opengl? I'm fairly ignorant around what is exposed in the compute libraries vs traditional drawing libraries.
- hellofunk 7y ago
  I think the main issue is where the output from the GPU shader is going. OpenGL for example, the output is the screen. That’s the only output. But for compute, you want the output to come back into the CPU world for storing or manipulating or etc. So compute shaders provide that API for shuffling memory back-and-forth between CPU and GPU and having access to general purpose output from the shader beyond just screen pixels.
geokon 7y ago
Long video, so I haven't watched it all yet, but I have a bit of a naiive question: Does Vulkan and WebGL/WebGPU "fall back" to running on the CPU if "appropriate" hardware isn't available? Or are developers required to maintain separate code paths based on hardware? I remember considering rewriting some tight loops in OpenCL but then the maintenance headache of having multiple code paths made the refactor seem not worth it. I'd guess this is/was generally a major speedbump for adoption. I know there is POCL which will run your kernels on the CPU, but it's not something you can't expect available on every platform. Maybe if POCL was part of the kernel or bundled with every distro then the situation would be different. I've seen some project do compute in OpenGL ES 2.0 b/c that's ubiquitous and will always run (I think it's even a requirement for Android systems)
- raphlinus 7y ago
  This is a good question. No, Vulkan and WebGPU will not fall back, they require hardware and driver support. What I think you're looking for is a higher level layer that will target GPU compute if it's available, otherwise CPU (or possibly other resources). Projects in this space include Halide and MLIR.
  - geokon 7y ago
    I have a more theoretical followup then :) So you spend a lot of time in the talk massaging your sequential CPU task into an appropriately GPU task. It's clearly tricky and involves reworking your algorithms to leverage the parallelism and the memory "features" of the GPU. But through rewriting, have you actually substantively hurt the performance of the sequential program? The big picture question is : is the reverse problem of going from a GPU program to a CPU problem ever problematic? I mean in a algorithmic sense - without looking at micro optimizations for leveraging CPU SIMD instructions or whatever. Or are you always going to be pretty okay with running your shaders sequentially one after another on your big luxurious CPU memory pool? and ultimately is there anything stopping you from compiling SPIR-V for a CPU? Could you not autgenerate in the kernel dispatcher just a switch that'll branch and run a precompiled on-CPU kernel if no driver is found? Then you'd finally really get compile-once-run-everywhere GPU code I guess since it's not being done then I'm missing something haha :) Maybe you are going to often hit scenarios where you'd say "No, if I'm going to run this on the CPU I need to fundamentally rewrite the algorithm"
jarrell_mark 7y ago
Tensorflow.js (tf.Tensor) can be used as an easy to use interface to GPU compute (similar API to numpy) which works on all gpus not just Nvidia because it's backed by webgl https://www.tensorflow.org/js/guide/platform_environment https://www.tensorflow.org/js/guide/platform_environment
01100011 7y ago
FWIW, if you're interested Nvidia-centric GPGPU stuff, Nvidia's tech conference went online-only this year and is free for all: https://www.nvidia.com/en-us/gtc/ https://www.nvidia.com/en-us/gtc/
ptrenko 7y ago
Just a question: Has anyone figured out serverless ml predictions yet?
thomas232233 7y ago
Any youtube video that has disabled comments is most likely a sign of unwilling to hear criticism and fearing i might waste my time i don't take that chance.
amelius 7y ago
I wonder at what point we finally decide that "G" in GPU doesn't make sense, and instead we start building compute modules that don't have 8 graphics connectors, and don't mess with graphics apis, and most importantly have open source drivers.
- gbrown 7y ago
  Except for that last part, you’re actually describing a return to the way things were decades ago, when accelerator cards of various types were common.
- avianlyric 7y ago
  Most “GPU”s found in datacentres don’t have any graphic connectors on the back of them. They do all their communication via PCIe. In some cases they may have a connectors for high speed communication fabric between many GPUs. NVLink is an example of this [0]. Outside of the consumer and workstation space GPUs really don’t look like anything you or I would recognise as a GPU anymore, and with APIs like CUDA, don’t look like GPUs to software either. Really all we’re missing is open source drivers. But I wouldn’t hold your breath. [0] https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/
eggy 7y ago
I loved his commentary on Co-dfns for APL at 54:38 in the video. I program in J[0], and dabble in Dyalog APL. Aaron Hsu's work on the gpu compiler, he's the implementor of Co-dfns, is nothing short of mind-blowing[1]. His arguments for using APL vs. more mainstream languages are cogent and persuasive. The paradigm shift from C-like or even Haskell (ML) type languages is too big a gap for most mainstream programmers to accept. My opinion is that aside from Python's clean, indented syntax, the real reason it took off for DL/ML, was Numpy and Pandas. Pandas was heavily influenced by the array languages per its creator Wes McKinney. Besides, I just like playing with J. I have looked briefly at Futhark, and the apltail compiler[2], but I am trying to focus on the APL family, because my time is limited. I am a GPU dilettante who has tried basic CUDA in C. I tried Rust, but Zig [3] is working out better for me given I make more progress with less effort, using C APIs are effortless, but my Rust difficulties may just be my bias with PLs. I find Haskell easier than Rust. I just read an HPC article today about a survey/questionnaire about the languages used among 57 experts. It's still predominantly C++, but with a lot pain expressed by the experts/users. I agree SPIR-V sounds promising, and I hope to check it out. Just like DL, I think people don't realize it needs to be knowledge domain and algorithms, or experts and software. Somebody has to setup the computation based on knowledge domain and software knowledge. This shows itself to me when I wind up running somebody's program, because I don't have my own specific problem I'd like to try and implement as a learning exercise and knowledge exercise. Great talk! I found it very understandable and paced just right! [0] https://jsoftware.com https://jsoftware.com [1] https://scholarworks.iu.edu/dspace/handle/2022/24749 https://scholarworks.iu.edu/dspace/handle/2022/24749 [2] https://futhark-lang.org/blog/2016-06-20-futhark-as-an-apl-compiler-target.html https://futhark-lang.org/blog/2016-06-20-futhark-as-an-apl-c... [3] https://ziglang.org https://ziglang.org
- dnautics 7y ago
  hey eggy I didn't know you were interested in zig as well, drop a line, contact's in my about profile.

280 points by raphlinus 7y ago

This is a talk I gave at Jane Street late February on GPU compute, especially using graphics API's such as Vulkan and the upcoming WGPU. Feel free to scan through the slides [1], and to ask me anything. [1]: https://docs.google.com/presentation/d/1FRH81IW9RffkJjm6ILFZ7raCgFAUPXYYFXfiyKmhkx8/edit?usp=sharing https://docs.google.com/presentation/d/1FRH81IW9RffkJjm6ILFZ...

Jhsto 7y ago
Thanks for the video and slides, super interesting! One question: in one of the slides, you mention the Vulkan 1.2 / SPIR-V 1.5 Vulkan Memory model. What exactly is this and why do you think this is meaningful? Would you have any links to learn more? When I read the SPIR-V specification, it mentions that this kind of memory model exists, but it does not put any effort into explaining why it's worthwhile or how it works on a high level.
- jabl 7y ago
  Perhaps in a few years when we'll start to see things like CXL providing cache-coherency between main memory and GPU memory this will become important?
- raphlinus 7y ago
  That could be the subject of a whole other talk :) The memory model is needed to do more sophisticated communication between the parallel workgroups. The classic example is to make a queue, even a simple one such as a simple single-producer single-consumer ring buffer (for communication between a pair of workgroups). The way this works, the producer fills an element in the buffer, then bumps an index. The consumer observes the index, then reads the element from the buffer. Without a memory model, there's no way to guarantee that this ordering is preserved, and (to map this to hardware), the write of the index bump could flow through the caches, while the write of the element could still be "dirty" in the cache on the producer, so the consumer reads stale data. What the memory model does is provide the programmer an explicit way to express ordering constraints. So the write of the index bump is with "release" semantics and the read of that by the consumer is with "acquire" semantics, which guarantees that producer writes before the release are visible to consumer reads after the acquire. This unlocks a whole range of sophisticated concurrent data structures, such as concurrent hash maps, fancier memory allocation (even without a memory model, a simple allocate-only bump allocator is feasible), and so on. Thanks for the question and kind words, I hope this illuminates.
  - Jhsto 7y ago
    Thanks, the answer with some examples was exactly what I was hoping to get answered! Thank you for taking the time to respond.
throwlaplace 7y ago
of the links you put on the last page (or other gpu languages you might not have included) what's the most mature way to write decent gpu code right now (even if not necessarily optimal). i'm right this moment working on a project where i'm trying to speed up a blob detection algorithm by porting it to gpu. i'm using pytorch because the primitives fit naturally but if that weren't the case i think i would be kind of at a loss for how to do it (i do not know, as of yet, how to write cuda kernels). btw the Julia link is dead
- raphlinus 7y ago
  It depends on what you're trying to do. If you want to get stuff done now then just get an Nvidia card and write CUDA. The tools and knowledge community are vastly ahead of other approaches. If you're doing image-like stuff, Halide is good, there's lots of stuff shipping on it. One other project I'd add is emu, which is built on wgpu. I think that's closest to the future I have in mind, but still in early stages. https://github.com/calebwin/emu https://github.com/calebwin/emu
  - throwlaplace 7y ago
    okay in that case can you recommend a good tut for writing cuda code?
    - alexhutcheson 7y ago
      The official docs are pretty good. This is probably a good starting point: https://devblogs.nvidia.com/even-easier-introduction-cuda/ https://devblogs.nvidia.com/even-easier-introduction-cuda/ Nvidia also sponsored a Udacity course, but I don't have any direct experience with it: https://developer.nvidia.com/udacity-cs344-intro-parallel-programming https://developer.nvidia.com/udacity-cs344-intro-parallel-pr...
- ZeroCool2u 7y ago
  If you're interested in getting started writing GPU code and you're currently dealing with PyTorch a good place to start might be using NUMBA for CUDA programming. It lets you do in Python what you'd normally do in C/C++. It's great, because setup is minimal and it has some good tutorial materials. Plus, you're mostly emulating what you'd end up doing in C/C++, so conceptually the code is very similar. When I was at GTC 2018 this is one of the training sessions I went to and you can now access that here[1]. The official NUMBA for CUDA docs are great too[2]. Also, you can avoid a lot of installation/dependencies headaches by using anaconda/miniconda if you're doing this stuff locally. [1]: https://github.com/ContinuumIO/gtc2018-numba https://github.com/ContinuumIO/gtc2018-numba [2]:https://numba.pydata.org/numba-doc/latest/cuda/overview.html https://numba.pydata.org/numba-doc/latest/cuda/overview.html
sorenjan 7y ago
Do you think other alternatives to Cuda will have the same drawback as OpenCL, that Nvidia won't support it properly and therefore not be a viable option on their hardware which is dominating the market at the moment?
- gmueckl 7y ago
  Well, what are the chances that less vendor lock in is in nVidia's interest?
  - angry_octet 7y ago
    Is it vendor lock-in if the clients want to be there? It's more like you can be in the club with the good music and hip people, or the club with the average unknown DJ and an empty dance floor.
    - bl0b 7y ago
      That makes sense for GPU-compute-in-the-cloud applications where you can pick the hardware and don't care if other hardware doesn't work, but some people want to write programs that use GPGPU that can run on anyone's machine.
- raphlinus 7y ago
  No, precisely because I see the emerging alternatives as being based on Vulkan and SPIR-V, which I believe have momentum far greater than OpenCL. The only question is whether a compute platform based on these standards and open source tools could be anywhere nearly as good as CUDA. I believe we're far from that today, but it is possible and there's a lot of good work in that direction.
  - muro 7y ago
    Doesn't that exclude Apple?
    - 7y ago
sprash 7y ago
Halfway through you were bringing up hopes for a great new way to do 2D rendering on GPUs but then there is no show of actual implementation. Rather anticlimactic! Even decades old Cairo is beating VulKan/GPU based renderers like Skia in some tasks. Can it be that hardware intended for ML or 3D-graphics is not suited at all for 2D rendering (parallelism vs. pipelining)? How will WebGPU solve those problems? To me this talk raises more questions than it answers...
- raphlinus 7y ago
  It's work in progress. I was hoping to get more done on the renderer but various things have been slowing me down. But lately I've been feeling a new burst of energy, so hope to have something more to show before long. I agree that the talk raises a lot of questions. Hopefully that's a good thing, as I think they deserve the attention of curious people! Regarding the mismatch between 3D hardware and 2D rendering, we'll need hard empirical numbers, but the approach I'm taking is basically to write a software renderer (not unlike Cairo or even libart which predates it), just one that's designed for massive parallelism so it can run on the GPU. This will have performance characteristics more similar to Cairo (just faster as so much more throughput is available) than rasterization-based approaches. The latter really fall down when there's soft clipping and lots of blending, as that requires generation of intermediate texture buffers. I do address this in the talk :) I hope I will have more solid answers for some of these questions before long.
  - sprash 7y ago
    I'm experimenting with GPU based 2D rendering using signed distance functions as drawing primitives for quiet a long time now. My experience is that as soon as the object tree exceeds a certain size the CPU will always beat the GPU. The only way the GPU can win is when you have high complexity drawing primitives and a low amount of objects which is exactly the opposite one usually encounters in the real world. Maybe the partitioning approach you are proposing will help but only real world implementations and benchmarks will tell.
    - 7y ago
zozbot234 7y ago
Is Vulkan really becoming viable for compute? It seems like something that's obviously worth exploring given the overlap with more traditional graphics workloads, but I have seen some claims that it's still lagging behind OpenCL wrt. some features, and that it could be good for some things but is not obviously competitive across the board. It's possible that this impression is outdated though.
- raphlinus 7y ago
  I think the answer to this is emerging, but I would say yes. Vulkan and SPIR-V are rapidly gaining features, while OpenCL is stagnating. Some of these features are very recent, like pointers and a memory model. Another area I'm expecting Vulkan to pull ahead is support for efficient float16 computation, which is important for ML workloads, and is increasingly well supported in hardware. One thing to keep in mind though, programming directly in Vulkan is too low-level for most users, while OpenCL is a fairly friendly API. We need tools that can compile to SPIR-V and deploy on Vulkan, but high enough level you can express your problem rather than (for example) manually placing pipeline barriers between dispatches. The lack of such tools, and having to build some of my own, is one of the reasons for the slowdown in progress on my renderer project.
  - Jasper_ 7y ago
    CUDA is unfortunately very, very far ahead of Vulkan. "Graphics Compute" is quite different from CUDA at the advanced levels if you've never used it.
  - pjmlp 7y ago
    In CUDA I can just use ISO C++17 (C++20 around the corner) plus CUDA extensions, or any other language that targets PTX, with very nice graphical debugging from what is going on the GPGPU. OpenCL is playing catch up with CUDA, but it requires at least version 2.0 for similar capabilities, while many vendors are still shipping 1.x drivers. Vulkan, well it seems to be the same story as OpenCL regarding compute tooling, but lets see how much love it will get from NVidia. I advise watching the GTC 2020 talks regarding CUDA and Vulkan compute.
Symmetry 7y ago
Any opinion on ROCm?
- mpreda 7y ago
  ROCm provides a good OpenCL 2.x implementation targeting AMD GCN. The compiler/optimizer is clang-based. It is open-source. One great thing is that ROCm OpenCL supports GCN assembly in the OpenCL source, which is handy in some corner-cases that are not properly handled by the optimizer.
- VHRanger 7y ago
  Doesn't support many platforms
Strilanc 7y ago
What is a reasonable lower limit on latency that I could expect from a GPU pipeline? For example, I know that if I use WebGL and issue a `readPixels` call then that seems to take on the order of 1 to 10 milliseconds no matter what I do. This suggests that if I needed to react to something in under 100 microseconds, I probably shouldn't have the computation of the reaction involve WebGL. But is that true of GPU compute in general, or just an artifact of the abstraction exposed by WebGL?
- jdashg 7y ago
  There's a bunch of intractable latency, but there are patterns that can help maintain pipelining, if you don't need the data immediately: https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/WebGL_best_practices#Non-blocking_async_data_downloadreadback https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/W...
  - Strilanc 7y ago
    Right, but if I go closer to the metal how much of the latency is really truly intractable? 100 microseconds? 1 millisecond? 10 milliseconds?
    - bl0b 7y ago
      Can't speak for WebGL APIs, but in OpenGL (and other C++ graphics/GPGPU platforms), it is certainly possible (and necessary) to write fully interactive real-time apps that rely on a few rounds of communication between GPU and CPU code every frame. At 60 fps, you have 16 ms per frame. As long as you aren't copying huge arrays back and forth, I'd say you can budget in the 1 ms order-of-magnitude per data transfer. I don't know much of that would be avoidable if you were actually closer to the metal than the raw OpenGL C++ API call
- kllrnohj 7y ago
  You do have to design around the GPU typically being almost like a server. It's very fast, but also "very far" away - both in latency & in bandwidth. PCI-E 3.0 x16 is 'only' 16GB/s. That's fast for I/O, but still in the ballpark of being like I/O & not like RAM. And that's if you even get x16 electrical - x8 for GPUs is pretty common in eg. laptops where the other lanes go to things like thunderbolt 3. Integrated graphics are a different story, they often just have a direct connection to DRAM and CPU<->GPU communication is therefore super fast & low latency. In theory, anyway, if the API abstractions even let you share memory between the two. So that means even for things where GPUs should do really well at it, like summing two arrays together, doing it on the CPU can still be a lot faster if the inputs & outputs are local to the CPU.
2008guy 7y ago
What were you talking about with the socialist revolution?
- mkl 7y ago
  He was giving a talk to people from Jane Street [1], who make money on the stock market using OCaml (mentioned in the same bit). [1] https://en.wikipedia.org/wiki/Jane_Street_Capital https://en.wikipedia.org/wiki/Jane_Street_Capital
  - 2008guy 7y ago
    Right but he basically asserted to everyone that a socialist revolution is coming to wipe them out. Is there something I don’t know? Should I be getting ready for this?
    - mkl 7y ago
      Huh? It was a joke. Explaining the joke will kill it, but here goes: He was saying that their OCaml skills would still be useful even if they couldn't make money in finance because capitalism fell over. The audience got the joke and laughed. It's nothing like what you're saying.
      - 2008guy 7y ago
        What I saw was this guy flatly asserting that a socialist revolution is coming, I thought at first he was talking about Bernie Sanders, and where there should be a punchline there is no punchline. Then everyone gives an awkward and half-cocked chuckle as one does when presented with something so bizarre.
fluffything 7y ago
I write a lot of compute kernels in CUDA, and my litmus test is prefix sum, for two reasons. First, you can implement it in pure CUDA C++, and max out the memory bandwidth of any nvidia or AMD GPU. The CUB library provides a state of the art implementation (using decoupled-lookback) that one can compare against new programming languages. Second, it is one of the most basic parallel algorithm building blocks. Many parallel algorithms use it, and many parallel algorithms are "prefix-sum-like". If I am not able to write prefix sum from scratch efficiently in your programming language / library, I can't use it. Every time someone shows a new programming language for compute, the examples provided are super basic (e.g. `map(...).fold(...)`), but I have yet to see a new programming language that can be used to implement the 2-3 most fundamental parallel algorithms from any parallel algorithms graduate course. For example, Futhark provides a prefix sum intrinsic, that just calls CUB - if you want to implement a prefix-sum-like algorithm, you are out of luck. In WGPU, it appears that prefix-sum will be an intrinsic of WHSL, which sounds like you would be out-of-luck too. You mentioned WGPU and Vulkan. Do you know how to implement prefix-sum from scratch on these? If so, do you know how the performance compare against CUB?
- raphlinus 7y ago
  That sounds like a great project, one I heartily recommend to people wanting to get their feet wet. First, the basic principles of how to write an efficient prefix sum are the same as before. Definitely read [1]. Also see [2] for an exploration into how much subgroups help over threadgroup shared memory. In fact, [2] has much of the answer you seek, as it's already written in Vulkan and GLSL (compiling to SPIR-V), though it does miss a direct performance comparison to CUB. Second, doing this depends on WebGPU exposing threadgroup shared memory and subgroups operations. I'm not sure where the standard and implementations are on this; there are significant challenges. For example, subgroup shuffle is not available in DX12, though subgroup reduce operations such as WavePrefixSum are. So in general an implementation will need to do runtime detection of capabilities and choose a most-optimum kernel based on that. In theory, it should be possible to achieve comparable performance. But it's likely that WebGPU implementations haven't really been tuned for performance yet, while CUB has seen a ton of work. So, as always, the thing to do is implement and measure. [1]: https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-co... [2]: https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-GPU https://cachemiss.xyz/blog/parallel-reduce-and-scan-on-the-G...
m463 7y ago
When I heard the definition SIMT, it helped me understand the if / do_a / do_b concept. But your subsequent explanation helped understand what went beyond that. Thank you for the talk.
unixhero 7y ago
Really good talk, thanks a lot.

moritonal 7y ago

Great technical talk, but an hour long video of one of the more visually appealing techs and not a single demo :( Demo's like this (https://paveldogreat.github.io/WebGL-Fluid-Simulation/ https://paveldogreat.github.io/WebGL-Fluid-Simulation/) help show people why many small processors can beat a few large ones.

raphlinus 7y ago
Noted, and I'll do my best to raise up demos for future presentations.
- rudedogg 6y ago
  I thought you did a really nice job with the slides. The diagrams made it easier to follow along.
- moritonal 6y ago
  Sorry, the other replies to this are completely correct that you should know your audience and flashy demos might also be a turn-off. This was a fantastic technical demo, I simply work in graphics so am biased towards that.
- doublesCs 7y ago
  As a counter-point to the above commenter, I'm someone who's in an adjacent industry (HFT) and to me personally spending time on pretty pictures is a turn off. So I guess the take away is: know your audience. I really really enjoyed your presentation, thanks for sharing.

streb-lo 7y ago

I'm not well versed in this area at all -- but something I find fascinating is what I have heard of the early days of GPU compute. Before CUDA and the like, you would apparently take your data you wanted to compute on, structure it as an image/texture and use the graphics API to ham fist your way to the result you wanted. No idea if that's true, but pretty neat if it is.

gh123man 7y ago
This is actually still true in a lot of cases where proper compute shaders are not available - notably webGL. See: https://threejs.org/examples/webgl_gpgpu_birds.html https://threejs.org/examples/webgl_gpgpu_birds.html and: https://github.com/yomboprime/GPGPU-threejs-demos https://github.com/yomboprime/GPGPU-threejs-demos Though WebGPU is coming which I believe will mitigate this.
Qasaur 7y ago
As the other comment mentioned, this is still true if you want to support legacy graphics APIs. A non-negligible amount (~20%) of Android phones cannot execute compute pipelines as they only support OpenGL ES 3.0 - compute shaders were introduced in GLES 3.1. While it is an headache if you want to support compute on legacy devices, I do think that writing regular vertex/fragment shaders for general purpose GPU computation is an underrated pleasure as you really need to break out the coding golf toolkit to squeeze out maximum performance.
- gh123man 7y ago
  To add to this - it's not just legacy/Android devices. Since Apple dropped support for OpenGL in favor of metal, cross platform OpenGL compute shaders are now impossible since iOS will never get OpenGL ES 3.1. This has caused headaches for me as I wrote a cross platform game and am now pinned to GLES 3.0 forever.
raphlinus 7y ago
That works well for certain kinds of computations, basically where each thread can compute a "pixel" independently. Modern GPU compute has threadgroup local storage, subgroups, atomic operations, etc. These unlock huge categories of workloads that would not fit nicely in the rasterization pipeline.

londons_explore 7y ago

When someone finds a way to use a GPU to accelerate all the code in massive react webapps, that will be the day GPU's turn from a specialists-only device to mainstream...

SQueeeeeL 7y ago
Are GPUs specialist devices now?
- londons_explore 7y ago
  Programming them is a specialist task... I'd take a guess <10% of programmers have ever written CUDA or something equivalent.
  - vmchale 7y ago
    You can use something like Futhark and link it with your program.

maleadt 7y ago

Great talk! Any thoughts on Intel's oneAPI?

raphlinus 7y ago
Thanks. First I've heard of it, in fact, so I don't have any thoughts on it. It's interesting though, and would definitely be worth adding to a list of attempts to build portable layers.
- maleadt 7y ago
  I'm looking at targeting it from Julia, and the lower-level (Level Zero) API seems rather nice, resembling the CUDA driver API but building on SPIR-V. It's also nice how the API is decoupled from the implementation, so let's hope more vendors implement it (apparently Codeplay is working on a CUDA-based implementation).

vmchale 7y ago

My experience with Futhark (and to some extent accelerate) is that they're not terribly hard to program in applicable cases. I'm not sure Futhark-generated code is as fast as specialist code but it's definitely a speedup compared to even skilled CPU implementations.

vmchale 7y ago

Question: what is the state of GPU compute w.r.t. Rust? Are the libraries on par with accelerate or (not exactly comparable) Futhark?

raphlinus 7y ago
There's a ton of low and mid level infrastructure for GPU being written in Rust. That includes the gfx-hal work, the wgpu implementation, and more experimental languages such as emu. But it's still early days, and I don't think there's a complete solution comparable to either of those yet.
VHRanger 7y ago
Expect a lot of rough patches

jklinger410 7y ago

https://www.google.com/search?q=how+to+pronounce+arbiter&oq=how+to+pronounce+arbiter&aqs=chrome..69i57j0l4.2632j0j1&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=how+to+pronounce+arbiter&oq=...

seanalltogether 7y ago

Is there a meaningful speed difference between creating a 2d drawing engine using "gpu compute" vs going through normal libraries like directx or opengl? I'm fairly ignorant around what is exposed in the compute libraries vs traditional drawing libraries.

hellofunk 7y ago
I think the main issue is where the output from the GPU shader is going. OpenGL for example, the output is the screen. That’s the only output. But for compute, you want the output to come back into the CPU world for storing or manipulating or etc. So compute shaders provide that API for shuffling memory back-and-forth between CPU and GPU and having access to general purpose output from the shader beyond just screen pixels.

geokon 7y ago

Long video, so I haven't watched it all yet, but I have a bit of a naiive question: Does Vulkan and WebGL/WebGPU "fall back" to running on the CPU if "appropriate" hardware isn't available? Or are developers required to maintain separate code paths based on hardware? I remember considering rewriting some tight loops in OpenCL but then the maintenance headache of having multiple code paths made the refactor seem not worth it. I'd guess this is/was generally a major speedbump for adoption. I know there is POCL which will run your kernels on the CPU, but it's not something you can't expect available on every platform. Maybe if POCL was part of the kernel or bundled with every distro then the situation would be different. I've seen some project do compute in OpenGL ES 2.0 b/c that's ubiquitous and will always run (I think it's even a requirement for Android systems)

raphlinus 7y ago
This is a good question. No, Vulkan and WebGPU will not fall back, they require hardware and driver support. What I think you're looking for is a higher level layer that will target GPU compute if it's available, otherwise CPU (or possibly other resources). Projects in this space include Halide and MLIR.
- geokon 7y ago
  I have a more theoretical followup then :) So you spend a lot of time in the talk massaging your sequential CPU task into an appropriately GPU task. It's clearly tricky and involves reworking your algorithms to leverage the parallelism and the memory "features" of the GPU. But through rewriting, have you actually substantively hurt the performance of the sequential program? The big picture question is : is the reverse problem of going from a GPU program to a CPU problem ever problematic? I mean in a algorithmic sense - without looking at micro optimizations for leveraging CPU SIMD instructions or whatever. Or are you always going to be pretty okay with running your shaders sequentially one after another on your big luxurious CPU memory pool? and ultimately is there anything stopping you from compiling SPIR-V for a CPU? Could you not autgenerate in the kernel dispatcher just a switch that'll branch and run a precompiled on-CPU kernel if no driver is found? Then you'd finally really get compile-once-run-everywhere GPU code I guess since it's not being done then I'm missing something haha :) Maybe you are going to often hit scenarios where you'd say "No, if I'm going to run this on the CPU I need to fundamentally rewrite the algorithm"

jarrell_mark 7y ago

Tensorflow.js (tf.Tensor) can be used as an easy to use interface to GPU compute (similar API to numpy) which works on all gpus not just Nvidia because it's backed by webgl https://www.tensorflow.org/js/guide/platform_environment https://www.tensorflow.org/js/guide/platform_environment

01100011 7y ago

FWIW, if you're interested Nvidia-centric GPGPU stuff, Nvidia's tech conference went online-only this year and is free for all: https://www.nvidia.com/en-us/gtc/ https://www.nvidia.com/en-us/gtc/

ptrenko 7y ago

Just a question: Has anyone figured out serverless ml predictions yet?

thomas232233 7y ago

Any youtube video that has disabled comments is most likely a sign of unwilling to hear criticism and fearing i might waste my time i don't take that chance.

amelius 7y ago

I wonder at what point we finally decide that "G" in GPU doesn't make sense, and instead we start building compute modules that don't have 8 graphics connectors, and don't mess with graphics apis, and most importantly have open source drivers.

gbrown 7y ago
Except for that last part, you’re actually describing a return to the way things were decades ago, when accelerator cards of various types were common.
avianlyric 7y ago
Most “GPU”s found in datacentres don’t have any graphic connectors on the back of them. They do all their communication via PCIe. In some cases they may have a connectors for high speed communication fabric between many GPUs. NVLink is an example of this [0]. Outside of the consumer and workstation space GPUs really don’t look like anything you or I would recognise as a GPU anymore, and with APIs like CUDA, don’t look like GPUs to software either. Really all we’re missing is open source drivers. But I wouldn’t hold your breath. [0] https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/

eggy 7y ago

I loved his commentary on Co-dfns for APL at 54:38 in the video. I program in J[0], and dabble in Dyalog APL. Aaron Hsu's work on the gpu compiler, he's the implementor of Co-dfns, is nothing short of mind-blowing[1]. His arguments for using APL vs. more mainstream languages are cogent and persuasive. The paradigm shift from C-like or even Haskell (ML) type languages is too big a gap for most mainstream programmers to accept. My opinion is that aside from Python's clean, indented syntax, the real reason it took off for DL/ML, was Numpy and Pandas. Pandas was heavily influenced by the array languages per its creator Wes McKinney. Besides, I just like playing with J. I have looked briefly at Futhark, and the apltail compiler[2], but I am trying to focus on the APL family, because my time is limited. I am a GPU dilettante who has tried basic CUDA in C. I tried Rust, but Zig [3] is working out better for me given I make more progress with less effort, using C APIs are effortless, but my Rust difficulties may just be my bias with PLs. I find Haskell easier than Rust. I just read an HPC article today about a survey/questionnaire about the languages used among 57 experts. It's still predominantly C++, but with a lot pain expressed by the experts/users. I agree SPIR-V sounds promising, and I hope to check it out. Just like DL, I think people don't realize it needs to be knowledge domain and algorithms, or experts and software. Somebody has to setup the computation based on knowledge domain and software knowledge. This shows itself to me when I wind up running somebody's program, because I don't have my own specific problem I'd like to try and implement as a learning exercise and knowledge exercise. Great talk! I found it very understandable and paced just right! [0] https://jsoftware.com https://jsoftware.com [1] https://scholarworks.iu.edu/dspace/handle/2022/24749 https://scholarworks.iu.edu/dspace/handle/2022/24749 [2] https://futhark-lang.org/blog/2016-06-20-futhark-as-an-apl-compiler-target.html https://futhark-lang.org/blog/2016-06-20-futhark-as-an-apl-c... [3] https://ziglang.org https://ziglang.org

dnautics 7y ago
hey eggy I didn't know you were interested in zig as well, drop a line, contact's in my about profile.