3 ms·

CUDA is better for sure, but the pure functional nature of the traditional shader is conceptually much simpler and I kind of relish the simplicity. There is no

CUDA is better for sure, but the pure functional nature of the traditional shader is conceptually much simpler and I kind of relish the simplicity. There is no crazy tiling or anything. Just per-pixel parallelism [1]. It won't be as fast as those real, highly-tuned kernels, but it's still nice to see something simple that does something non-trivial. It reminded me of the early "GPGPU" days (early 2000s?) [1] https://github.com/nathan-barry/gpt2-webgl/blob/main/src/gpt2_webgl.ts https://github.com/nathan-barry/gpt2-webgl/blob/main/src/gpt...

3abiton 1y ago
> GPGPU I haven't heard this one in a long while
- glkindlmann 1y ago
  a decade does feel like an eternity here; I remember this book* when it came out * https://www.oreilly.com/library/view/gpgpu-programming-for/9781466595354/ https://www.oreilly.com/library/view/gpgpu-programming-for/9...
sigmoid10 1y ago
Fun fact: Long before the dawn of the GPU deep learning hype (and even before CUDA was a thing), a bunch of CS nerds from Korea managed to train a neural network on an ATI (now AMD) Radeon 9700 Pro using nothing but shaders [1]. They saw an even bigger performance improvement than Hinton and his group did for AlexNet 8 years later using CUDA. [1] https://ui.adsabs.harvard.edu/abs/2004PatRe..37.1311O/abstract https://ui.adsabs.harvard.edu/abs/2004PatRe..37.1311O/abstra...
- math_dandy 1y ago
  Cool, I had not heard about this. Adding this paper to my machine learning teaching bibliography. Even though the start of the deep learning renaissance is typically dated to 2012 with Alexnet, things were in motion week before that. As you point out, GPU training was validated at least 8 years previously. Concurrently, some very prescient researchers like Li were working hard to generate large scale datasets like ImageNet (CVPR 2009, https://www.image-net.org/static_files/papers/imagenet_cvpr09.pdf https://www.image-net.org/static_files/papers/imagenet_cvpr0...). And in 2012 it all came together.
  - sigmoid10 1y ago
    Since shaders were designed for, well, shading, this early experiment was more of an academic playground exercise than useful research. But AlexNet still wasn't the first deep neural network trained using CUDA. It had already been done three years earlier: https://dl.acm.org/doi/10.1145/1553374.1553486 https://dl.acm.org/doi/10.1145/1553374.1553486 The ImageNet competition had also been around since 2010. So the ingredients were actually all there before.

3abiton 1y ago

> GPGPU I haven't heard this one in a long while

glkindlmann 1y ago
a decade does feel like an eternity here; I remember this book* when it came out * https://www.oreilly.com/library/view/gpgpu-programming-for/9781466595354/ https://www.oreilly.com/library/view/gpgpu-programming-for/9...

sigmoid10 1y ago

Fun fact: Long before the dawn of the GPU deep learning hype (and even before CUDA was a thing), a bunch of CS nerds from Korea managed to train a neural network on an ATI (now AMD) Radeon 9700 Pro using nothing but shaders [1]. They saw an even bigger performance improvement than Hinton and his group did for AlexNet 8 years later using CUDA. [1] https://ui.adsabs.harvard.edu/abs/2004PatRe..37.1311O/abstract https://ui.adsabs.harvard.edu/abs/2004PatRe..37.1311O/abstra...

math_dandy 1y ago
Cool, I had not heard about this. Adding this paper to my machine learning teaching bibliography. Even though the start of the deep learning renaissance is typically dated to 2012 with Alexnet, things were in motion week before that. As you point out, GPU training was validated at least 8 years previously. Concurrently, some very prescient researchers like Li were working hard to generate large scale datasets like ImageNet (CVPR 2009, https://www.image-net.org/static_files/papers/imagenet_cvpr09.pdf https://www.image-net.org/static_files/papers/imagenet_cvpr0...). And in 2012 it all came together.
- sigmoid10 1y ago
  Since shaders were designed for, well, shading, this early experiment was more of an academic playground exercise than useful research. But AlexNet still wasn't the first deep neural network trained using CUDA. It had already been done three years earlier: https://dl.acm.org/doi/10.1145/1553374.1553486 https://dl.acm.org/doi/10.1145/1553374.1553486 The ImageNet competition had also been around since 2010. So the ingredients were actually all there before.