3 ms·

A few weeks back, I implemented GPT-2 using WebGL and shaders. Here's a write-up over how I made it, covering how I used textures and frame buffer objects to st

A few weeks back, I implemented GPT-2 using WebGL and shaders. Here's a write-up over how I made it, covering how I used textures and frame buffer objects to store and move around weights and outputs from calculations while using WebGL.

vessenes 1y ago
Request -- big bold link to a working web page right at the top? I read the page, I read your github, and I saw instructions to clone and run a node site, and was like "..nah" I think github pages will serve this up for free if you like. p.s. Cool!
- nathan-barry 1y ago
  Yeah, that would have been a good thing to set up. Main thing to add would be loading the weights into the browser
nathan-barry 1y ago
Here's a link to the github repo. At the top of the README it has a demo of GPT-2 running and the visualizations of the attention matrices and transformer block outputs Repo: https://github.com/nathan-barry/gpt2-webgl https://github.com/nathan-barry/gpt2-webgl
pjmlp 1y ago
Kudos for going down the WebGL route and not the Chrome only WebGPU approach that most likely some people expect. It is going to take at least yet another year, for WebGPU 1.0 to be available on stable versions on other browsers, Chrome still hasn't stable WebGPU on GNU/Linux, and it is already much far ahead with extensions that most likely won't be on the 1.0 MVP of the other browsers. Interesting article.
- erichdongubler 1y ago
  Member of Firefox's WebGPU team here. I'm curious what extensions you're referring to! We're hoping to ship on Windows stable in the next couple of Firefox releases. Other platforms should follow in the following year. If you want, I'd encourage you to try out WebGPU on Nightly on Linux and see if you run into any issues!
jokoon 1y ago
I also implemented Perlin noise using glsl shaders. It was a bit tedious to output a texture as float, but I did it. Although obviously it's not deterministic depending on the GPU.

vessenes 1y ago

Request -- big bold link to a working web page right at the top? I read the page, I read your github, and I saw instructions to clone and run a node site, and was like "..nah" I think github pages will serve this up for free if you like. p.s. Cool!

nathan-barry 1y ago
Yeah, that would have been a good thing to set up. Main thing to add would be loading the weights into the browser

nathan-barry 1y ago

Here's a link to the github repo. At the top of the README it has a demo of GPT-2 running and the visualizations of the attention matrices and transformer block outputs Repo: https://github.com/nathan-barry/gpt2-webgl https://github.com/nathan-barry/gpt2-webgl

pjmlp 1y ago

Kudos for going down the WebGL route and not the Chrome only WebGPU approach that most likely some people expect. It is going to take at least yet another year, for WebGPU 1.0 to be available on stable versions on other browsers, Chrome still hasn't stable WebGPU on GNU/Linux, and it is already much far ahead with extensions that most likely won't be on the 1.0 MVP of the other browsers. Interesting article.

erichdongubler 1y ago
Member of Firefox's WebGPU team here. I'm curious what extensions you're referring to! We're hoping to ship on Windows stable in the next couple of Firefox releases. Other platforms should follow in the following year. If you want, I'd encourage you to try out WebGPU on Nightly on Linux and see if you run into any issues!

jokoon 1y ago

I also implemented Perlin noise using glsl shaders. It was a bit tedious to output a texture as float, but I did it. Although obviously it's not deterministic depending on the GPU.