3 ms·
How do they load 70B weights with the ~4GB heap limit of JS/WASM?
by modeless 3y ago
How do they load 70B weights with the ~4GB heap limit of JS/WASM?
- deleted 3y ago[deleted]
- andy_ppp 3y agoIt uses WebGPU which I’m guessing is allocated differently than the 4gb limit?
- modeless 3y agoWebGPU doesn't ingest data directly, it always goes through JS/WASM. Presumably they are streaming the data in small chunks through into WebGPU. I could imagine several ways of doing it, but sometimes in browsers, things that ought to work don't, so a demonstrated working method would be interesting to know about, if someone can locate the actual code that does it. (Also, it's pretty silly that you can allocate far more VRAM than RAM on the web platform.)