7 ms·
Web AI Model Testing: WebGPU, WebGL, and Headless Chrome
- FL33TW00D 3y agoThis can also be done in Rust using the excellent `wasm_bindgen_test`!
- not_a_dane 3y agoAFAIK, there is still a memory barrier in chrome which is set to 4gb per tab.
- abxytg 3y agoI hate it so much. So arbitrary and capricious. I would say this is currently the number one blocker for the web as a serious platform. And they're doing it on purpose.
- vicktorium 3y agowhat apps can't run on 4GB? games? 3D? Editing? have you tried forking chrome and increasing this limit?
- KeplerBoy 3y agoSome AI apps. You can't really load a capable LLM in 4 GB. Or does this limit not apply when dealing with WASM and WebGPU?
- paulgb 3y agoVideo editors are a big one. I've heard of people crashing a browser tab with Figma as well. For data exploration tools it's very easy to want to use 4GB+ of memory. I found the limit cumbersome while working on financial tools. It usually comes up in internal tools where you reliably have a fast internet connection; it's harder to reach the limit for public-facing tools because there the slowness of sending 4GB+ to the browser is the more limiting factor. The annoying part isn't just that the limit is there, but that you can't really handle it gracefully as the developer -- when the browser decides you've hit the limit, it may just replace the page with an error message.
- 10000truths 3y agoFor a video editor, only a small portion of the video needs to be in memory at any given time. The rest can be squirreled away in an IndexedDB store, which has no hard size limits on most browsers.
- jjcm 3y agoIt's one of our big barriers over at Figma. Creative tools in general hit this limit pretty quickly. For context, I was a very heavy user of Photoshop back in the day. Even a decade ago I remember hitting 20GB of active memory use for Photoshop. Things get really big really quick, especially when you're storing uncompressed versions of raster elements in memory. To frame things in a different way, 4GB is 22 seconds of 1080p video if you're loading the raw frames into memory.
- whatshisface 3y agoI guess the policy is that tabs can use 100% of the available resources on low end devices, but only 10% of the available resources on high end devices.
- skybrian 3y agoI think the desktop policy might be better. In the tablets I've used, tabs sometimes get killed when I switch tabs and visit another website with a lot of ads. It's an annoying way to lose work in an unsubmitted form. It doesn't seem to happen for desktop.
- NavinF 3y agoThat's because most (all?) phones don't swap to a pagefile whereas every desktop OS has swap enabled by default. The only practical solution is to buy a phone with more memory. IMO 6GB is the bare minimum in 2024
- mkesper 3y agoNo, just use an ad blocker and save the environment from more waste.
- NavinF 3y agoAd blockers certainly shave off a few MBs, but in my experience the vast majority of RAM usage is not caused by ads. Unlike first party content, ads are automatically benchmarked by ad exchanges and penalized for using too much resources. I also don't think a 200gram phone is the kind of waste that we should be concerned about. Think bigger
- PoignardAzur 3y ago> I also don't think a 200gram phone is the kind of waste that we should be concerned about. Think bigger A lot of raw ore is processed to get those 200 grams in your hand. That said, a quick search tells me the carbon footprint of producing a phone is around 55kg, which is about 320km of car travel; it's not trivial, but it's not as much of a bottleneck I thought it might be.
- amelius 3y ago4GB ought to be enough for anybody.
- jsheard 3y agoAt least on desktop you generally know where the line is, on mobile there's a mystery limit you're not allowed to cross, and you're also not allowed to know where the line is until you reach it, which might gracefully throw an error or might result in your tab being force-killed, and you're not allowed to know which of those will happen either.
- fragmede 3y agoMobile Safari just has a fixed limit of 500 tabs
- Me1000 3y agoIf you switch to private browsing mode you can get an extra 500 tabs. :)
- wlesieutre 3y agoWhat about other tab groups?
- paulgb 3y agoThis is a user-facing limit; jsheard is talking about how as an app developer you don't know whether your app is below the limit or whether the next allocation will kill the browser tab.
- echelon 3y agoI'm building a list of "second class citizen" mobile web issues for Android and Apple. I wasn't aware of this one! Do you know of anything else like this?
- jsheard 3y agohttps://github.com/WebAssembly/design/issues/1397 https://github.com/WebAssembly/design/issues/1397 > Currently allocating more than ~300MB of memory is not reliable on Chrome on Android without resorting to Chrome-specific workarounds, nor in Safari on iOS. That's about allocating CPU memory but the GPU memory situation is similar. The specs don't want to reveal information about how much memory you're allowed to use because it could be used for fingerprinting, but that means that it's practically impossible to build reliable applications which use (or can optionally use) a lot of memory. Every allocation you make past a few hundred MB risks blowing up the app immediately, or putting it into the danger zone where it's the first in line to get killed when running in the background, either way without any warning or last-chance opportunity to release memory to avert getting killed.
- FL33TW00D 3y agoThis is a 7B parameter model at int4, lots to play with!
- jmayes 3y agoHello there, I am one of the authors of the piece. Fun fact just for the lols we have tried running a 1.3B parameter unoptimized TensorFlow.js model in this system just to see if it would work (could be much more memory efficient with tweaks), and it does. It uses about 6GB RAM and 14GB VRAM when using V100 GPU on Colab (15GB VRAM limit) but runs pretty fast otherwise once the initial load is complete! Obviously plenty of room to make this use much less memory in the future - we just wanted to check we could run such things as a test for now.
- sylware 3y agoIsn't that exactly the modern, AI based, mouse and keyboard BOT? (trained with click farms)
- refulgentis 3y agoReal, but naive, question: does TensorFlow have meaningful share outside Google? I've been in the HuggingFace ecosystem and it's overwhelmingly PyTorch, IIRC 93%, (I can't find the blog post that said it, but only gave it 2 minutes)
- nwoli 3y agoBest alternative for web imo (perf generally beats onnx for web)
- hatthew 3y agoTF used to be the most popular framework by a large margin, so a lot of things that were started 5+ years ago are still on it. PyTorch is most popular in places that only started more recently or have the ability to switch easily, e.g. new startups, research, LLMs, education, and companies that have the resources to do a migration project.
- summerlight 3y agoA fun thing is that even in Google JAX is now preferred across researchers and slowly taking over the share.
- ZeroCool2u 3y agoTbf JAX is super nice. Even easier than PyTorch in many ways and astonishingly fast. XLA is super powerful.
- leeeeeepw 3y ago[flagged]
- antimora 3y agoGreat! For Burn project, we have WebGPU example and I was looking into how we could add automated tests in the browser. Now it seems possible. Here is the image classification example if you'd like to check out: https://github.com/tracel-ai/burn/tree/main/examples/image-classification-web https://github.com/tracel-ai/burn/tree/main/examples/image-c...
- lxe 3y agoI think better SIMD support for webassembly is more inclusive than relying on / expecting WebGPU
- jmayes 3y agoFor this blog post we are using Chrome for the testing environment which has WebGPU turned on by default now and other common browsers should hopefully follow suit, but given we are using Chrome here we know WebGPU will be available if the WebAI is using that (which many people are turning to for diffusion models and LLMs as its so much faster to run those types of models). But yes, I am all for better support on all the things too, we have many WASM users too, and when anything new comes out there, this set of instructions can still be used to leverage testing that too as its just Chrome running on Linux essentially with the right flags set.
- NavinF 3y agoCPU inference is 10x slower. Not good enough for most use cases
- tcper 3y agoEverything will run on a browser, eventually
- LoveMortuus 3y agoSince the browsers work as a sort of unified experience (to some extent) that would be quite good. But sadly, I haven't seen the wide adoption of PWA or similar technology. Most usually just create their own app, which in many cases really isn't even needed, since the app is just a wrapped version of their website.
- rubatuga 3y agoHopefully this will solve some of the incompatibility with training models on AMD vs NVIDIA. Just use Google Chrome.
- dblitt 3y agoReminds me of “The Birth & Death of JavaScript” https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...
- bhakunikaran 3y agoA question that comes to mind is: How significant is the performance difference between using CPUs and GPUs for these machine learning models in web applications, and are there specific types of applications where one would significantly outperform the other?
- FL33TW00D 3y agoVery significant in the current paradigm.