18 ms·
So it's loading an instruct model for inference? That seems a fair bit less useful than a base model, at least for more advanced use cases. What about running
by SquareWheel 2y ago
So it's loading an instruct model for inference? That seems a fair bit less useful than a base model, at least for more advanced use cases.
What about running LoRAs, adjusting temperature, configuring prompt templates, etc? It seems pretty early to build something like this into the browser. The technology is still changing so rapidly, it might look completely different in 5 years.
I'm a huge fan of local AI, and of empowering web browsers as a platform, but I'm feeling pretty stumped by this one. Is this a good inclusion at this time? Or is the Chrome team following the Google-wide directive to integrate AI _everywhere_, and we're getting a weird JS API as a result?
At the very least, I hope to see the model decoupled from the interface. In the same way that font-family loads locally installed fonts, it should be pluggable for other local models.
- niutech 2y agoThe base model can be found on HF (https://huggingface.co/wave-on-discord/gemini-nano https://huggingface.co/wave-on-discord/gemini-nano) and run in any web browser using MediaPipe on WebGPU: https://x.com/niu_tech/status/1807073666888266157 https://x.com/niu_tech/status/1807073666888266157 As for temperature and topK, you can set them in the AITextSessionOptions object as an argument to `window.ai.createTextSession(options)` (source: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/ai/ai.h;drc=0b5079a88833a5f8b6e4ae9f14bbf85e49dfff70;l=48 https://source.chromium.org/chromium/chromium/src/+/main:thi...) You should also be able to set it by adding the switches: `chrome --args --enable-features=OptimizationGuideOnDeviceModel:on_device_model_temperature/0.5/on_device_model_topk/8` (source: https://issues.chromium.org/issues/339471377#comment12 https://issues.chromium.org/issues/339471377#comment12) The default temperature is 0.8 and default topK is 3 (source: https://source.chromium.org/chromium/chromium/src/+/main:components/optimization_guide/core/optimization_guide_features.cc;drc=f1b23d439be181e78a30cb23a955a95bd50f1702;l=827 https://source.chromium.org/chromium/chromium/src/+/main:com...) As for LoRA, Google will provide a Fine-Tuning (LoRA) API in Chrome: https://developer.chrome.com/docs/ai/built-in#browser_architecture_and_apis https://developer.chrome.com/docs/ai/built-in#browser_archit...
- SquareWheel 2y agoAppreciate the info and links.