5 ms·
You can try it on Android right now: Download the Edge Gallery apk from github: https://github.com/google-ai-edge/gallery/releases/tag/1.0.0 https://github.com
by nolist_policy 1y ago
You can try it on Android right now:
Download the Edge Gallery apk from github: https://github.com/google-ai-edge/gallery/releases/tag/1.0.0 https://github.com/google-ai-edge/gallery/releases/tag/1.0.0
Download one of the .task files from huggingface: https://huggingface.co/collections/google/gemma-3n-preview-682ca41097a31e5ac804d57b https://huggingface.co/collections/google/gemma-3n-preview-6...
Import the .task file in Edge Gallery with the + bottom right.
You can take pictures right from the app. The model is indeed pretty fast.
- deleted 1y ago[deleted]
- nolist_policy 1y agoOkay from some first tries with story writing, gemma-3n-E4B-it seems to perform between plain Gemma 3 4B and 12B. It definitely retains the strong instruction following which is good. Hint: You have to set the Max tokens to 32000 for longer conversations. The slider makes it look like it's limited to 1024, just enter it manually.
- lousken 1y agowaiting for approval, is there a magnet?
- hadlock 1y agoif you go into the app and click the first icon it directs you to a workflow to get approved after clicking on a button that is the same color as the background and jump through some hoops about providing user data and analytics etc then it will auto-approve you
- KoolKat23 1y agoThanks for this guide it's great. Okay perhaps my phones not great and perhaps this isn't optimized/pruned for phone use but it's unusably slow. The answers are solid from my brief test. I wouldn't exactly say phone use, unless you have no internet and you don't mind a bit of a wait. Really impressive, regardless.
- ignoramous 1y agoAnd the libraries to embed Gemma-series in your iOS/Android app: https://ai.google.dev/edge/litert https://ai.google.dev/edge/litert Or, run them on a microcontroller! https://github.com/tensorflow/tflite-micro https://github.com/tensorflow/tflite-micro
- philipkglass 1y agoI assume that "pretty fast" depends on the phone. My old Pixel 4a ran Gemma-3n-E2B-it-int4 without problems. Still, it took over 10 minutes to finish answering "What can you see?" when given an image from my recent photos. Final stats: 15.9 seconds to first token 16.4 tokens/second prefill speed 0.33 tokens/second decode speed 662 seconds to complete the answer
- the_pwner224 1y agoI did the same thing on my Pixel Fold. Tried two different images with two different prompts: "What can you see?" and "Describe this image" First image ('Describe', photo of my desk) - 15.6 seconds to first token - 2.6 tokens/second - Total 180 seconds Second image ('What can you see?', photo of a bowl of pasta) - 10.3 seconds to first token - 3.1 tokens/second - Total 26 seconds The Edge Gallery app defaults to CPU as the accelerator. Switched to GPU. Pasta / what can you see: - It actually takes a full 1-2 minutes to start printing tokens. But the stats say 4.2 seconds to first token... - 5.8 tokens/second - 12 seconds total Desk / describe: - The output is: while True: print("[toxicity=0]") - Bugged? I stopped it after 80 seconds of output. 1st token after 4.1 seconds, then 5.7 tokens/second.
- the_pwner224 1y agoPixel 4a release date = August 2020 Pixel Fold was in the Pixel 8 generation but uses the Tensor G2 from the 7s. Pixel 7 release date = October 2022 That's a 26 month difference, yet a full order of magnitude difference in token generation rate on the CPU. Who said Moore's Law is dead? ;)
- deleted 1y ago[deleted]
- sujayk_33 1y ago8 has G3 chip
- z2 1y agoAs a another data point, on E4B, my Pixel 6 Pro (Tensor v1, Oct 2021) is getting about 4.4 t/s decode on a picture of a glass of milk, and over 6 t/s on text chat. It's amazing, I never dreamed I'd be viably running an 8 billion param model when I got it 4 years ago. And kudos to the Pixel team for including 12 GB of RAM when even today PC makers think they can get away with selling 8.
- resource_waste 1y agoIt reminds me of GPT3 quality answers. Kind of impressive. Although my entire usecase of local models is amoral questions, which it blocks. Excited for the abliterated version.
- deleted 1y ago[deleted]
- rao-v 1y agoWhy are we still launching models without simple working python example code (or llama.cpp support)?
- thomashop 1y agoWho runs python code on mobile?
- andrepd 1y agoSuggest giving it no networking permissions (if indeed this is about on-device AI).
- nicholasjarnold 1y agoNetworking perms seem to be required on initial startup of the app. I just installed the apk on a GrapheneOS endpoint (old Pixel 7 Pro) without the Google Play Services installed. The app requires network access to contact Hugging Face and download the model through your HF account. It also requires some interaction/permission agreement with Kaggle. Upon install _with_ network perms the app works, and I'm getting decent performance on the Gemma-3n-E2B-it-int4 model (5-6 token/s). Ok, cool. Now kill the app, disable network permissions and restart it. Choose one of the models that you downloaded when it had network access. It still works. It does appear to be fully local. Yay.
- tootie 1y agoOn Pixel 8a, I asked Gemma 3n to play 20 questions with me. It says it has an object in mind for me to guess then it asks me a question about it. And several attempts to clarify who is supposed to ask questions have gone in circles.
- TiredOfLife 1y agoIs there a list of which SOCs support the GPU acceleration?
- nolist_policy 1y agoIt uses tflite in the background which can GPU accelerate with OpenGL ES 3.1 or OpenCL[0]. So it should work on pretty much any SOC. And you really notice that the model is dumber on GPU, because OpenGL doesn't take accuracy that seriously. [0] https://blog.tensorflow.org/2020/08/faster-mobile-gpu-inference-with-opencl.html?m=1 https://blog.tensorflow.org/2020/08/faster-mobile-gpu-infere...