6 ms·
MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use
- sourcecodeplz 2y agoNice, could one use this to train models for Windows PCs also? I don't have a lot of ram.
- skiexperte 2y agoTraining models is not OS dependend. RAM is dependend on the size and i would argue this should be a lot easier to finetune with less GPU Ram. Nonetheless the endgoal will probably be downloading a model like this or paying for finetuning than downloading and using it through an optimized Neuralchip. Its currently more a question of when this will happen. The newest Windows cert already requires some neuralchip and even my google pixel 8 pro can host small models (i know the pixel is not a cheap phone, but the coprocessor should still be much more affordable than a big GPU)
- mmastrac 2y ago> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely).
- phkahler 2y ago>> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.
- woodson 2y agoASR systems already use language models during decoding, though mostly not large decoder-only LLMs. However, incorporating LLMs into ASR is currently at the center of a lot of research, e.g. using a speech encoder like wav2vec 2.0 or the whisper encoder with a Qformer etc. and a LoRA adapter on an LLM trained for ASR.
- omarelb 2y agoReally interested in this! Do you know of some good reading in this area?
- cjtrowbridge 2y agoLlama-3-8b runs fine on raspberry pi
- inhumantsar 2y agohow fast is that for you?
- choppaface 2y agoBut imagine if these models were baked into your Instagram app and then used for ad targeting using your own compute. Then Facebook gets to look at tons of other data and for less cost (and much less litigation risk) to them. In this application it’s unfair to compare tiny models to cloud models. Moreover any incremental precision boosts to tiny models would be notable (and directly translate to revenue).
- HanClinto 2y ago> I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely). I'm not sure that RPi is the right target for the next step of local LLMs, and I think that it's worth considering web-deployment on engines like WebLLM [1]. A 7B model may "run fine" on a Raspberry Pi, but I've (personally) found 7B models to be a bit larger than I want to download / run for web-based interfaces. However, a solid 125M model is the sort of thing that I can run on a webpage, and the time it takes to download to the local user's browser (combined with my bandwidth costs) aren't exorbitant. [1] https://github.com/mlc-ai/web-llm https://github.com/mlc-ai/web-llm
- zurfer 2y agoWhile this is interesting, I wonder what the use case is, other than better autocomplete?
- potatoman22 2y agoIt could power simple agents like Siri under the hood. Helping with natural language understanding, intent classification, retrieval, and other agent tasks.
- rvnx 2y agoLike the Rabbit R1 or Humane AI Pin
- deleted 2y ago[deleted]
- redox99 2y agoLocal agent like siri that can do simple tasks, and route more complex requests.
- skiexperte 2y agoReading emails, replying to emails, scheduling tasks, using apis for services. Basically everything which doesn't need knowledge but actions. "Tell my wife i'm late" and it will use some configured magic to talk to service xy and just does it. Siri is very good in doing homeautomatistaion without the internet, the old google agent and alexa were absolutly not and i don't think they were ever available offline. This basically gives you a local (local-first!) good working assistent
- Narhem 2y agoWould be very nice to have my schedule automatically managed by Siri. Already has a few nice things but I genuinely have trust issues, especially with AI.
- Havoc 2y agoWhat apps can one currently use to run them on say an iPhone? Only aware of the MLC one which has literally 3 old models only
- 5cott0 2y agowat https://huggingface.co/mlc-ai https://huggingface.co/mlc-ai
- Havoc 2y agoOn my iphone there doesn’t seem to be an option to download more. Vaguely recall there being a button initially but don’t see it anymore
- pickettd 2y agoThe Android apk for MLC is updated frequently with recent models built-in. And a Samsung S24+ can comfortably run 7-8B models at reasonable speeds (10ish tokens/sec). https://llm.mlc.ai/docs/deploy/android.html https://llm.mlc.ai/docs/deploy/android.html
- woadwarrior01 2y agoI have an (mlc-llm based) app on the App Store that supports over 2 dozen models, including some recent ones.
- ukuina 2y agocnvrs runs GGUFs on iOS: https://testflight.apple.com/join/ERFxInZg https://testflight.apple.com/join/ERFxInZg
- yshvrdhn 2y agoAm I missing something but can't something like distillation help here ?
- imurray 2y agoThe paper says they tried that: https://arxiv.org/abs/2402.14905 https://arxiv.org/abs/2402.14905 Deep link to the relevant snippet in html version: https://ar5iv.labs.arxiv.org/html/2402.14905#S3.SS5 https://ar5iv.labs.arxiv.org/html/2402.14905#S3.SS5 "So far, we trained compact models from scratch using next tokens as hard labels. We explored Knowledge Distillation (KD)... Unfortunately KD increases training time (slowdown of 2.6−3.2×) and exhibits comparable or inferior accuracy to label-based training (details in appendix)."
- PoignardAzur 2y agoI wonder how much you can push the "deeper and thinner" part. At some point your entire FFN fits into your L2 cache, you're bound to get some performance jumps.
- sigmoid10 2y agoOther research from Meta FAIR actually suggests that you should prune deeper layers if you want to improve performance while maintaining accuracy [1]. So there must be a cutoff point for smaller networks where this approach still works, otherwise the results are contradictory. Or we could drastically improve these new models even further. [1] https://arxiv.org/html/2403.17887v1 https://arxiv.org/html/2403.17887v1
- woodson 2y agoThat reminds me of the findings of Google’s paper on EfficientT5 (https://arxiv.org/abs/2109.10686 https://arxiv.org/abs/2109.10686). They refer to it as “DeepNarrow”.
- Narhem 2y ago[flagged]
- ejdhshsuwisjsh 2y agoAnyone is aware of custom mobile llms? Optimizing and loading in your own voice, selecting your primary language and adding a little bit of personal knowledge like nicknames, location and stuff? My pixel 8 apparently can use / load local models but don't have the time right now to follow that rabbit hole
- euniceee3 2y agoTensor chips are not open enough for an optimized mobile LLM to be ran on them.
- vhiremath4 2y agoIt seems like the smaller models get the largest size decrease by embedding share/weight tying between the linear head and token embeddings. Is there any research going into how to further reduce size from there?
- cztomsik 2y agoIf you mean that LM-head is just inverted embedding matrix then this was already done in GPT-2. Unfortunately, the only thing I found out about this is that bigger models benefit from separate layer. But this was only mentioned somewhere in discord, so no paper to read and my personal hunch is that it should work for bigger models too. After all, GPT-3 was just scaled GPT-2. From my personal experiments, models learn better if you give them harder task. And tied weights could be one of such things. Multi-token prediction could be another and bitnet could be also considered such... (and dropout too)
- cjtrowbridge 2y agoWhy no mmlu or gsm8k?
- lawlessone 2y agoDoes it have to stay on mobile devices? Bit of niche but if its not a resource hog it could be handy for giving NPC's in games more interesting dialogue without having use Even better if it could be tuned in someway to allow dialogue to influence NPC behavior or actions.
- janalsncm 2y agoIt would be fascinating if NPCs had more backstory to them and more complex behaviors. Although I would imagine it would be near impossible to test since anything could influence their behavior.
- lawlessone 2y agoyeah definitely testing would be nightmare. especially if conversations could influence the wider game. You'd have someone on youtube cheesing games by running scamming npcs.
- HanClinto 2y agoI'm definitely interested in exploring this sort of thing. How much can we do with creating interesting characters and interesting circumstances? Makes me think of the way that characters are set up in AI Alibis -- each with their own secrets, but also with clues about other NPC's secrets. That feels like clever design, and it's the first use-case of using LLMs for NPC dialogue that feels interesting to me: https://news.ycombinator.com/item?id=40921990 https://news.ycombinator.com/item?id=40921990
- kevingadd 2y agoWould it be interesting dialogue? You could generate more dialogue, but would it have anything underpinning it of interest to the player? i.e. you could suddenly have townspeople that would talk about local scenery or their relationships with other NPCs, but none of that stuff they describe would actually exist in the game. I would personally be weirded out if NPCs started making stuff up. I can imagine training some sort of LLM on your game data such that NPCs are able to actually describe the game world, but I can't imagine what kind of scale you'd need to operate at for that to be cheaper than just paying someone to write the dialogue. Maybe at Ubisoft's scale where your team sizes are in the thousands (AFAIK, they have been investigating using AI for writing, but it's mostly for things like combat barks which are very repetitive and basically noise.)
- KTibow 2y agoWhen Gemma 2 2b releases it would be interesting to compare its scaling with this
- pmontra 2y agoInteresting research, but Meta do not have any device worth talking about (at least at scale,) unless they want to ship that as part of their apps.
- TeMPOraL 2y agoThey have Oculus.
- ynx 2y agoDismissiveness like this tends to radiate ignorance, not insight. Quests have shipped roughly ~1/2 PS5 sales. Certainly a scale only a handful of technologically advanced product lines outside of phones ever reach. Incidentally, the enabling technology for the Quest? On-device ML that grew out of - you guessed it - developing on-device inference for their apps.
- HanClinto 2y ago125M parameters feels very feasible to ship as part of apps -- even web-based apps.
- banish-m4 2y agoHey HN. I actually have a current need for on-device wake-word-like STT. Which model(s) have the lowest WER and can run on an RPi 4B? I've been looking at openWakeWord. It's for an DIY inventory system.
- BaculumMeumEst 2y agoDo Apple Watches have the hardware capability to run inference on a small model? Do I need a developer account to develop on one?
- mark336 2y agoHow about instead of Gen AI on the desktop, just AI on the desktop. Could organize all my files, emails, and notes and let me search for information from my own data.
- mark336 2y agoBtw, for anyone interested I have AI news summaries and ideas for startups at my website here: https://asiaviewnews.com/gigabots/Threads?p=20007 https://asiaviewnews.com/gigabots/Threads?p=20007