3 ms·

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models,

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely).

phkahler 2y ago
>> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.
- woodson 2y ago
  ASR systems already use language models during decoding, though mostly not large decoder-only LLMs. However, incorporating LLMs into ASR is currently at the center of a lot of research, e.g. using a speech encoder like wav2vec 2.0 or the whisper encoder with a Qformer etc. and a LoRA adapter on an LLM trained for ASR.
  - omarelb 2y ago
    Really interested in this! Do you know of some good reading in this area?
cjtrowbridge 2y ago
Llama-3-8b runs fine on raspberry pi
- inhumantsar 2y ago
  how fast is that for you?
choppaface 2y ago
But imagine if these models were baked into your Instagram app and then used for ad targeting using your own compute. Then Facebook gets to look at tons of other data and for less cost (and much less litigation risk) to them. In this application it’s unfair to compare tiny models to cloud models. Moreover any incremental precision boosts to tiny models would be notable (and directly translate to revenue).
HanClinto 2y ago
> I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely). I'm not sure that RPi is the right target for the next step of local LLMs, and I think that it's worth considering web-deployment on engines like WebLLM [1]. A 7B model may "run fine" on a Raspberry Pi, but I've (personally) found 7B models to be a bit larger than I want to download / run for web-based interfaces. However, a solid 125M model is the sort of thing that I can run on a webpage, and the time it takes to download to the local user's browser (combined with my bandwidth costs) aren't exorbitant. [1] https://github.com/mlc-ai/web-llm https://github.com/mlc-ai/web-llm

phkahler 2y ago

>> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.

woodson 2y ago
ASR systems already use language models during decoding, though mostly not large decoder-only LLMs. However, incorporating LLMs into ASR is currently at the center of a lot of research, e.g. using a speech encoder like wav2vec 2.0 or the whisper encoder with a Qformer etc. and a LoRA adapter on an LLM trained for ASR.
- omarelb 2y ago
  Really interested in this! Do you know of some good reading in this area?

cjtrowbridge 2y ago

Llama-3-8b runs fine on raspberry pi

inhumantsar 2y ago
how fast is that for you?

choppaface 2y ago

But imagine if these models were baked into your Instagram app and then used for ad targeting using your own compute. Then Facebook gets to look at tons of other data and for less cost (and much less litigation risk) to them. In this application it’s unfair to compare tiny models to cloud models. Moreover any incremental precision boosts to tiny models would be notable (and directly translate to revenue).

HanClinto 2y ago

> I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely). I'm not sure that RPi is the right target for the next step of local LLMs, and I think that it's worth considering web-deployment on engines like WebLLM [1]. A 7B model may "run fine" on a Raspberry Pi, but I've (personally) found 7B models to be a bit larger than I want to download / run for web-based interfaces. However, a solid 125M model is the sort of thing that I can run on a webpage, and the time it takes to download to the local user's browser (combined with my bandwidth costs) aren't exorbitant. [1] https://github.com/mlc-ai/web-llm https://github.com/mlc-ai/web-llm