3 ms·
I'm not sure if you're implying that very small language models would be run in your raspberry pi example, but for use cases like the time series one, wouldn't
by mnahkies 1y ago
I'm not sure if you're implying that very small language models would be run in your raspberry pi example, but for use cases like the time series one, wouldn't something like an LSTM or TiDE architecture make more sense than a language model?
These are typically small and performant both in compute and accuracy/utility from what I've seen.
I think with all the hype at the moment sometimes AI/ML has become too synonymous with LLM
- greenavocado 1y agoHe's talking about general purpose zero shot models.
- antirez 1y agoSure if you have a specific need you can specialize some NN with the right architecture, collecting the data, doing the training several times, testing the performances, ... Or: you can download an already built LLM and write a prompt.
- galangalalgol 1y agoWhat zero shot would you suggest for that task on an rpi? A temporal fusion thing?
- antirez 1y agoThe small gemma 3 and Qwen 3 models can do wonders for simple tasks as bag of algorithms.
- galangalalgol 1y agoThose would use more ram than most rpi have wouldn't they? Gemma uses 4GB right?
- antirez 1y agoNope, gemma3 and qwen3 exist of many sizes, including very small ones, that 4-bit quantized can run on very small systems. Qwen3-0.6B, 1.7B, ... imagine if you quantize those to 4 bit. But there is the space for the KV cache, if we don't want to limit the runs to very small prompts.
- nolist_policy 1y agoGemma 3 4B QAT int4 quantized from bartowsky should barely fit in a 4GB Raspberry Pi, but without the vision encoder. However the brand-new Gemma 3n E2B and E4B models might fit with vision.
- antirez 1y agoYep, the Gemma 3 1B would be 815MB, with enough margin for a longer prompt. Probably more realistic.
- mnahkies 1y agoSo one of the use cases we're serving in production is predicting energy consumption for a home. Whilst I've not tried, I'm very confident that providing an LLM the historical consumption and asking it to predict future consumption will under perform compared to our forecasting model. The compute required is also several orders of magnitude lower compared to an LLM