3 ms·
Sure if you have a specific need you can specialize some NN with the right architecture, collecting the data, doing the training several times, testing the perf
by antirez 1y ago
Sure if you have a specific need you can specialize some NN with the right architecture, collecting the data, doing the training several times, testing the performances, ... Or: you can download an already built LLM and write a prompt.
- galangalalgol 1y agoWhat zero shot would you suggest for that task on an rpi? A temporal fusion thing?
- antirez 1y agoThe small gemma 3 and Qwen 3 models can do wonders for simple tasks as bag of algorithms.
- galangalalgol 1y agoThose would use more ram than most rpi have wouldn't they? Gemma uses 4GB right?
- antirez 1y agoNope, gemma3 and qwen3 exist of many sizes, including very small ones, that 4-bit quantized can run on very small systems. Qwen3-0.6B, 1.7B, ... imagine if you quantize those to 4 bit. But there is the space for the KV cache, if we don't want to limit the runs to very small prompts.
- nolist_policy 1y agoGemma 3 4B QAT int4 quantized from bartowsky should barely fit in a 4GB Raspberry Pi, but without the vision encoder. However the brand-new Gemma 3n E2B and E4B models might fit with vision.
- antirez 1y agoYep, the Gemma 3 1B would be 815MB, with enough margin for a longer prompt. Probably more realistic.
- mnahkies 1y agoSo one of the use cases we're serving in production is predicting energy consumption for a home. Whilst I've not tried, I'm very confident that providing an LLM the historical consumption and asking it to predict future consumption will under perform compared to our forecasting model. The compute required is also several orders of magnitude lower compared to an LLM