4 ms·
I'm more interested in running distributed inference for purpose built small language models than these coding LLMs. Say a distributed inference for image proc
by Abishek_Muthian 3mo ago
I'm more interested in running distributed inference for purpose built small language models than these coding LLMs.
Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.
Nicely done OP.
- unrvl22 3mo agoSomething like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.
- Abishek_Muthian 3mo agoTrue, I've been telling that coding LLMs have been detrimental to local AI scene. What happened to those nifty ML projects which can run on raspberry pi zero.
- sig_kill 3mo agoThis is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM