3 ms·
> This would virtually make inference free right? Not really. Inference is never "free" unless you cache the result (which is just a static output) or unless y
by talldayo 2y ago
> This would virtually make inference free right?
Not really. Inference is never "free" unless you cache the result (which is just a static output) or unless you reduce complexity (which yields procedurally less-usable outputs).
- FezzikTheGiant 2y agoCan you explain further? Why would it not be free if it's running locally
- talldayo 2y agoI thought you meant "free" in terms of computational cost; running local is technically free of charge, but also requires a lot of processing power. Inferecing a properly-sized LLM will potentially starve the rest of your software from GPU/CPU/memory access, so you have to plan accordingly.