4 ms·
> it also would use less electricity How would it use less electricity? I’d like to learn more.
by thih9 7mo ago
> it also would use less electricity
How would it use less electricity? I’d like to learn more.
- jychang 7mo agoThat's completely not true. LLM on device would use MORE electricity. Service providers that do batch>1 inference are a lot more efficient per watt. Local inference can only do batch=1 inference, which is very inefficient.