12 ms·
I think its worth noting that if you are paying for electricity Local LLM is NOT free. In most cases you will find that Haiku is cheaper, faster, and better th
by yaur 6mo ago
I think its worth noting that if you are paying for electricity Local LLM is NOT free. In most cases you will find that Haiku is cheaper, faster, and better than anything that will run on your local machine.
- gyrovagueGeist 6mo agoElectricity (on continental US) is pretty cheap assuming you already have the hardware: Running at a full load of 1000W for every second of the year, for a model that produces 100 tps at 16 cents per kWh, is $1200 USD. The same amount of tokens would cost at least $3,150 USD on current Claude Haiku 3.5 pricing.
- ac29 6mo agoThis 35B-A3B model is 4-5x cheaper than Haiku though, suggesting it would still be cheaper to outsource inference to the cloud vs running locally in your example
- postalrat 6mo agoIf you need the heating then it is basically free.
- mrob 6mo agoOnly if you use resistive electric heating, which is usually the most expensive heating available.