4 ms·
> overall my productivity is 10-15% better. That is nothing to sneeze at, but it isn't 10x. It is something to sneeze at if you are 10-15% more expensive to em
by louthy 1y ago
> overall my productivity is 10-15% better. That is nothing to sneeze at, but it isn't 10x.
It is something to sneeze at if you are 10-15% more expensive to employ due to the cost of the LLM tools. The total cost of production should always be considered, not just throughput.
- bravesoul2 1y agoThat's a good insight be because with perfect competition it means you need to share your old salary with an LLM!
- votepaunchy 1y ago> if you are 10-15% more expensive to employ due to the cost of the LLM tools How is one spending anywhere close to 10% of total compensation on LLMs?
- CharlesW 1y ago> It is something to sneeze at if you are 10-15% more expensive to employ due to the cost of the LLM tools. Claude Max is $200/month, or ~2% of the salary of an average software engineer.
- m4rtink 1y agoDoes anyone actually know what the real cost for the customers will be once the free AI money no longer floods those companies?
- wubrr 1y agoI'm no LLM evangelist, far from it, but I expect models of similar quality to the current bleeding-edge, will be freely runnable on consumer hardware within 3 years. Future bleeding-edge models may well be more expensive than current ones, who knows.
- ls612 1y agoHow do the best models that can run on say a single 4090 today compare to GPT 3.5?
- electroglyph 1y agoQwen 2.5 32B which is an older model at this point clearly outperforms it: https://llm-stats.com/models/compare/gpt-3.5-turbo-0125-vs-qwen-2.5-32b-instruct https://llm-stats.com/models/compare/gpt-3.5-turbo-0125-vs-q...
- ls612 1y agoEven when quantized down to 4 bits to fit on a 4090?
- FuckButtons 1y agoNot in my experience, running qwen3:32b is good, but it’s not as coherent or useful as 3.5 at a 4bit quant. But the gap is a lot narrower than llama 70b.
- TeMPOraL 1y agoFor the purpose of keeping the costs of LLM-dependent services down, you don't need to run bleeding-edge models on single consumer GPUs. Even if it takes a hundred GPUs, it still means people can start businesses around hosting those models, and compete with the large vendors.
- jppope 1y agoyeah there was an analysis that came out on hackernews the other day. between low demand side economics, virtually no impact to GDP, and corporate/vc subsidies going away soon we're close to finding out. Sam Altman did convince Softbank to do a 40B round though so it might be another year or two. Current estimates are that its cheaper than search to run so its probabilistic that there will be more search features swapped. OpenAi hasn't dropped their ad platform yet though, so interested to see how that goes
- adi_kurian 1y agoRapid progress in open source says otherwise.
- petra 1y agoThere's a potential for 100x+ lower cost of chips/energy for inference with compute-in-memory technology. So they'll probably find a reasonable cost/value ratio.
- TeMPOraL 1y agoToo cheap to meter? Inference is cheap and there's no long-term or even mid-term moat here. As long as the courts don't shut down Meta over IP issues with LLama training data, that is. I can't stress that enough: "open source" models are what can stop the "real costs" for the customers from growing. Despite popular belief, inference isn't that expensive. This isn't Uber - stopping isn't going to make LLMs infeasible; at worst, it's just going to make people pay API prices instead of subscription prices. As long as there are "open source" models that are legally available and track SOTA, anyone with access to some cloud GPUs can provide "SOTA of 6-12 months ago" for the price of inference, which puts a hard limit on how high OpenAI, et al. can hike the prices. But that's only as long as there are open models. If Meta loses and LLama goes away, the chilling effect will just let OpenAI, Microsoft, Anthropic and Google to set whatever prices they want. EDIT: I mean LLama legally going away. Of course the cat is now out of the bag, the Pandora's box has been opened; the weights are out there and you can't untrain or uninvent them. But keeping the commercial LLM offerings' prices down requires a steady supply of improved open models, and the ability for smaller companies to make a legal business out of hosting them.
- bobbob27 1y agoYou can't just take cost of training out of the equation... If these companies plan to stay afloat, they have to actually pay for the tens of billions they've spent at some point. That's what the parent comment meant by "free AI"
- TeMPOraL 1y agoYes, you can - because of LLama. Training is expensive, but it's not that expensive either. It takes just one of those super-rich players to pay the training costs and then release the weights, to deny other players a moat.
- assuagering 1y agoIf your economic analysis depends on "one of those super-rich players to pay" for it to work, it isn't as much analysis as wishful thinking. All the 100s of billions of $ put into the models so far were not donations. They either make it back to the investors or the show stops at some point. And with a major chunk of proponent's arguments being "it will keep getting better", if you lose that what you got? "This thing can spit out boilerplate code, re-arrange documents and sometimes corrupts data silently and in hard to detect ways but hey you can run it locally and cheaply"?
- selfhoster11 1y agoIn the US, maybe. Several times that by percentage in other places around the world.
- GeoAtreides 1y agothe average software engineer makes $10000 a month after taxes?!