8 ms·
Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering
- sakuraiben 4mo agoOne thing I've noticed using agents for coding is that they really like to write thousands of unit tests but not dynamically test.
- make3 4mo agoyou can just tell them to do more dynamic testing. I think dynamic testing is partly frowned upon because it slows things down & can take down software where you wouldn't expect
- deleted 4mo ago[deleted]
- drivebyhooting 4mo agoAnd they like to burn a ton of tokens writing and debugging tests that are semantically corrupt.
- gib444 4mo agoAnd AWS heavily pushes a complex lambda solution stringing together as many chargeable AWS services as possible for a simple requirement Their interests are often not your interests. In this case they want you to unnecessary money on useless work (let's stop the euphemism of "tokens" btw)
- simianwords 4mo agoThis kind of cute conspiracy theories don’t actually hold true in real life. The companies want to make useful products.
- gib444 4mo agoThey're all just in it for the love of making customers happy, for sure. Amazon actually donates all their profit to charity. Bez os is currently building a foundation to shift all his wealth to, and he's going to become an alpaca farmer /s
- esperent 4mo agoUnit tests are a type of dynamic testing. As opposed to static testing which is linting/typechecking etc. If you want a difference kind of dynamic testing besides unit tests, have you tried writing it in as a requirement during the planning/PRD phase?
- drivebyhooting 4mo agoIn the past Google et al would hire engineers based on how well they could optimize the infrastructure. Maybe soon companies will look at how engineers can optimize the token efficiency of AI.
- Retric 4mo agoThat assumes Tokens will remain a meaningful expense. I’m not sure developers will find uses for ever more tokens nearly as quickly as the prices fall.
- dnlosx 4mo ago[dead]
- ares623 4mo agoHow are we so confident that prices will fall? Isn't the exact opposite happening, right now, during arguably the most critical part of this whole saga (pre-IPO to make things appear as beautiful and as not-obviously-illegal as possible)? And the only reason they were "falling" previously was for hyper growth.
- Retric 4mo agoI don’t think we can extrapolate from current API pricing, but dramatically improving hardware in terms of cost:performance is the underlying reality. Betting against that you need to assume exponentially more expensive models every year.
- jpatt 4mo agoThe Growth aspect mentioned is that VCs are subsidizing the bill right now, so it is hard to know if at the current moment the demand curve would promote as much usage without it, but assuming demand remained constant (not even growing), you could expect token prices to be competed down. It is a commodity without a moat. Now that we have pretty decent open source models, anyone can create a new business to supply more tokens. Sure there’s short term scarcity: energy, GPUs, cooling, but this is a scale up problem. More token demand = more data center build = more energy plant build. This downward pressure will also keep frontier private model prices in check. Differentiation seems to be happening at the harness level, whereby we can expect token spend to be a metric to compete on and drive down for the customer (at least hoping tools in the application space don’t continue token based billing as their primary revenue stream). These are not short term hyper growth forces, but a fundamental alignment of incentives.
- Waffle2180 4mo ago[flagged]
- satvikpendem 4mo agoTokenomics is already a word used to describe cryptocurrency economics, not sure why they'd try to redefine it for AI even if a different sort of token is used.
- NewJazz 4mo agoNew fad. Forget about the old fad. This one will be old soon, you better get on board before its too late!
- dkersten 4mo agoCrypto was already a term before cryptocurrencies made it about them. Web 3.0 was already a thing before crypto bros made web 3 about cryptocurrencies. So what? Terms are reused in different contexts all the time. And most people have moved on from cryptocurrencies anyway, so there’s little chance it’ll confuse anyone.
- deleted 4mo ago[deleted]
- mariusor 4mo agoTokenomics had been already used by marijuana enthusiasts for a long time.
- alchemism 4mo agocryptocurrency economics = cryptonomics You're welcome! =)
- bonigv 4mo ago[dead]
- SubiculumCode 4mo agoReminded me of this paper from last year trying to optimize efficient token usage providing budget guidance information. [1] [1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Steering+LLM+Thinking+with+Budget+Guidance&btnG=#d=gs_qabs&t=1780805077514&u=%23p%3DXGZPDOCIsBQJ https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Stee...
- andrewvu0203 4mo ago[dead]
- baarse 4mo ago[dead]
- senectus1 4mo agoamusing side note: Was in a meeting reviewing a potential new product, it was going well until they showed us that they had added AI to it (of course they have). It was pretty obviously just shoehorned in, and one part of that obviousness was that they had a column that showed how many tokens it took to make each query. I asked who is paying for the tokens, they said its included in the license. I said, so is there a budget or is it all you can eat. they said good question they didnt know and would get back to me. I said the reason i asked was just one query there had a 250k token burn on it. and it was a fairly simple query about one device. then, one of the execs on their side was heard saying out loud "Why are we even showing this to the customers?" it have us quite a chuckle. But lesson learned... the cost of adding AI to anything isnt really being accounted for let alone the true cost of actually running the AI. all things AI are going to get more expensive. even if you dont want the AI aspect.
- prymitive 4mo agoAIshittification
- winphoto 4mo ago[flagged]
- sedatk 4mo agoOne month I could use Github Copilot fully with no disruptions. The next month, after pricing changes, I’ve run out of tokens in two days. Such drastic changes tell me that pricing of tokens is arbitrary, and AI business is running out of money fast.
- lucaspiller 4mo agoI think it's more a consequence of pushing for the biggest valuation/IPO. Rumoured profits on inference are north of 70%. Taking SpaceX as an example, they have increased prices across all their consumer products over the past six months. But they definitely aren't short on money with Alphabet and Anthropic combined paying them over $2 billion per month. Microsoft/GitHub lost out here as they were just repacking other people's products.
- deleted 4mo ago[deleted]
- lefra 4mo agoInference can only happen after having invested in training and datacenter construction. Arguing about "inference profitability" sounds a lot to me like ignoring large cost centers of these comanies.
- jurgenburgen 4mo ago> Rumoured profits on inference are north of 70%. Rumors are worth squat when they’re most likely put in motion by the people with a vested interest in this industry. Let’s talk about profits when there’s real data from the IPO documentation.
- NitpickLawyer 4mo ago> Rumors are worth squat You can make some educated guesses and find out some limits on inferencing cost by looking at 3rd party providers on platforms like openrouter. You can get some median cost /tok for a given model size. Then make some educated guesses on SotA model sizes, and you can get an estimate on pure cost of serving a model. Error bars and all that, of course. But still a range, with some limits.
- deleted 4mo ago[deleted]
- gmerc 4mo agoIt’s just like Airline reward miles and offers no benefit to companies over just renting bare metal GPU time
- emsign 4mo agoI hope this horrible time will soon be over when cheaper NPUs come available from more hardware companies, and also when model size get optimized down further. I wonder what hyperscaled compute farms and models will be good for at that running cost when most AI needs can be fulfilled by on-prem and on-device hardware and models. Probably only customer left are big governments. So in the end the tax payer has to pay for those billions of investments by the AI cartel.
- zozbot234 4mo agoThe typical NPU is only marginally helpful for on-prem inference. A GPU can read quantized data from main memory and dequantize/pad it locally (making effective use of memory throughput); a NPU often needs to read padded data directly from memory, which is wasteful. So it only helps a little bit wrt. prefill. Also, smaller models can obviously be used but a smaller model will be a lot weaker in real-world knowledge and this tends to limit their smarts in a way that can't be compensated by more thinking.
- emsign 4mo agoAt its current iteration the AI tech market is not economically sustainable, not for the other markets outside the AI economy, and most deadly not even for the main target customers or AI tech companies themselves. There have been several news of companies having overspent their token budget month after month. The hardware monopolist and his network of buddy companies can determine the token price as freely as they want, there are no competitors, their only "competitor" is when people stop using AI alltogether.
- scotty79 4mo agoI don't think business is interested in any sustainability of anything. There's zero incentives for that for anyone.
- becomevocal 4mo agoFirst thought was "only 30 tasks" however the findings map to what I've seen personally: code review consumes majority of tokens
- zozbot234 4mo agoCode review could also be run as an unattended/batched task though, possibly with at least some use of on-prem inference (which excels at this). That would be a major saving compared to the usual cloud inference scenario.
- jwnin 4mo agowith which models, though?
- becomevocal 4mo agoYeah wasn’t there a report recently on how local models after energy costs didn’t weren’t actually more efficient to complete the same task?
- eddysir 4mo ago[flagged]
- bob1029 4mo ago> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9% I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code. I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with the agent or the codebase is new / empty.
- zozbot234 4mo agoIf input tokens dominate the cost to that extent, this implies that major gains are possible by making better use of caching. You could basically ask the model to do a one-time "compaction" step including a dump of the relevant portions of the code, and use that as the cached prefix for a large amount of "swarm" subagent calls.
- kolinko 4mo agoDid you experiment with giving agent better tools to navigate and document the codebase? Asts, language servers and so on? A million tokens (not cached) sounds like a lot.
- bob1029 4mo agoThe target codebase is very large. A million tokens is a drop in the proverbial bucket. I still don't understand how caching helps me very much. I must be misunderstanding it because I thought the user's prompt (which is the biggest variable) necessarily sits prior to all of these token intensive tool calls. How can we cache the reading of codebase if the prefix is always moving?
- Phemist 4mo agoIf an agent makes a tool call, the LLM provider will receive the full context again after the result of the tool call becomes available in order to decide the next move. Everything up to the point of the tool call being made will no longer change and could thus in theory be cached. If the agent makes a ton of tool calls, then for every tool call one should be hitting the cache an equal amount of times. A new instruction by the user will be appended at the end if it done in the same conversation. Thus only has influence on the cacheability of the original agent prompt, but not of subsequent tool calls.
- monkeydust 4mo agoI have a MA system setup for personal use. You give it a problem, you then refine that problem where a fast, cheaper model asks you questions which you answer to get a better input prompt. You then choose a MA strategy for example take problem break up to sections then final judge concludes or you do multi turn where agents debate then judge summarises debate. The best approach is what I call 'all angles' where all these strategies run in parallel the final meta-judge synthesise the response - the most useful part of this which I recently added is a view to see the variance in each strategy. Been using this for life stuff - housing search, schools, family challenges! Perhaps I should make a video of it in action if people in HN community interested let me know.
- uxhacker 4mo agoSo what harness are you using? And what LLM’s
- monkeydust 4mo agoHomebrew harness and all frontier ones plus deepseek. All via Openrouter at the moment. Works well enough but can get expensive so use for real high value challenges. Interestingly the refine feature has been most useful to me and people I have shown, essentially people are lazy when expressing the initial problem (me included!), refine asks relevant questions to initial problem then refines the initial statement, user can accept/reject/edit before submitting.
- Cherub0774 4mo agoI came to a similar conclusion. I think the default options in many IDEs (Ask/Plan/Agent) are limited... 'Refine' feels like an improved 'Plan' in that it doesn't just jump right into building a list of tasks based on the initial prompt, because who knows what sort of flaws or deficiencies were present in the initial prompt! Can't always get everything right in the first try. XP I don't think a specific harness is even necessary to get a boost from 'Refine'. Even a simple custom agent is portable enough... it's easy enough to take the existing 'Plan' agent definition present in VS Code and tweak it to be 'Refine' instead.
- deleted 4mo ago[deleted]
- knightops_dev 4mo ago[flagged]
- friendlygeorge 4mo ago[flagged]
- jlcases 4mo ago[dead]
- zcw100 4mo agoI wrote a Subsack post on this topic back in December https://open.substack.com/pub/zacharywhitley/p/the-coming-age-of-tokenomics?r=rc2kf&utm_campaign=post&utm_medium=web https://open.substack.com/pub/zacharywhitley/p/the-coming-ag...
- JoaoBerne 4mo ago[flagged]
- samdonovan 4mo ago[dead]
- jazzen 4mo ago[flagged]
- Pranavsingh431 4mo ago[flagged]