15 ms·

Managing AI Coding Costs at Scale

317 points by moonikakiss 2mo ago

nh43215rgb 2mo ago
This is interesting. So as of August 6, 2026 the "Databricks recommended models" are glm 5.2, gpt 5.6 sol, and.... Opus 4.8. not opus 5.
bogota 2mo ago
Really? Because removing it from my company has saved us over 2 million a year and we were able to speed up processing. The chargeback model for databricks is predatory at best.
- smt88 2mo ago
  I think you’ve misunderstood the article. It’s about how Databricks reduced their own costs, not about how adopting Databricks will reduce anyone else’s costs.
- SteveNuts 2mo ago
  What did you move to and what type of workload, if I may ask?
platinumrad 2mo ago
Careful. If you admit to using models that weren't trained by OpenAI or Anthropic then you might hauled in front of Congress: https://www.scmp.com/news/china/diplomacy/article/3362616/us-lawmakers-investigate-doordashs-use-moonshot-ais-kimi-k26-model https://www.scmp.com/news/china/diplomacy/article/3362616/us...
- axus 2mo ago
  Why would it matter if foreign companies analyzed DoorDash data? Pizza deliveries to the Pentagon is all I can come up with, but that's publicly available at https://www.pizzint.watch/ https://www.pizzint.watch/ I would bet my entire Polymarket balance ($0) that some military contractors have already asked AIs on the public Internet to design software for them.
- seizethecheese 2mo ago
  I'd prefer congress to be asking questions (this is all they are doing so far, based on the article) before doing any legislating.
- Footprint0521 2mo ago
  That is a wild article! Hopefully the recent US open source model initiatives can pan out to cut costs, because I’m not a fan of dependence on Chinese models either, but I’m in a spot where I need to maximize intelligence on minimal cost… There’s not even a distant comparison US lab rn which is sad
GiorgioG 2mo ago
Too bad their AI query generation is next to useless.
aliasxneo 2mo ago
First time hearing of Omnigent. Anyone have experience using it?
- notduckrabbit 2mo ago
  I've tested Omnigent superficially, attracted to its thinking around policy, governance, sandboxing, and ui. But it's still alpha at present. I forked its Polly model and got working a somewhat more complex multiagent workflow that I've also modeled in Sandcastle and Gas City but the agent broke after the next update which I would have needed to patch to maintain functionality. Subjectively I also noticed individual models seemed to be performing somewhat worse when wrapped in the platform's framework, presumably due to the extra context introduced (token use was measurably higher). Promising project that I'll revisit when it's further along and I do not doubt the outcomes Databricks claims in committedly dogfooding it.
  - DenisM 2mo ago
    Double-harnessing distortions seems like a weak point. I wonder if it’s just a temporary measure and long term it’s about writing custom harness going straight to inference APIs across all models.
- vehemenz 2mo ago
  I've been using it for a week or so. The main draw for me is that I can keep my sessions in one database regardless of the model/provider I use. The webapp can access everything remotely, which is convenient when I'm on my phone. I haven't gotten a chance to test the multi-agent capabilities, but the DeepSeek Flash prices are so low that I probably will soon.
- deathmonger5000 2mo ago
  If you’re into web based meta-harnesses you might like Circus Chief: https://github.com/ferrislucas/Circus-Chief https://github.com/ferrislucas/Circus-Chief Similar idea re using the same UI across all models, but the agent can modify the harness config as well as start/schedule sessions, etc. The Kanban board feature can be used to orchestrate agent driven workflows, and the agent itself can modify the Kanban lanes. Basically the agent can do all the same things you use the UI for. Example prompt: “schedule a session using Opus 5 with max thinking for every feature in the PRD on the canvas. For every session make the prompt instruct the agent to review end to end test coverage for the feature and create a report on the canvas for test gaps that you find. Schedule the sessions overnight tonight spread evenly throughout the night, and have each session set to retry upon token exhaustion.” I think it’s more meta than other meta-harnesses, but I’m biased because it’s my pet project.
bisonbear 2mo ago
This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs. Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity I'm actually working on building generic repo-specific benchmarks at https://stet.sh https://stet.sh ;)
- pwendell 2mo ago
  The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks. In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
  - bisonbear 2mo ago
    Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustration with the experience of working with Opus 5, despite it topping benchmarks and being objectively extremely powerful/competent. However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957 https://arxiv.org/abs/2606.29957 https://labs.scale.com/papers/swe-interact https://labs.scale.com/papers/swe-interact. Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks
skullone 2mo ago
Yawn. Databricks and their half baked overly expensive platform.
lubujackson 2mo ago
These seem like the obvious tweaks akin to "using a cheaper hosting platform". I think the real savings come from careful context control for programmatic agents, careful tool awareness and usage to reduce thrashing, distilling workflows into deterministic processes and, moat importantly, adding friction and boundaries for non-technical users who tend to burn tokens making insane asks like "analyze all documents and give me a summary".
dan_q 2mo ago
[flagged]
wxw 2mo ago
> Rapidly adopting newer, more efficient models delivers the largest cost wins of any technique. I think the more interesting lever is the fourth they mention: token efficiency. > By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include. I think there’s still lots of low hanging fruit in regards to monitoring and improving agent work. Look at your sessions. Look at how much time and context is being spent on, say, a web search returning dozens of results when one good single-pager doc would’ve been better.
- ankitmathur 2mo ago
  100% - there's a lot to learn from traces from real-life sessions with coding tools! For example, I found it pretty eye-opening to see how wide the distribution of tasks truly is. There's also subtle things like how a poorly designed MCP API surface can cause a massive amount of token waste from the model just iterating on finding the right way to call it.
- gabrigoo 2mo ago
  [dead]
sellmethepen 2mo ago
is this opensource or have to buy from Databricks?
- mjuarez 2mo ago
  It seems Databricks open-sourced it a while ago: https://www.databricks.com/blog/introducing-omnigent-meta-harness-combine-control-and-share-your-agents https://www.databricks.com/blog/introducing-omnigent-meta-ha... https://github.com/omnigent-ai/omnigent https://github.com/omnigent-ai/omnigent
  - mandeepj 2mo ago
    Omniagent looks quite similar to OpenRouter (https://openrouter.ai/ https://openrouter.ai/)
    - ankitmathur 2mo ago
      Omnigent and OpenRouter are different in the sense that OpenRouter is where you can go to call the actual model but Omnigent is intended to be the place where you go describe the high level task to be done, and work is farmed out to various harnesses and models. Those sandboxes can themselves be using OpenRouter for capacity! We're calling the layer coordinating harnesses "meta-harness'
      - jvican 2mo ago
        Omnigent seems to compete more against Orca https://github.com/stablyai/orca https://github.com/stablyai/orca They both went to be the Agent IDE layer, where you come with your tasks and everything is taken care of. I've been using Orca for a handful of tasks and have been largely enjoying it. My default barebones workflow is ghostty + zmx on ssh connections.
        mandeepj 2mo ago
        These tools casually like to claim they are orchestrators, but unfortunately, none of them are.
sandeepkd 2mo ago
I find this funny and interesting at some levels 1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes 2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.
- justincormack 2mo ago
  Databricks will be using the API anyway, thats all you get with an enterprise agreement.
- ChoosesBarbecue 2mo ago
  ... where in the article did they say they were using subscriptions? I'm fairly certain enterprises can't access subscription pricing in any case, they're all API costs (Anthropic doesn't support more than 150 on subscription pricing [0][1]). [0]: https://support.claude.com/en/articles/9797531-what-is-the-enterprise-plan https://support.claude.com/en/articles/9797531-what-is-the-e... [1]: https://support.claude.com/en/articles/9266767-what-is-the-team-plan https://support.claude.com/en/articles/9266767-what-is-the-t...
  - sandeepkd 2mo ago
    Going through their harness (codex, claude) is subscription (app use) which is heavily? subsidized. Anyone using the enterprise plan are charged the API pricing, however the article is not clear if Databricks is using enterprise plan or not which is why added the following disclaimer > Unless Databricks has some agreement in place
    - therealdrag0 2mo ago
      They certainly have an enterprise plan?
extr 2mo ago
I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign) - Use Opus 5 or Sol Med to execute - Auto-fix bugs and CI until green + thermonuclear review skill x3. - Manual interrogation of change/nits - Come up with QA plan and have Codex Computer Use execute on it - Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc) I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.
- biophysboy 2mo ago
  Do you have tips for generating clean productive output per dollar?
  - the_sleaze_ 2mo ago
    in my humble experience it boils down to mastery. Are you at least conversational in the subject matter? You're gonna have a good time just by paying attention and adjusting your workflow. If you're getting a lot of back and forth with it, its asking a lot of planning type questions, stop, step back, rethink the whole feature, and start again from the beginning with everything more fleshed out. If you are in a brand new field, there's no way to bridge that divide. The issue is you don't know what is good or bad, or whether what you have learned is good or bad. You're in a sports car and you don't know how to drive much less what's track and what's field. You can spend a lot of effort getting good at prompting towards writing tests and E2E tests to at least verify your app does what you expect it to, regardless of experience.
lbriner 2mo ago
There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs. Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.
- jgalt212 2mo ago
  > we started using AI tools and ended up spending millions per year This is how AWS made its fortune.
  - chadash 2mo ago
    Not only this, but perhaps even more nefarious is that AWS gives lots of startups $100k+ in credits. This feels generous when you get it. In reality, it means that (unless you are in a compute intensive startup) you can go for months or years before you hit this, but by the time you do, you already have very solid monthly spend. Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business. And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
    - andrekandre 2mo ago
      > And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually. the best thing to do is avoid the first cigarette (but everyone at the office smokes, and if you want a promotion you'll smoke with them too)
cyanydeez 2mo ago
Probably coulda got every dev a local model for how much they spent; what a brialliant set of economists
machinatools 2mo ago
[flagged]
chis 2mo ago
It’s funny how different everyone’s experience is with this stuff. To me the diminishing returns are more around not going crazy with prototyping or running with xmax thinking all the time. I haven’t found it hard to stay under the usage limit of one $200/mo Claude and one $200/mo Codex subscription. If my company told me yeah we’ve decided you don’t get Fable or Opus 5 because it’s too pricey, you gotta use GLM whatever, I’d be displeased.
shay_ker 2mo ago
how do any of these routing approaches handle kv cache misses? Devin Fusion is the only one that explicitly addresses this, though it does so by switching models during compaction (not sure this isn't still a cache miss though)
- chris_money202 2mo ago
  The kv cache is wiped as soon as you get your answer, cloud hosts are not going to hold the GPU memory for your entire session. You're probably referring to some agent level cache
- ankitmathur 2mo ago
  We're going to do a followup blog detailing our routing approach soon! In short, the router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before (this is the advantage of having a meta-harness that is actually planning all the sub-agents centrally) Maintaining the cache is extremely, extremely important, so we're iterating fast but that's a major factor we track in the router's development. Couple things I'd look at: 1. The cache is generally reset after a compaction - this is the best time to make a switch if you want. 2. In many cases, the max duration of a cache is 1h, so if a session is being resumed after a long time, that's also a good time to re-assess the complexity. We're iterating fast here and learning a lot! Definitely a lot to think about it in this area.
pwendell 2mo ago
I authored this - happy to answer any questions.
dude250711 2mo ago
First the mofos force you to use AI then they become stingy about it. An AI-edited post by the way.
- quikoa 2mo ago
  Well yes, first hit is free.
nichochar 2mo ago
Surprisingly pragmatic and info packed article.. Kudos to databricks, I also find it interesting that such different companies (Stripe, Ramp, Databricks) are all building the exact same internal tools. I think building companies is going to look more generic in the future because intelligence is an API now.
- pwendell 2mo ago
  Thank you for the feedback. We wrote this because after discussing with some of our peer companies, I realized everyone was roughly doing similar things. And I thought it would be good for someone to just systematically write down what those are so that others can try out the techniques if they find them useful.
  - DenisM 2mo ago
    +1 well written, well paced article. Pleasure to read. Have you tried measuring Gemini? now that you have the router it should be a simple task. Thanks!
dyauspitr 2mo ago
So did we. I just asked my team to get personal accounts that I reimburse them for. It’s just a golden age loop though, the gravy train can’t go on forever unless we start building out thousands of data centers and associated renewable energy.
salmonfamine 2mo ago
I think there is a lot of dev cope in this thread. My workflow is very simple: 1. develop requirements for code change 2. take manual notes for implementation, maybe use LLM for some discovery/investigation 3. present notes to frontier LLM 4. develop implementation plan (bulk of work) 5. let LLM rip 6. review diff, manually fixing/refactoring code as necessary, sometimes prompting for revisions 7. get automated LLM review 8. get human review this reliably produces the work of 2-3 pre-AI senior engineers with a lower bug rate, equivalent performance, robust edge-case consideration, etc. Does the LLM produce over-engineered solutions? All the time. I stop it from doing that, or manually fix it myself. Does the LLM always adhere to the best system design? No, not at all. I often have to guide its design into a better, north-star aligned one. I don't just sit in front of my terminal and say, "Ok Claude, build the app." It is a very iterative process, and not without its potential pitfalls. But it is very, very productive.
- mortalapeman 2mo ago
  This has been my experience as well. The best model I have access to right now is Opus 4.8. It's really good at fixing bugs in an established architecture or adding a similar feature, but it's absolutely mid tier at putting together a simple effective architecture for even common software problems. I can tell that functional programming practices are not a large part of it's training material.
  - salmonfamine 2mo ago
    Fortunately I’m at a startup with basically infinite Fable 5, and it is legitimately a huge step forward over Opus 4.8. Would recommend. Hopefully some open weight models catch up to its capabilities soon.
- myegorov 2mo ago
  My workflow as well. If I had to gauge the direction of the wind, this produces the work of somewhere between 1-2 pre-AI engineers. My own code quality probably went down, along with my focus and the depth of understanding. Bug rates have increased. I don't have any stats to prove this.
behat 2mo ago
Appreciate the detail in this and the previous post on creating internal benchmarks! Have you all attempted finetuning smaller OSS models on your repos for coding?
- pwendell 2mo ago
  We do this for a lot of our customers (fine tuned to save cost when inference volume is high). Right now for internal coding we are using off-the-shelf models but we are considering fine tuning as well to squeeze more efficiency out.
dgellow 2mo ago
What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to the user and isn’t too relevant. One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)
- OrangeDelonge 2mo ago
  Do you think Anthropic or OpenAI will eventually try to crack down on routing harnasses? Provide a more vertically integrated experience? They are already trying ro ship hardware products.
  - gizmodo59 2mo ago
    I will believe there is no moat when the revenues for Anthropic is not 70B. It seems like people want to throw away money and they don’t like switching
    - thereitgoes456 2mo ago
      There is no evidence that Anthropic's revenue is 70B.
      - antonvs 2mo ago
        https://www.linkedin.com/posts/sytaylor_anthropic-is-now-tracking-74bn-in-annualized-activity-7486155086398795776-MqXv/ https://www.linkedin.com/posts/sytaylor_anthropic-is-now-tra...
        dgellow 2mo ago
DenisM 2mo ago
How can Smart Router achieve higher task completion rate compared to any of the base models if all it does is dynamically switch base models based on cost??
XCSme 2mo ago
As a solo dev, this gives me hope. I feel like I have an advantage over big companies, if I can use the best models on a subscription and not worry about costs much, when they can't do the same as outlined in the article.
nphardon 2mo ago
good engineer + llm = good engineer. bad engineer + llm = bad engineer.
sashank_1509 2mo ago
I suspect that when it comes to hard complex software products, you’re better off ignoring agents and doing “trad coding”. What you lose in short term speed you gain in manageable complex codebases. If you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term. Now of course, there are products that just involve lots of code but are not actually complex. This is generally the project with like hundreds or thousands of features but most of the features are separate and don’t actually interact in complex ways. Think a task management app with hundreds of features like calendar, email integration etc. there I think agents gives you more bang for the buck. Just my thought, using agents at work.
- SOLAR_FIELDS 2mo ago
  > if you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term A bold claim to make with little to no supporting evidence
  - delusional 2mo ago
    We are 7 years into a cycle of people baselessly claiming that "AI" will soon develop superhuman abilities and might just take over the world, before killing us all. I don't think there's any way to take part in this discourse without making somewhat unsupported claims.
    - zaphirplane 2mo ago
      We haven’t had LLM agents for 7 years
      - delusional 2mo ago
        They claimed these problems LONG before agents were the new thing. To my eye it started with the bullshit around GPT-2, and that was in 2019, 7 years ago.
- 2mo ago
semessier 2mo ago
not sure about the use of exponential and efficiency frontier here, these have formal sides to them but seem to be used rather inflationary and colloquially.
vira28 2mo ago
Reads like an add to Omnigent or whatever harness (wait it’s meta harness?.
throwatdem12311 2mo ago
“use lower cost models” “use price controls” Truly revolutionary stuff.
thelastgallon 2mo ago
> nearly every company deploying AI tools at scale has hit the same wall: exponentially growing costs. That curve is unsustainable - left unchecked it will eventually overtake revenue. But the question remains, AI hasn't shown any explosion in growth/revenue for most companies. The effects seem muted.
- williamstein 2mo ago
  If only one company had AI and nobody else did, then that one company would be expected to start seeing an explosion in growth/revenue. As it is, everybody has access to AI, so extensive use of AI should just result in keeping up with the competition. The ROI to spending on AI is "not dying".
  - skydhash 2mo ago
    AI is sold as a power tool. Even if everyone is using it, you should see at least some absolute improvement in boost, even though they are still relatively the same. No such boost has been observed.
Sattyamjjain 2mo ago
[flagged]
arendtio 2mo ago
I think the important aspects are making individual developers responsible for their costs and restricting the most expensive models when costs are too high. That way, people have a chance to learn when they tend to use the most expensive models for trivial tasks.
- jgilias 2mo ago
  Definitely. I can easily cut my individual AI spend by 50% at least. But I won’t, until someone tells me to. For three reasons: 1. Thinking about how to optimize AI spend is one more thing to think about when I have too many things to think about already; 2. If there’s even a minuscule chance that me selecting down the model results in a task getting delayed because for that particular thing the model happened to be “too dumb”, I’m just not going to do that. Because I’m evaluated for what I ship and saying, “oh, not happening this week because I downgraded to Haiku for the exploration phase, and that resulted in a faulty plan, so it slipped a day” just doesn’t cut it. 3. Why should I spend time thinking about costs, and risk being slower because of that, if the next guy doesn’t?
  - emanuelecella 2mo ago
    You shouldn't have to. The gate should be automatic and not a decision you make per request but a hard cap enforced at the infrastructure layer before the call happens from your providers
ENTROPY-X 2mo ago
[flagged]
resonance_dev 2mo ago
[flagged]
imilev 2mo ago
Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.
- Terretta 2mo ago
  DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade. That said, this type of post needs to be read with product marketing context in mind. > I think the most important aspect from it is the auto-routing On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously not shown in their opening graphic. As they put it in their own words near the end: "simple tuning of… caching settings… 50% reduction in … costs, with no observed quality degradation…" They also aren't showing the harness called ‘pi’ which generally tops results (not only with ‘open’ models), and is consistent with Anthropic's recent "works better when we don't stuff 100k system prompts into context" about Opus 5. Recent models do better with less jank trying to prescribe behavior. So then one wonders: why is this post written to say you should use a meta router but no mention that's offset by the cache busting, and show developers cost savings but no mention that Anthropic and OpenAI both have developer-facing utilization dashes already? Perhaps it's so engineers can show enterprise procurement why they want to buy exactly what Databricks happens to have for sale.
  - yeswecatan 2mo ago
    Do you know they are using pi or are you inferring it?
    - Terretta 2mo ago
      Sorry, didn't mean they're using pi. They're itemizing ways to get better results but not showing pi, while other testing shows most models (including Opus) achieve better from within pi.
ianmarcinkowski 2mo ago
Maybe these larger models are better at producing concise, internally-consistent, non-polluting (of the wider codebase) features. I just spent 3 days reviewing the code of a vibe coding maximalist at work. They needed to touch 96 front-end files, ~50 back-end files. We now have `is_uk_company` spread throughout the entire codebase every time we need to output a date format. I would say 30% of the code can be replaced by a moderately more-wise approach of "you ain't gonna need this", 20% is actively polluting and/or harmful to the coherence of the feature. I'm pretty sure this plan came from a braindump of a verbal meeting they had with our product team, using OpenAI's top models. There were 2 initial spreadsheets that were ABSOLUTELY INDECIPHERABLE, and now we have a new one labeled "FeatureX - Phase 2 to 4". Just the date fields is a hilarious example of why I'm skeptical. Python (babel.date) and Javascript (the browser's locale) both have locale-aware date rendering. Using a library for country-based date and time parsing would have eliminated ~12 country-specific change sites... but instead we got `is_uk` and raw date format strings (`%d/%m/%y`) in-line inside already-enormous functions. Also, at a meta level, the backend code shouldn't usually be formatting the dates (data export may be an exception), we should be sending ISO dates and having the frontend format the code if anyone gives a crap. I'm ~$500/day and I spent 3 days reviewing this stuff. Our CTO spent ~2 days this week reviewing this feature, I'm not sure what his daily rate would end up being. Just the date-formatting part cost me half a day... that's at least $250. A few hundred dollars on Xhigh tokens burned $2500-3500 in human review effort, and I was delayed from: 1. Getting an entire sentiment analysis feature deployed by helping our interns 2. Getting a load balancer solution working to automate a manual process 3. Improving our deterministic Cypress test suite 4. Getting a demo feature launched for our sales team that would bring in more money
- patrickmay 2mo ago
  > Just the date fields is a hilarious example of why I'm skeptical. Python (babel.date) and Javascript (the browser's locale) both have locale-aware date rendering. Using a library for country-based date and time parsing would have eliminated ~12 country-specific change sites... but instead we got `is_uk` and raw date format strings (`%d/%m/%y`) in-line inside already-enormous functions. Also, at a meta level, the backend code shouldn't usually be formatting the dates (data export may be an exception), we should be sending ISO dates and having the frontend format the code if anyone gives a crap. Did you reject the PR or is that not a career-enhancing approach in your organization?
tosh 2mo ago
> Using harnesses that are “less chatty” (more token efficient), or tuning existing harnesses to generate less token overhead. agree, this works, undervalued! look at minimal agents that protect the context window: - pi (https://github.com/earendil-works/pi) - smol (https://github.com/smol-env/smol) some thoughts on the other tips (for coding): 1) stronger models are more token efficient for open ended tasks because at the limit … - stronger models can solve tasks that the weaker models can not solve - stronger models make fewer mistakes, compose things better (cli, abstractions, …) - navigate the code base better - are better at removing and simplifying the code base again that of course is difficult to benchmark, so most attention goes to simple benchmarks that show cheaper models can get similar results on 'closed' tasks with easy to 'eval' results 2) dynamic request and task routing sounds great/obvious but is very very hard - to benefit from caching you don't want to switch model or inference endpoint - to _know_ a certain request can be routed to a weaker/cheaper model needs good context and a strong model to get right and often is still unknowable because the active coding session can go many ways and turn from trivial to challenging in a few turns, always in motion is the future, if you get it wrong you are back in the problem space of #1 using cheaper models and auto-routing do work well for 'closed' tasks where you have something repeatable and can evaluate whether a certain quality threshold is reached that you are comfortable with for open ended coding sessions it is not so easy that said: cheaper does not have to mean weaker, you want to look at the pareto frontier and stay up to date on new good models there are many models like deepseek v4 flash and luna that are both cheaper and way better than most other models
adamddev1 2mo ago
> "Exponentially growing costs" I don't think people thing don't think enough about the exponentially growing costs with AI. The cost of tokens, the cost of bugs, the cost of technical debt, the cost of software bloat and slowdown, the cost on the RAM market, on the environment, the costs of the little lies and hallucinations that creep into our codebases, blogs, books, and search results. What seems like a manageable margin suddenly starts growing and destroying everything.
- andrekandre 2mo ago
  > What seems like a manageable margin suddenly starts growing and destroying everything probably where the 1 trillion+ dollar valuations are coming from.... lots of thick saas margins they can siphon/skim from the whole industry.
lukasco 2mo ago
With self-driving agents, the costs stop being evident as you go, and show up after the fact. I've been building governors that slow the agents down, and then also look at odd things some of the harnesses do, such as constantly making mistakes in tool calls. But overall, it's quite a task, and you really have to decide what you are prioritizing for. Do I want my agents doing lots of work, or (in my case), leaving some of my subscriptions available for me to do work.
- deleted 2mo ago
  [deleted]
readingrainbow 2mo ago
Another AI Psychosis/AI Derangement Syndrome post from a shite company. Yay!
desbo 2mo ago
“agentic coding has measurably improved every velocity metric we track and, in some teams, driven an order-of-magnitude gains in output” Of course it has. But is that a good enough measure of value? What about the business and customer outcomes from all the extra lines of code and closed JIRA tickets? Was it worth it?
felixlu2026 2mo ago
[dead]

317 points by moonikakiss 2mo ago

This is interesting. So as of August 6, 2026 the "Databricks recommended models" are glm 5.2, gpt 5.6 sol, and.... Opus 4.8. not opus 5.

bogota 2mo ago

Really? Because removing it from my company has saved us over 2 million a year and we were able to speed up processing. The chargeback model for databricks is predatory at best.

smt88 2mo ago
I think you’ve misunderstood the article. It’s about how Databricks reduced their own costs, not about how adopting Databricks will reduce anyone else’s costs.
SteveNuts 2mo ago
What did you move to and what type of workload, if I may ask?

platinumrad 2mo ago

Careful. If you admit to using models that weren't trained by OpenAI or Anthropic then you might hauled in front of Congress: https://www.scmp.com/news/china/diplomacy/article/3362616/us-lawmakers-investigate-doordashs-use-moonshot-ais-kimi-k26-model https://www.scmp.com/news/china/diplomacy/article/3362616/us...

axus 2mo ago
Why would it matter if foreign companies analyzed DoorDash data? Pizza deliveries to the Pentagon is all I can come up with, but that's publicly available at https://www.pizzint.watch/ https://www.pizzint.watch/ I would bet my entire Polymarket balance ($0) that some military contractors have already asked AIs on the public Internet to design software for them.
seizethecheese 2mo ago
I'd prefer congress to be asking questions (this is all they are doing so far, based on the article) before doing any legislating.
Footprint0521 2mo ago
That is a wild article! Hopefully the recent US open source model initiatives can pan out to cut costs, because I’m not a fan of dependence on Chinese models either, but I’m in a spot where I need to maximize intelligence on minimal cost… There’s not even a distant comparison US lab rn which is sad

GiorgioG 2mo ago

Too bad their AI query generation is next to useless.

aliasxneo 2mo ago

First time hearing of Omnigent. Anyone have experience using it?

notduckrabbit 2mo ago
I've tested Omnigent superficially, attracted to its thinking around policy, governance, sandboxing, and ui. But it's still alpha at present. I forked its Polly model and got working a somewhat more complex multiagent workflow that I've also modeled in Sandcastle and Gas City but the agent broke after the next update which I would have needed to patch to maintain functionality. Subjectively I also noticed individual models seemed to be performing somewhat worse when wrapped in the platform's framework, presumably due to the extra context introduced (token use was measurably higher). Promising project that I'll revisit when it's further along and I do not doubt the outcomes Databricks claims in committedly dogfooding it.
- DenisM 2mo ago
  Double-harnessing distortions seems like a weak point. I wonder if it’s just a temporary measure and long term it’s about writing custom harness going straight to inference APIs across all models.
vehemenz 2mo ago
I've been using it for a week or so. The main draw for me is that I can keep my sessions in one database regardless of the model/provider I use. The webapp can access everything remotely, which is convenient when I'm on my phone. I haven't gotten a chance to test the multi-agent capabilities, but the DeepSeek Flash prices are so low that I probably will soon.
deathmonger5000 2mo ago
If you’re into web based meta-harnesses you might like Circus Chief: https://github.com/ferrislucas/Circus-Chief https://github.com/ferrislucas/Circus-Chief Similar idea re using the same UI across all models, but the agent can modify the harness config as well as start/schedule sessions, etc. The Kanban board feature can be used to orchestrate agent driven workflows, and the agent itself can modify the Kanban lanes. Basically the agent can do all the same things you use the UI for. Example prompt: “schedule a session using Opus 5 with max thinking for every feature in the PRD on the canvas. For every session make the prompt instruct the agent to review end to end test coverage for the feature and create a report on the canvas for test gaps that you find. Schedule the sessions overnight tonight spread evenly throughout the night, and have each session set to retry upon token exhaustion.” I think it’s more meta than other meta-harnesses, but I’m biased because it’s my pet project.

bisonbear 2mo ago

This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs. Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity I'm actually working on building generic repo-specific benchmarks at https://stet.sh https://stet.sh ;)

pwendell 2mo ago
The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks. In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
- bisonbear 2mo ago
  Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustration with the experience of working with Opus 5, despite it topping benchmarks and being objectively extremely powerful/competent. However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957 https://arxiv.org/abs/2606.29957 https://labs.scale.com/papers/swe-interact https://labs.scale.com/papers/swe-interact. Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks

skullone 2mo ago

Yawn. Databricks and their half baked overly expensive platform.

lubujackson 2mo ago

These seem like the obvious tweaks akin to "using a cheaper hosting platform". I think the real savings come from careful context control for programmatic agents, careful tool awareness and usage to reduce thrashing, distilling workflows into deterministic processes and, moat importantly, adding friction and boundaries for non-technical users who tend to burn tokens making insane asks like "analyze all documents and give me a summary".

dan_q 2mo ago

[flagged]

wxw 2mo ago

> Rapidly adopting newer, more efficient models delivers the largest cost wins of any technique. I think the more interesting lever is the fourth they mention: token efficiency. > By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include. I think there’s still lots of low hanging fruit in regards to monitoring and improving agent work. Look at your sessions. Look at how much time and context is being spent on, say, a web search returning dozens of results when one good single-pager doc would’ve been better.

ankitmathur 2mo ago
100% - there's a lot to learn from traces from real-life sessions with coding tools! For example, I found it pretty eye-opening to see how wide the distribution of tasks truly is. There's also subtle things like how a poorly designed MCP API surface can cause a massive amount of token waste from the model just iterating on finding the right way to call it.
gabrigoo 2mo ago
[dead]

sellmethepen 2mo ago

is this opensource or have to buy from Databricks?

mjuarez 2mo ago
It seems Databricks open-sourced it a while ago: https://www.databricks.com/blog/introducing-omnigent-meta-harness-combine-control-and-share-your-agents https://www.databricks.com/blog/introducing-omnigent-meta-ha... https://github.com/omnigent-ai/omnigent https://github.com/omnigent-ai/omnigent
- mandeepj 2mo ago
  Omniagent looks quite similar to OpenRouter (https://openrouter.ai/ https://openrouter.ai/)
  - ankitmathur 2mo ago
    Omnigent and OpenRouter are different in the sense that OpenRouter is where you can go to call the actual model but Omnigent is intended to be the place where you go describe the high level task to be done, and work is farmed out to various harnesses and models. Those sandboxes can themselves be using OpenRouter for capacity! We're calling the layer coordinating harnesses "meta-harness'
    - jvican 2mo ago
      Omnigent seems to compete more against Orca https://github.com/stablyai/orca https://github.com/stablyai/orca They both went to be the Agent IDE layer, where you come with your tasks and everything is taken care of. I've been using Orca for a handful of tasks and have been largely enjoying it. My default barebones workflow is ghostty + zmx on ssh connections.
      - mandeepj 2mo ago
        These tools casually like to claim they are orchestrators, but unfortunately, none of them are.

sandeepkd 2mo ago

I find this funny and interesting at some levels 1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes 2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.

justincormack 2mo ago
Databricks will be using the API anyway, thats all you get with an enterprise agreement.
ChoosesBarbecue 2mo ago
... where in the article did they say they were using subscriptions? I'm fairly certain enterprises can't access subscription pricing in any case, they're all API costs (Anthropic doesn't support more than 150 on subscription pricing [0][1]). [0]: https://support.claude.com/en/articles/9797531-what-is-the-enterprise-plan https://support.claude.com/en/articles/9797531-what-is-the-e... [1]: https://support.claude.com/en/articles/9266767-what-is-the-team-plan https://support.claude.com/en/articles/9266767-what-is-the-t...
- sandeepkd 2mo ago
  Going through their harness (codex, claude) is subscription (app use) which is heavily? subsidized. Anyone using the enterprise plan are charged the API pricing, however the article is not clear if Databricks is using enterprise plan or not which is why added the following disclaimer > Unless Databricks has some agreement in place
  - therealdrag0 2mo ago
    They certainly have an enterprise plan?

extr 2mo ago

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign) - Use Opus 5 or Sol Med to execute - Auto-fix bugs and CI until green + thermonuclear review skill x3. - Manual interrogation of change/nits - Come up with QA plan and have Codex Computer Use execute on it - Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc) I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.

biophysboy 2mo ago
Do you have tips for generating clean productive output per dollar?
- the_sleaze_ 2mo ago
  in my humble experience it boils down to mastery. Are you at least conversational in the subject matter? You're gonna have a good time just by paying attention and adjusting your workflow. If you're getting a lot of back and forth with it, its asking a lot of planning type questions, stop, step back, rethink the whole feature, and start again from the beginning with everything more fleshed out. If you are in a brand new field, there's no way to bridge that divide. The issue is you don't know what is good or bad, or whether what you have learned is good or bad. You're in a sports car and you don't know how to drive much less what's track and what's field. You can spend a lot of effort getting good at prompting towards writing tests and E2E tests to at least verify your app does what you expect it to, regardless of experience.

lbriner 2mo ago

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs. Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.

jgalt212 2mo ago
> we started using AI tools and ended up spending millions per year This is how AWS made its fortune.
- chadash 2mo ago
  Not only this, but perhaps even more nefarious is that AWS gives lots of startups $100k+ in credits. This feels generous when you get it. In reality, it means that (unless you are in a compute intensive startup) you can go for months or years before you hit this, but by the time you do, you already have very solid monthly spend. Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business. And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
  - andrekandre 2mo ago
    > And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually. the best thing to do is avoid the first cigarette (but everyone at the office smokes, and if you want a promotion you'll smoke with them too)

cyanydeez 2mo ago

Probably coulda got every dev a local model for how much they spent; what a brialliant set of economists

machinatools 2mo ago

[flagged]

chis 2mo ago

It’s funny how different everyone’s experience is with this stuff. To me the diminishing returns are more around not going crazy with prototyping or running with xmax thinking all the time. I haven’t found it hard to stay under the usage limit of one $200/mo Claude and one $200/mo Codex subscription. If my company told me yeah we’ve decided you don’t get Fable or Opus 5 because it’s too pricey, you gotta use GLM whatever, I’d be displeased.

shay_ker 2mo ago

how do any of these routing approaches handle kv cache misses? Devin Fusion is the only one that explicitly addresses this, though it does so by switching models during compaction (not sure this isn't still a cache miss though)

chris_money202 2mo ago
The kv cache is wiped as soon as you get your answer, cloud hosts are not going to hold the GPU memory for your entire session. You're probably referring to some agent level cache
ankitmathur 2mo ago
We're going to do a followup blog detailing our routing approach soon! In short, the router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before (this is the advantage of having a meta-harness that is actually planning all the sub-agents centrally) Maintaining the cache is extremely, extremely important, so we're iterating fast but that's a major factor we track in the router's development. Couple things I'd look at: 1. The cache is generally reset after a compaction - this is the best time to make a switch if you want. 2. In many cases, the max duration of a cache is 1h, so if a session is being resumed after a long time, that's also a good time to re-assess the complexity. We're iterating fast here and learning a lot! Definitely a lot to think about it in this area.

pwendell 2mo ago

I authored this - happy to answer any questions.

dude250711 2mo ago

First the mofos force you to use AI then they become stingy about it. An AI-edited post by the way.

quikoa 2mo ago
Well yes, first hit is free.

nichochar 2mo ago

Surprisingly pragmatic and info packed article.. Kudos to databricks, I also find it interesting that such different companies (Stripe, Ramp, Databricks) are all building the exact same internal tools. I think building companies is going to look more generic in the future because intelligence is an API now.

pwendell 2mo ago
Thank you for the feedback. We wrote this because after discussing with some of our peer companies, I realized everyone was roughly doing similar things. And I thought it would be good for someone to just systematically write down what those are so that others can try out the techniques if they find them useful.
- DenisM 2mo ago
  +1 well written, well paced article. Pleasure to read. Have you tried measuring Gemini? now that you have the router it should be a simple task. Thanks!

dyauspitr 2mo ago

So did we. I just asked my team to get personal accounts that I reimburse them for. It’s just a golden age loop though, the gravy train can’t go on forever unless we start building out thousands of data centers and associated renewable energy.

salmonfamine 2mo ago

I think there is a lot of dev cope in this thread. My workflow is very simple: 1. develop requirements for code change 2. take manual notes for implementation, maybe use LLM for some discovery/investigation 3. present notes to frontier LLM 4. develop implementation plan (bulk of work) 5. let LLM rip 6. review diff, manually fixing/refactoring code as necessary, sometimes prompting for revisions 7. get automated LLM review 8. get human review this reliably produces the work of 2-3 pre-AI senior engineers with a lower bug rate, equivalent performance, robust edge-case consideration, etc. Does the LLM produce over-engineered solutions? All the time. I stop it from doing that, or manually fix it myself. Does the LLM always adhere to the best system design? No, not at all. I often have to guide its design into a better, north-star aligned one. I don't just sit in front of my terminal and say, "Ok Claude, build the app." It is a very iterative process, and not without its potential pitfalls. But it is very, very productive.

mortalapeman 2mo ago
This has been my experience as well. The best model I have access to right now is Opus 4.8. It's really good at fixing bugs in an established architecture or adding a similar feature, but it's absolutely mid tier at putting together a simple effective architecture for even common software problems. I can tell that functional programming practices are not a large part of it's training material.
- salmonfamine 2mo ago
  Fortunately I’m at a startup with basically infinite Fable 5, and it is legitimately a huge step forward over Opus 4.8. Would recommend. Hopefully some open weight models catch up to its capabilities soon.
myegorov 2mo ago
My workflow as well. If I had to gauge the direction of the wind, this produces the work of somewhere between 1-2 pre-AI engineers. My own code quality probably went down, along with my focus and the depth of understanding. Bug rates have increased. I don't have any stats to prove this.

behat 2mo ago

Appreciate the detail in this and the previous post on creating internal benchmarks! Have you all attempted finetuning smaller OSS models on your repos for coding?

pwendell 2mo ago
We do this for a lot of our customers (fine tuned to save cost when inference volume is high). Right now for internal coding we are using off-the-shelf models but we are considering fine tuning as well to squeeze more efficiency out.

dgellow 2mo ago

What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to the user and isn’t too relevant. One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)

OrangeDelonge 2mo ago
Do you think Anthropic or OpenAI will eventually try to crack down on routing harnasses? Provide a more vertically integrated experience? They are already trying ro ship hardware products.
- gizmodo59 2mo ago
  I will believe there is no moat when the revenues for Anthropic is not 70B. It seems like people want to throw away money and they don’t like switching
  - thereitgoes456 2mo ago
    There is no evidence that Anthropic's revenue is 70B.
    - antonvs 2mo ago
      https://www.linkedin.com/posts/sytaylor_anthropic-is-now-tracking-74bn-in-annualized-activity-7486155086398795776-MqXv/ https://www.linkedin.com/posts/sytaylor_anthropic-is-now-tra...
      - dgellow 2mo ago

DenisM 2mo ago

How can Smart Router achieve higher task completion rate compared to any of the base models if all it does is dynamically switch base models based on cost??

XCSme 2mo ago

As a solo dev, this gives me hope. I feel like I have an advantage over big companies, if I can use the best models on a subscription and not worry about costs much, when they can't do the same as outlined in the article.

nphardon 2mo ago

good engineer + llm = good engineer. bad engineer + llm = bad engineer.

sashank_1509 2mo ago

I suspect that when it comes to hard complex software products, you’re better off ignoring agents and doing “trad coding”. What you lose in short term speed you gain in manageable complex codebases. If you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term. Now of course, there are products that just involve lots of code but are not actually complex. This is generally the project with like hundreds or thousands of features but most of the features are separate and don’t actually interact in complex ways. Think a task management app with hundreds of features like calendar, email integration etc. there I think agents gives you more bang for the buck. Just my thought, using agents at work.

SOLAR_FIELDS 2mo ago
> if you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term A bold claim to make with little to no supporting evidence
- delusional 2mo ago
  We are 7 years into a cycle of people baselessly claiming that "AI" will soon develop superhuman abilities and might just take over the world, before killing us all. I don't think there's any way to take part in this discourse without making somewhat unsupported claims.
  - zaphirplane 2mo ago
    We haven’t had LLM agents for 7 years
    - delusional 2mo ago
      They claimed these problems LONG before agents were the new thing. To my eye it started with the bullshit around GPT-2, and that was in 2019, 7 years ago.
2mo ago

semessier 2mo ago

not sure about the use of exponential and efficiency frontier here, these have formal sides to them but seem to be used rather inflationary and colloquially.

vira28 2mo ago

Reads like an add to Omnigent or whatever harness (wait it’s meta harness?.

throwatdem12311 2mo ago

“use lower cost models” “use price controls” Truly revolutionary stuff.

thelastgallon 2mo ago

> nearly every company deploying AI tools at scale has hit the same wall: exponentially growing costs. That curve is unsustainable - left unchecked it will eventually overtake revenue. But the question remains, AI hasn't shown any explosion in growth/revenue for most companies. The effects seem muted.

williamstein 2mo ago
If only one company had AI and nobody else did, then that one company would be expected to start seeing an explosion in growth/revenue. As it is, everybody has access to AI, so extensive use of AI should just result in keeping up with the competition. The ROI to spending on AI is "not dying".
- skydhash 2mo ago
  AI is sold as a power tool. Even if everyone is using it, you should see at least some absolute improvement in boost, even though they are still relatively the same. No such boost has been observed.

Sattyamjjain 2mo ago

[flagged]

arendtio 2mo ago

I think the important aspects are making individual developers responsible for their costs and restricting the most expensive models when costs are too high. That way, people have a chance to learn when they tend to use the most expensive models for trivial tasks.

jgilias 2mo ago
Definitely. I can easily cut my individual AI spend by 50% at least. But I won’t, until someone tells me to. For three reasons: 1. Thinking about how to optimize AI spend is one more thing to think about when I have too many things to think about already; 2. If there’s even a minuscule chance that me selecting down the model results in a task getting delayed because for that particular thing the model happened to be “too dumb”, I’m just not going to do that. Because I’m evaluated for what I ship and saying, “oh, not happening this week because I downgraded to Haiku for the exploration phase, and that resulted in a faulty plan, so it slipped a day” just doesn’t cut it. 3. Why should I spend time thinking about costs, and risk being slower because of that, if the next guy doesn’t?
- emanuelecella 2mo ago
  You shouldn't have to. The gate should be automatic and not a decision you make per request but a hard cap enforced at the infrastructure layer before the call happens from your providers

ENTROPY-X 2mo ago

[flagged]

resonance_dev 2mo ago

[flagged]

imilev 2mo ago

Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.

Terretta 2mo ago
DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade. That said, this type of post needs to be read with product marketing context in mind. > I think the most important aspect from it is the auto-routing On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously not shown in their opening graphic. As they put it in their own words near the end: "simple tuning of… caching settings… 50% reduction in … costs, with no observed quality degradation…" They also aren't showing the harness called ‘pi’ which generally tops results (not only with ‘open’ models), and is consistent with Anthropic's recent "works better when we don't stuff 100k system prompts into context" about Opus 5. Recent models do better with less jank trying to prescribe behavior. So then one wonders: why is this post written to say you should use a meta router but no mention that's offset by the cache busting, and show developers cost savings but no mention that Anthropic and OpenAI both have developer-facing utilization dashes already? Perhaps it's so engineers can show enterprise procurement why they want to buy exactly what Databricks happens to have for sale.
- yeswecatan 2mo ago
  Do you know they are using pi or are you inferring it?
  - Terretta 2mo ago
    Sorry, didn't mean they're using pi. They're itemizing ways to get better results but not showing pi, while other testing shows most models (including Opus) achieve better from within pi.

ianmarcinkowski 2mo ago

Maybe these larger models are better at producing concise, internally-consistent, non-polluting (of the wider codebase) features. I just spent 3 days reviewing the code of a vibe coding maximalist at work. They needed to touch 96 front-end files, ~50 back-end files. We now have `is_uk_company` spread throughout the entire codebase every time we need to output a date format. I would say 30% of the code can be replaced by a moderately more-wise approach of "you ain't gonna need this", 20% is actively polluting and/or harmful to the coherence of the feature. I'm pretty sure this plan came from a braindump of a verbal meeting they had with our product team, using OpenAI's top models. There were 2 initial spreadsheets that were ABSOLUTELY INDECIPHERABLE, and now we have a new one labeled "FeatureX - Phase 2 to 4". Just the date fields is a hilarious example of why I'm skeptical. Python (babel.date) and Javascript (the browser's locale) both have locale-aware date rendering. Using a library for country-based date and time parsing would have eliminated ~12 country-specific change sites... but instead we got `is_uk` and raw date format strings (`%d/%m/%y`) in-line inside already-enormous functions. Also, at a meta level, the backend code shouldn't usually be formatting the dates (data export may be an exception), we should be sending ISO dates and having the frontend format the code if anyone gives a crap. I'm ~$500/day and I spent 3 days reviewing this stuff. Our CTO spent ~2 days this week reviewing this feature, I'm not sure what his daily rate would end up being. Just the date-formatting part cost me half a day... that's at least $250. A few hundred dollars on Xhigh tokens burned $2500-3500 in human review effort, and I was delayed from: 1. Getting an entire sentiment analysis feature deployed by helping our interns 2. Getting a load balancer solution working to automate a manual process 3. Improving our deterministic Cypress test suite 4. Getting a demo feature launched for our sales team that would bring in more money

patrickmay 2mo ago
> Just the date fields is a hilarious example of why I'm skeptical. Python (babel.date) and Javascript (the browser's locale) both have locale-aware date rendering. Using a library for country-based date and time parsing would have eliminated ~12 country-specific change sites... but instead we got `is_uk` and raw date format strings (`%d/%m/%y`) in-line inside already-enormous functions. Also, at a meta level, the backend code shouldn't usually be formatting the dates (data export may be an exception), we should be sending ISO dates and having the frontend format the code if anyone gives a crap. Did you reject the PR or is that not a career-enhancing approach in your organization?

tosh 2mo ago

> Using harnesses that are “less chatty” (more token efficient), or tuning existing harnesses to generate less token overhead. agree, this works, undervalued! look at minimal agents that protect the context window: - pi (https://github.com/earendil-works/pi) - smol (https://github.com/smol-env/smol) some thoughts on the other tips (for coding): 1) stronger models are more token efficient for open ended tasks because at the limit … - stronger models can solve tasks that the weaker models can not solve - stronger models make fewer mistakes, compose things better (cli, abstractions, …) - navigate the code base better - are better at removing and simplifying the code base again that of course is difficult to benchmark, so most attention goes to simple benchmarks that show cheaper models can get similar results on 'closed' tasks with easy to 'eval' results 2) dynamic request and task routing sounds great/obvious but is very very hard - to benefit from caching you don't want to switch model or inference endpoint - to _know_ a certain request can be routed to a weaker/cheaper model needs good context and a strong model to get right and often is still unknowable because the active coding session can go many ways and turn from trivial to challenging in a few turns, always in motion is the future, if you get it wrong you are back in the problem space of #1 using cheaper models and auto-routing do work well for 'closed' tasks where you have something repeatable and can evaluate whether a certain quality threshold is reached that you are comfortable with for open ended coding sessions it is not so easy that said: cheaper does not have to mean weaker, you want to look at the pareto frontier and stay up to date on new good models there are many models like deepseek v4 flash and luna that are both cheaper and way better than most other models

adamddev1 2mo ago

> "Exponentially growing costs" I don't think people thing don't think enough about the exponentially growing costs with AI. The cost of tokens, the cost of bugs, the cost of technical debt, the cost of software bloat and slowdown, the cost on the RAM market, on the environment, the costs of the little lies and hallucinations that creep into our codebases, blogs, books, and search results. What seems like a manageable margin suddenly starts growing and destroying everything.

andrekandre 2mo ago
> What seems like a manageable margin suddenly starts growing and destroying everything probably where the 1 trillion+ dollar valuations are coming from.... lots of thick saas margins they can siphon/skim from the whole industry.

lukasco 2mo ago

With self-driving agents, the costs stop being evident as you go, and show up after the fact. I've been building governors that slow the agents down, and then also look at odd things some of the harnesses do, such as constantly making mistakes in tool calls. But overall, it's quite a task, and you really have to decide what you are prioritizing for. Do I want my agents doing lots of work, or (in my case), leaving some of my subscriptions available for me to do work.

deleted 2mo ago
[deleted]

readingrainbow 2mo ago

Another AI Psychosis/AI Derangement Syndrome post from a shite company. Yay!

desbo 2mo ago

“agentic coding has measurably improved every velocity metric we track and, in some teams, driven an order-of-magnitude gains in output” Of course it has. But is that a good enough measure of value? What about the business and customer outcomes from all the extra lines of code and closed JIRA tickets? Was it worth it?

felixlu2026 2mo ago

[dead]