5 ms·

What I don't understand is why some companies are so stingy on AI token usage. If you're paying an engineer $X and they're getting 3x the amount of work done y

by qurren 4mo ago

What I don't understand is why some companies are so stingy on AI token usage. If you're paying an engineer $X and they're getting 3x the amount of work done you should be happy paying up to $2X in AI tool usage. In reality many companies start complaining at their employees when they hit $0.1*X or less.

aryehof 4mo ago
They may be getting 3 times the work done, but what work? Quality effective code that meets requirements? Or something else?
- oblio 4mo ago
  > Quality effective code that meets requirements... ... and that also improves the company's financials. Otherwise it's all fluff.
steveBK123 4mo ago
Most companies are not seeing 2x or 3x of value produced from developers with AI is the reason
- brandensilva 4mo ago
  Yeah the cost doesn't justify the value. Spending x3 on tokens is meaningless if they aren't seeing the profit side scale with it. Going faster on product features doesn't mean it translates to more money. In fact I'm not even sure some customers can handle the speed at which a team could move with feature delivery. So scaling horizontally in different markets seems like an advantage if a product is already mature. Which is exactly what Anthropic and Open AI are doing because they want to put their tentacles in everything.
  - steveBK123 4mo ago
    > Going faster on product features doesn't mean it translates to more money. Exactly this. Unless the new features directly drive new users or new revenue from existing users, for many products iterating 2x faster does not mean 2x ROI. Additionally, from anecdotes here, at work, and in my network.. a lot of the unlocked developer velocity is going to fun/frivolous/extra things. I think part of it is developers have their own features they want for themselves that are the easiest and most direct thing to deploy LLMs against, in the absence of good direction. Yes it's cool you finally achieved 100% test coverage, or you wrote a new utility that makes your job easier, or cleared the 2 year old ticket that was 100 deep in your backlog, etc. But there were ROI reasons these things were not done previously. That's already putting aside the fact that development going 3x faster doesn't increase end to end output by 3x because <100% of a SWEs job is development.
    - oblio 4mo ago
      > a lot of the unlocked developer velocity is going to fun/frivolous/extra things https://en.wiktionary.org/wiki/yak_shaving https://en.wiktionary.org/wiki/yak_shaving And I'll have you know I'm a Grand Master Yak Shaver.
mkozlows 4mo ago
10% of a developer's cost is something like $4000/month. Many companies are complaining at a point that's well, well below that. (I think they are being irrational, and that the mental model they have of AI costs -- "how much are we spending on tooling for this developer?" -- is going to shift over time to something more sensible, but those kinds of short-sighted companies are the ones that are having cost panics.)
- bluefirebrand 4mo ago
  > 10% of a developer's cost is something like $4000/month What company pays developers $40000/month and are they hiring?
  - pchristensen 4mo ago
    A company's fully loaded cost (payroll taxes, insurance, fixed company costs, infra, real estate, etc) is usually about double the employee's salary. Half of $40k is 20k/mo, which is ~$250k/hr. That's a lot but very in-bounds for software engineering salaries.
    - christophilus 4mo ago
      Hm. I’ve been way, way underselling myself this whole time.
      - oblio 4mo ago
        Not necessarily. Those jobs are very hard to get and you have to live in the right places. At most 1% of software devs will make that much, at any point in their careers.
    - imtringued 4mo ago
      $250k/hour is half a billion USD per year. Nobody is paid that much. Based on such a basic mistake, I wouldn't trust the rest of your comment. Everything you listed feels like made up arbitrary excuses. Most of the overhead per employee is fixed, that doubling rule of thumb is assuming an employee with a salary closer to the median. At the salary you're claiming, the overhead shrinks to something like 30% on top of the base salary. If anyone took your numbers seriously they would run into the weird conclusion that an employee paid $50k actually costs a company $300k. That's the type of math error you're doing here.
thisoneisreal 4mo ago
I hesitate to say this because I think the AI hype is generally overdone, but I was contemplating the other day how much I would recommend my employer spend on AI tooling for me based on my salary and it's got to be in the tens of thousands of dollars per year if I'm being honest. I'm a contractor. On my most recent client, I was able to learn a completely new domain and business, design a solution and build a full stack POC (a complicated one with technologies I had never used before) nearly singlehandedly in less than 2 months. That includes all of the onboarding time, getting access to the right people, repositories, databases etc. What made it possible was 1) the AI helped me understand technologies I had never used before, from a 30 year old Java stack to graph databases, 2) it could analyze DDLs and make correct inferences about what tables and columns meant in both business and technical terms, 3) it generated mostly-correct code in multiple languages to achieve what I needed it to do, and 4) it referenced libraries I didn't know existed to accomplish tasks I needed done. I could cycle between these activities and iterate FAR more rapidly than I ever have before. To me this is the thing the AI companies should really be emphasizing. I'm not convinced autonomous agents are really working out, but I'm 1000% sold on their ability to empower developers. And I say this as a pretty skeptical late adopter who only knows the most basic AI tooling.
- oblio 4mo ago
  For consulting, sure. Most of the work is hitting the ground running and figuring out the old tech stacks, creating PoCs and then moving them to production and them moving on to the next "engagement". I think the problem is more for long lived projects and developers as regular employees. As you keep working on the product, you should be developing your own mental model of everything: business, tech stack, how the company approaches things. And that's where LLM value drops down a lot, or it should, otherwise there is a bigger problem with the developer. A huge LLM cost for a consultant is in my opinion a lot more justified than a huge recurring cost for someone who's been working on the same thing for at least 9+ months.
  - fc417fc802 4mo ago
    Even when I know a domain well an LLM used well allows me to arrive at an answer in minutes that would otherwise have taken hours. It also accelerates onboarding an unfamiliar stack but the benefit doesn't suddenly disappear once that first hurdle is cleared. After arriving at a seemingly workable design it helps spot oversights I've made either directly or by providing a stand in for a person to walk though the logic with (unlike a real person it doesn't become annoyed or impatient). Then it magically materializes any and all boilerplate along with 90% of the solution near instantaneously. Yeah it confidently makes some truly bizarre errors that no half competent person ever would. Failing to go over the output with a fine toothed comb is inevitably leaving landmines in place. But I agree with you that your own mental model is required - if you have that then you will be able to spot the pitfalls assuming you actually put in the effort to read the output and really think about it instead of skimming. (As an example an LLM recently materialized an nftables filter chain for me that would have taken me quit a while to figure out on my own but for no apparent reason omitted a single key detail that would have completely broken all networking if it had been used blindly.)
vjvjvjvjghv 4mo ago
Yeah. It’s weird. I just booked 2 flights to different meetings with very questionable value in the next few weeks. The whole travel costs around 3000. I would much rather spend this money on tokens but even 300 is too much per month.
Gigachad 4mo ago
Because work done isn’t the same as value produced or profits increased. Spending millions of dollars on useless busywork that no one would have bothered doing manually isn’t productive.
steve-atx-7600 4mo ago
Especially if the company could simply reimburse $200/month Claude or codex. Those are a steal. Hard to exhaust those limits.
imtringued 4mo ago
When you look at the salaries that the average HN poster brags about, that's at least ten thousand dollars worth of tokens or a billion input tokens per year. That doesn't sound like an unreasonable spending limit.

by qurren 4mo ago

aryehof 4mo ago

They may be getting 3 times the work done, but what work? Quality effective code that meets requirements? Or something else?

oblio 4mo ago
> Quality effective code that meets requirements... ... and that also improves the company's financials. Otherwise it's all fluff.

steveBK123 4mo ago

Most companies are not seeing 2x or 3x of value produced from developers with AI is the reason

brandensilva 4mo ago
Yeah the cost doesn't justify the value. Spending x3 on tokens is meaningless if they aren't seeing the profit side scale with it. Going faster on product features doesn't mean it translates to more money. In fact I'm not even sure some customers can handle the speed at which a team could move with feature delivery. So scaling horizontally in different markets seems like an advantage if a product is already mature. Which is exactly what Anthropic and Open AI are doing because they want to put their tentacles in everything.
- steveBK123 4mo ago
  > Going faster on product features doesn't mean it translates to more money. Exactly this. Unless the new features directly drive new users or new revenue from existing users, for many products iterating 2x faster does not mean 2x ROI. Additionally, from anecdotes here, at work, and in my network.. a lot of the unlocked developer velocity is going to fun/frivolous/extra things. I think part of it is developers have their own features they want for themselves that are the easiest and most direct thing to deploy LLMs against, in the absence of good direction. Yes it's cool you finally achieved 100% test coverage, or you wrote a new utility that makes your job easier, or cleared the 2 year old ticket that was 100 deep in your backlog, etc. But there were ROI reasons these things were not done previously. That's already putting aside the fact that development going 3x faster doesn't increase end to end output by 3x because <100% of a SWEs job is development.
  - oblio 4mo ago
    > a lot of the unlocked developer velocity is going to fun/frivolous/extra things https://en.wiktionary.org/wiki/yak_shaving https://en.wiktionary.org/wiki/yak_shaving And I'll have you know I'm a Grand Master Yak Shaver.

mkozlows 4mo ago

10% of a developer's cost is something like $4000/month. Many companies are complaining at a point that's well, well below that. (I think they are being irrational, and that the mental model they have of AI costs -- "how much are we spending on tooling for this developer?" -- is going to shift over time to something more sensible, but those kinds of short-sighted companies are the ones that are having cost panics.)

bluefirebrand 4mo ago
> 10% of a developer's cost is something like $4000/month What company pays developers $40000/month and are they hiring?
- pchristensen 4mo ago
  A company's fully loaded cost (payroll taxes, insurance, fixed company costs, infra, real estate, etc) is usually about double the employee's salary. Half of $40k is 20k/mo, which is ~$250k/hr. That's a lot but very in-bounds for software engineering salaries.
  - christophilus 4mo ago
    Hm. I’ve been way, way underselling myself this whole time.
    - oblio 4mo ago
      Not necessarily. Those jobs are very hard to get and you have to live in the right places. At most 1% of software devs will make that much, at any point in their careers.
  - imtringued 4mo ago
    $250k/hour is half a billion USD per year. Nobody is paid that much. Based on such a basic mistake, I wouldn't trust the rest of your comment. Everything you listed feels like made up arbitrary excuses. Most of the overhead per employee is fixed, that doubling rule of thumb is assuming an employee with a salary closer to the median. At the salary you're claiming, the overhead shrinks to something like 30% on top of the base salary. If anyone took your numbers seriously they would run into the weird conclusion that an employee paid $50k actually costs a company $300k. That's the type of math error you're doing here.

thisoneisreal 4mo ago

I hesitate to say this because I think the AI hype is generally overdone, but I was contemplating the other day how much I would recommend my employer spend on AI tooling for me based on my salary and it's got to be in the tens of thousands of dollars per year if I'm being honest. I'm a contractor. On my most recent client, I was able to learn a completely new domain and business, design a solution and build a full stack POC (a complicated one with technologies I had never used before) nearly singlehandedly in less than 2 months. That includes all of the onboarding time, getting access to the right people, repositories, databases etc. What made it possible was 1) the AI helped me understand technologies I had never used before, from a 30 year old Java stack to graph databases, 2) it could analyze DDLs and make correct inferences about what tables and columns meant in both business and technical terms, 3) it generated mostly-correct code in multiple languages to achieve what I needed it to do, and 4) it referenced libraries I didn't know existed to accomplish tasks I needed done. I could cycle between these activities and iterate FAR more rapidly than I ever have before. To me this is the thing the AI companies should really be emphasizing. I'm not convinced autonomous agents are really working out, but I'm 1000% sold on their ability to empower developers. And I say this as a pretty skeptical late adopter who only knows the most basic AI tooling.

oblio 4mo ago
For consulting, sure. Most of the work is hitting the ground running and figuring out the old tech stacks, creating PoCs and then moving them to production and them moving on to the next "engagement". I think the problem is more for long lived projects and developers as regular employees. As you keep working on the product, you should be developing your own mental model of everything: business, tech stack, how the company approaches things. And that's where LLM value drops down a lot, or it should, otherwise there is a bigger problem with the developer. A huge LLM cost for a consultant is in my opinion a lot more justified than a huge recurring cost for someone who's been working on the same thing for at least 9+ months.
- fc417fc802 4mo ago
  Even when I know a domain well an LLM used well allows me to arrive at an answer in minutes that would otherwise have taken hours. It also accelerates onboarding an unfamiliar stack but the benefit doesn't suddenly disappear once that first hurdle is cleared. After arriving at a seemingly workable design it helps spot oversights I've made either directly or by providing a stand in for a person to walk though the logic with (unlike a real person it doesn't become annoyed or impatient). Then it magically materializes any and all boilerplate along with 90% of the solution near instantaneously. Yeah it confidently makes some truly bizarre errors that no half competent person ever would. Failing to go over the output with a fine toothed comb is inevitably leaving landmines in place. But I agree with you that your own mental model is required - if you have that then you will be able to spot the pitfalls assuming you actually put in the effort to read the output and really think about it instead of skimming. (As an example an LLM recently materialized an nftables filter chain for me that would have taken me quit a while to figure out on my own but for no apparent reason omitted a single key detail that would have completely broken all networking if it had been used blindly.)

vjvjvjvjghv 4mo ago

Yeah. It’s weird. I just booked 2 flights to different meetings with very questionable value in the next few weeks. The whole travel costs around 3000. I would much rather spend this money on tokens but even 300 is too much per month.

Gigachad 4mo ago

Because work done isn’t the same as value produced or profits increased. Spending millions of dollars on useless busywork that no one would have bothered doing manually isn’t productive.

steve-atx-7600 4mo ago

Especially if the company could simply reimburse $200/month Claude or codex. Those are a steal. Hard to exhaust those limits.

imtringued 4mo ago

When you look at the salaries that the average HN poster brags about, that's at least ten thousand dollars worth of tokens or a billion input tokens per year. That doesn't sound like an unreasonable spending limit.