5 ms·

Does it really? Your average software engineer is like £20-30 an hour, for the cost of 1m output tokens you can get a Dev for a full week.

sheepscreek 2y ago
The math doesn’t check out. A day maybe. Also it’s not just about a placeholder dev. The person needs to know your use-case and have the tech chops to deliver successfully in that timeframe. Now to have that delivered to you in less than an hour? That’s a huge win.
- zipy124 2y ago
  No, even contractors who are more expensive per day can be had for £200-300 per diem. Employee costs you are looking at closer to £165 including national insurance and pension on a standard £35k a year salary. Even an employee on £50k a year is only £241.97 a day. IR35 rules largely closed down a lot of the high-value contracting business and pushed day rates down heavily.
  - refulgentis 2y ago
    Sure. So why are you saying we'd get a dev for a week for $600? :) Even we take that at face value, now we're claiming devs are $30K/year. (This was sort of baseline full-time pay when I was in high school, 20 years ago) Why do I need 1,000,000 tokens for 500 loc? I don't think further futzing with the numbers makes this work, it's off by multiple OOMs while barely credible
    - zipy124 2y ago
      Ok closer to 4 days than a week, still an insane differential in cost. As we all know it's never just 500 lines of code, and 500 lines can take up quite a lot of tokens, especially on a reasoning model like o1-pro which we know can spend quite a hefty chunk of tokens on thinking. Once you add in iteration and testing you can find yourself racking up quite the bill even on small projects.
      - refulgentis 2y ago
        In a separate thread I estimated cost of doing the full output 10x. $18. It certainly adds up, yes, the thing I get indignant about (not with you, just, in general) is even the most expensive is definitely an OOM off and most likely 2 OOMs off from paying an engineer. It's non-trivial work that requires domain knowledge, i.e. it would have been a 4-6 week project for a noogler, and I would have been impressed if they did it without heavy guidance. (Emergent requirements around purposely using an older API, stuff like that).
        zipy124 2y ago
        I don't agree with your cost estimate, but even accepting it there's no way your spending £1800/4-6 weeks on something as small as 500 lines of code and as simple as PCM audio player code. Your description sounds more like the thing an undergraduate might make in a Hackathon.
        refulgentis 2y ago
        That's fair, for better or worse, I'm not going to spend my time proving it was hard.
refulgentis 2y ago
Leaving the dissection of this to the separate reply, let's estimate cost: - 80 chars per line, 30 occupied (avg'd across 300 KLOC in codebase) - 500 lines of code - 15000 characters - 4 chars / token - 3750 tokens output - 10 full iterations, and don't apply cached token pricing that's 90% off - 37,500 tokens req'd in output - $600 / 1M tokens - $0.60 / 1K tokens - $18
- intelVISA 2y ago
  Been a minute since I touched audio code but isn't PCM quite basic? Really hard to beat $18 though! Even the 2hrs it'd take a decent SWE would be easily 30x that.
- zipy124 2y ago
  Even cached input is only half price with openAI (and they don't even offer it for o1-pro). Further we also aren't counting input here which can get long since it includes the previous output, which for the last request will be 33,750 + reasoning + any prompts, which will increase your cost quite a bit there. But yes that is more reasonable than I'd expect I must admit, but I still think it needs to be at least an order of magnitude cheaper to compete against the other models out there. I'm not sure I know a lot of employees who would allocate that sort of constant funding to a consumables tool for an employee, given that's your usual monthly cost for a typical saas product.
intelVISA 2y ago
The avg. SWE is a toss up if they create more issues than they solve over time. Factor in on-boarding, bugs and taking time away from other expensive people becomes >$100/hr real quick.
- zipy124 2y ago
  If this were true there would be a lot less hiring of suggests engineers in the world and a lot more people getting laid off.
  - intelVISA 2y ago
    There would be if it was easy to recognize. You generally can't see this due to all the middle men and bloat, and that no corp really wants to measure dev productivity against revenue like this as it'd raise a lot of uncomfortable questions by both devs and shareholders alike.

sheepscreek 2y ago

The math doesn’t check out. A day maybe. Also it’s not just about a placeholder dev. The person needs to know your use-case and have the tech chops to deliver successfully in that timeframe. Now to have that delivered to you in less than an hour? That’s a huge win.

zipy124 2y ago
No, even contractors who are more expensive per day can be had for £200-300 per diem. Employee costs you are looking at closer to £165 including national insurance and pension on a standard £35k a year salary. Even an employee on £50k a year is only £241.97 a day. IR35 rules largely closed down a lot of the high-value contracting business and pushed day rates down heavily.
- refulgentis 2y ago
  Sure. So why are you saying we'd get a dev for a week for $600? :) Even we take that at face value, now we're claiming devs are $30K/year. (This was sort of baseline full-time pay when I was in high school, 20 years ago) Why do I need 1,000,000 tokens for 500 loc? I don't think further futzing with the numbers makes this work, it's off by multiple OOMs while barely credible
  - zipy124 2y ago
    Ok closer to 4 days than a week, still an insane differential in cost. As we all know it's never just 500 lines of code, and 500 lines can take up quite a lot of tokens, especially on a reasoning model like o1-pro which we know can spend quite a hefty chunk of tokens on thinking. Once you add in iteration and testing you can find yourself racking up quite the bill even on small projects.
    - refulgentis 2y ago
      In a separate thread I estimated cost of doing the full output 10x. $18. It certainly adds up, yes, the thing I get indignant about (not with you, just, in general) is even the most expensive is definitely an OOM off and most likely 2 OOMs off from paying an engineer. It's non-trivial work that requires domain knowledge, i.e. it would have been a 4-6 week project for a noogler, and I would have been impressed if they did it without heavy guidance. (Emergent requirements around purposely using an older API, stuff like that).
      - zipy124 2y ago
        I don't agree with your cost estimate, but even accepting it there's no way your spending £1800/4-6 weeks on something as small as 500 lines of code and as simple as PCM audio player code. Your description sounds more like the thing an undergraduate might make in a Hackathon.
        refulgentis 2y ago
        That's fair, for better or worse, I'm not going to spend my time proving it was hard.

refulgentis 2y ago

Leaving the dissection of this to the separate reply, let's estimate cost: - 80 chars per line, 30 occupied (avg'd across 300 KLOC in codebase) - 500 lines of code - 15000 characters - 4 chars / token - 3750 tokens output - 10 full iterations, and don't apply cached token pricing that's 90% off - 37,500 tokens req'd in output - $600 / 1M tokens - $0.60 / 1K tokens - $18

intelVISA 2y ago
Been a minute since I touched audio code but isn't PCM quite basic? Really hard to beat $18 though! Even the 2hrs it'd take a decent SWE would be easily 30x that.
zipy124 2y ago
Even cached input is only half price with openAI (and they don't even offer it for o1-pro). Further we also aren't counting input here which can get long since it includes the previous output, which for the last request will be 33,750 + reasoning + any prompts, which will increase your cost quite a bit there. But yes that is more reasonable than I'd expect I must admit, but I still think it needs to be at least an order of magnitude cheaper to compete against the other models out there. I'm not sure I know a lot of employees who would allocate that sort of constant funding to a consumables tool for an employee, given that's your usual monthly cost for a typical saas product.

intelVISA 2y ago

The avg. SWE is a toss up if they create more issues than they solve over time. Factor in on-boarding, bugs and taking time away from other expensive people becomes >$100/hr real quick.

zipy124 2y ago
If this were true there would be a lot less hiring of suggests engineers in the world and a lot more people getting laid off.
- intelVISA 2y ago
  There would be if it was easy to recognize. You generally can't see this due to all the middle men and bloat, and that no corp really wants to measure dev productivity against revenue like this as it'd raise a lot of uncomfortable questions by both devs and shareholders alike.