11 ms·

There's no accounting for taste, but keep in mind that all of these services are currently losing money, so how much would you actually be willing to pay for th

There's no accounting for taste, but keep in mind that all of these services are currently losing money, so how much would you actually be willing to pay for the service you're currently getting in order to let it break even? There was a report that Microsoft is losing $20 for every $10 spent on Copilot subscriptions, with heavy users costing them as much as $80 per month. Assuming you're one of those heavy users, would you pay >$80 a month for it? Then there's chain-of-thought being positioned as the next big step forwards, which works by throwing more inferencing at the problem, so that cost can't be amortized over time like training can...

HPsquared 2y ago
It's called investment. You need to spend money to make money. Their costs will certainly come down.
JanSt 2y ago
1) The costs will go down over time, much of the cost is the margin of NVIDIA and training new models 2) Absolutely. Thats like one hour of an engineer salary for a whole month.
- sofixa 2y ago
  > The costs will go down over time, much of the cost is the margin of NVIDIA and training new models Isn't each new model bigger and heavier and thus requries more compute to train?
  - JanSt 2y ago
    Yes, but 1) you only need to train the model once and the inference is way cheaper. Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it. 2) the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA
    - sofixa 2y ago
      > Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? > the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA Where is the hardware that can compete with NVIDIA going to come from? And if they don't have competition, which they don't, why would they bring down prices?
      - JanSt 2y ago
        The point is not that every lab will be profitable. There only needs to be one model in the end to increase our productivity massively, which is the point I'm making. Huge margins lead to a lot of competition trying to catch up, which is what makes market economies so successful.
      - ben_w 2y ago
        > Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? Eventually one of you runs out of money, but your customers keep getting better models until then; and if the loser in this race releases the weights on a suitable gratis license then your businesses can both lose. But that still leaves your customers with access to a model that's much cheaper to run than it was to create.
      - Workaccount2 2y ago
        Gemini models are trained and run on Google's in house TPU's, which frankly are incredible compared to H100's. In fact Claude was trained on TPUs. Google however does not sell these, you can only lease time on them via GCP.
  - robrenaud 2y ago
    Then those new models get distilled into smaller ones. Raising the max intelligence of the models tends to raise the intelligence of all the models via distillation.
brookst 2y ago
Is there any reason to believe costs won’t come down with scale and hardware iteration, just like they did for everything else? Short term pricing inefficiency is not relevant to long term impact.
- HarHarVeryFunny 2y ago
  Of course, but every token generated by a 100B model is going to take minimally 100B FLOPS, and if this is being used as an IDE typing assistant then there is going to be a lot of tokens being generated. If there is a common shift to using additional runtime compute to improve quality of output, such as OpenAI's GPT-o1, then FLOPs required goes up massively (OpenAI has said it takes exponential increase in FLOPS/cost to generate linear gains in quality). So, while costs will of course decrease, those $20-30K NVIDEA chips are going to be kept burring, and are not going to pay for themselves ... This may end up like the shift to cloud computing that sounds good in theory (save the cost of running your own data center), but where corporate America balks when the bill comes in. It may well be that the endgame for corporate AI is to run free tools from the likes of Meta (or open source) in their own datacenter, or maybe even locally on "AI PCs".
  - wongarsu 2y ago
    Which is why the work to improve the results of small models is so important. Running a 3B or even 1B model as typing assistant and reserving the 100B model for refactoring is a lot more viable.
  - brookst 2y ago
    > but every token generated by a 100B model is going to take minimally 100B FLOPS Drop the S, I think. There’s no time dimension. And FLOP is a generalized capability meaning you can do any operation. Hardware optimizations for ML can deliver the same 100B computations faster and cheaper by not being completely generalized. Same way ray tracing acceleration works: it does not use the same amount of compute as ray tracing in general CPU’s.
    - 2y ago
archerx 2y ago
I can already pay $0 a month and use uncensored local models for both text and images. Llama, Mixtral, Stable diffusion and Flux are a lot of fun and free to run locally, you should try them out.
- jsheard 2y ago
  You can pay $0 for those models because a company paid $lots to train them and then released them for free. Those models aren't going away now of course, but lets not pretend that being able to download the product of millions of dollars worth of training completely free of charge is sustainable for future developments. Especially when most of the companies releasing these open models are wildly unprofitable and will inevitably bankrupt themselves when investments dry up unless they change their trajectory.
  - likium 2y ago
    Much could be said about open source libraries that companies release for free to use (kubernetes, react, firecracker, etc). It might be strategically make sense for them so in the meantime we’ll just reap the benefits.
    - skydhash 2y ago
      All of these require maintenance, and mostly it's been a treadmill just applying updates to React codebases. Complex tools are brittle and often only makes sense at the original source.
  - archerx 2y ago
    You’re acting as if computing power isn’t going to get better. With time training the models will get faster. Let me use CG rendering as an example. Back in the day only the big companies could afford to do photoreal 3D rendering because only they had access to the compute and even then it would take days to render a frame. Eventually people could do these renders at home with consumer hardware but it still took forever to render. Now we can render photoreal with path tracing at near realtime speeds. If you could go back twenty years and show CG artists the Unreal Engine 5 and show them it’s all realtime they would lose their minds. I see the same for A.I., now it’s only the big companies that can do it, then we will be able to do it at home but it will be slow and finally we will be able to train it at home for quick and cheap.
binocarlos 2y ago
I would pay hundreds of dollars per month for the combination of cursor and claude - I could not get my head around it when my beginner lever colleague said "I just coded this whole thing using cursor". It was an entire web app, with search filters, tree based drag and drop GUIs, the backend api server, database migrations, auth and everything else. Not once did he need to ask me a question. When I asked him "how long did this take" and expected him to say "a few weeks" (it would have taken me - a far more experienced engineer - 2 months minimum). His answer was "a few days". What I'm not saying is "AGI is close" but I've seen tangible evidence (only in the last 2 months), that my 20 year software engineering career is about to change and massively for the upside. Everyone is going to be so much more productive using these tools is how I see this.
- threeseed 2y ago
  > 20 year software engineering career is about to change I have also been developing for 20+ years. And have heard the exact same thing about IDEs, Search Engines, Stack Overflow, Github etc. But in my experience at least how fast I code has never been the limiting factor in my project's success. So LLMs are nice and all but isn't going to change the industry all that much.
  - pluc 2y ago
    There will be a whole industry of people who fix what AI has created. I don't know if it will be faster to build the wrong thing and pay to have it fixed or to build the right thing from the get go, but after having seen some shit, like you, I have a little idea.
    - dumbfounder 2y ago
      Correction: a whole industry of AI that will fix what AI has created.
      - vocram 2y ago
        Will AI also be on call when things break in production?
gotaran 2y ago
$80 a month is a no brainer given the productivity multiplier.
ema 2y ago
If it makes software developers 10% more productive there sure would be many companies who'd pay $80 a month per seat.
- apwell23 2y ago
  It actually makes them less productive and creates havoc in codebases with hidden bugs and verbose code that ppl are copy pasting.
- renegade-otter 2y ago
  It's like saying "AI is going to replace book writers because they are so much more productive now". All you will get is more mediocre content that someone will have to fix later - the same with code. 10% more productive. What does that mean? If you mean lines of code, then it's an incredibly poor metric. They write more code, faster. Then what? What are the long-term consequences? Is it ultimately a wash, or even a detriment? https://stackoverflow.blog/2024/03/22/is-ai-making-your-code-worse/ https://stackoverflow.blog/2024/03/22/is-ai-making-your-code...
  - ben_w 2y ago
    LLMs set a new minimum level; because of this they can fill in the gaps in a skillet — if I really suck at writing unit tests, they can bring me up from "none" to "it's a start". Likewise all the other specialities within software. Personally I am having a lot of fun, as an iOS developer, creating web games. No market in that, not really, but it's fun and I wouldn't have time to update my CSS and JS knowledge that was last up-to-date in 1998.
- HarHarVeryFunny 2y ago
  Maybe there are people out there working in coding sweatshops churning out boilerplate code 8 hours a day, 50 weeks a year - people who's job is 100% coding (not what I would call software engineers or developers - just coders). It's easy to imagine that for such people (but do they even exist?!) there could be large productivity gains. However, for a more typical software engineer, where every project is different, you have full lifecycle responsibility from design through coding, occasional production support, future enhancements, refactorings, updates for 3rd party library/OD updates, etc/etc, then how much of your time is actually spent pure coding (non-stop typing) ?! Probably closer to 10-25%, and certainly no-where near 100%. The potential overall time saving from a tool that saves, let's say, 10-25% of your code typing is going to be 1-5%, which is probably far less than gets wasted in meetings, chatting with your work buddies, or watching bullshit corporate training videos. IOW the savings is really just inconsequential noise. In many companies the work load is cyclic from one major project to the next, with intense periods of development interspersed with quieter periods in-between. Your productivity here certainly isn't limited by how fast you can type.
mklepaczewski 2y ago
I would, and I don't use chatgpt as much as other people. I would pay for it for each of my employees.
christkv 2y ago
Also at some point you can run the equivalent model locally. There is no long term moat here i think and facebook seems hellbent of ensuring there will be no new google from llms
- KoolKat23 2y ago
  I think physics at some point will get in the way, well at least for a while. An H100 costs like $20k-$30k and there's only so much compression/efficiency they can gain without beginning to lose intelligence, purely because you can't compute out of thin air.
ben_w 2y ago
> There's no accounting for taste, but keep in mind that all of these services are currently losing money, so how much would you actually be willing to pay for the service you're currently getting in order to let it break even Ok models already run locally; that aside, as the hosted ones are kinda similar quality to interns (though varying by field), the answer is "what you'd pay an intern". Could easily be £1500/month, depending on domain.
infecto 2y ago
Definitely. My time is valuable and I would spend multiples more on the current subscription costs.
fassssst 2y ago
Why do you assume they’re losing money on inference?
jejeyyy77 2y ago
- it won't work. - ok it works, but it won't be useful. - ok it's useful, but it won't scale. - ok it scales, but it won't make any money. - ok it makes money, but it's not going to last. etc etc
- pdinny 2y ago
  Retrospectively framing technologies that succeeded despite doubts at the time discounts those that failed. After all, you could have used the exact same response in defense of web3 tech. That doesn't mean LLMs are fated to be like web3, but similarly the outcome that the current expenditure can be recouped is far from a certainty just because there are doubters.
- farts_mckensy 2y ago
  There certainly has been some goal post moving over the past few months. A lot of the people in here have some kind of psychological block when it comes to technology that may potentially replace them one day.
  - KoolKat23 2y ago
    Yeah currently the sentiment seems to be "okay fine it works for simple stuff but won't deal with my complex query so it can be dismissed outright." Save yourselves some time and use it for that simple stuff folks.
chmod775 2y ago
> There was a report that Microsoft is losing $20 for every $10 spent on Copilot subscriptions, with heavy users costing them as much as $80 per month. Assuming you're one of those heavy users, would you pay >$80 a month for it? I'm probably one of those "heavy users", though I've only been using it for a month to see how well it does. Here's my review: Large completions (10-15 lines): It will generally spit out near-working code for any codemonkey-level framework-user frontend code, but for anything more it'll be at best amusing and a waste of time. Small completions (complete current line): Usually nails it and saves me a few keystrokes. The downside is that it competes for my attention/screen space against good old auto-completion, which costs me productivity every time it fucks up. Having to go back and fix identifiers in which it messed up the capitalization/had typos, where basic auto-complete wouldn't have failed is also annoying. I'd pay about about $40 right now because at least it has some entertainment value, being technologically interesting.
- jeremy151 2y ago
  I find tools where I am manually shepherding the context into an LLM to work much better than Copilot at current. If I think thru the problem enough to articulate it and give the model a clear explanation, and choose the surrounding pieces of context (the same stuff I would open up and look at as a dev) I can be pretty sure the code generated (even larger outputs) will work and do what I wanted, and be stylistically good. I am still adding a lot in this scenario, but it's heavier on the analysis and requirements side, and less on the code creation side. If what I give it is too open ended, doesn't have enough info, etc, I'll still get a low quality output. Though I find I can steer it by asking it to ask clarifying questions. Asking it to build unit tests can help a lot too in bolstering, a few iterations getting the unit tests created and passing can really push the quality up.
jsemrau 2y ago
Just a thought exercise. If we would have an AI with the intellectual capabilities of a Ph.D holding professor in a hard science. How much would it be worth for you to have access to that AI? 100,000 ? 500,000 ?
- salawat 2y ago
  0 unless what I'm interested in is that Professor's very narrowly tailored niche. It's called Piled Higher and Deeper for a reason.
refulgentis 2y ago
CoT is not RL'ing over reasoning traces, costs have come down 87.5% since that article, and I agree generally that "free" is a bad price point
BaculumMeumEst 2y ago
I don't find this very compelling. Hardware is becoming more available and cheaper as production ramps up, and smaller models are constantly seeing dramatic improvements.
lupire 2y ago
People hate paying specifically for stuff. If Copilot came for free and Azure cost a tiny bit more, nobody would even blink.
CamperBob2 2y ago
There's no accounting for taste, but keep in mind that all of these services are currently losing money, so how much would you actually be willing to pay for the service you're currently getting in order to let it break even? For ChatGPT in its current state, probably $1K/month.
lhl 2y ago
When was this profitability report, because the cost per token generation has dropped significantly. When GPT4 was launched last year, the API cost was about $36/M blended tokens, but you can now get GPT4o tokens for about $4.4/M tokens, Gemini 1.5 Pro for $2.2/M or DeepSeek-V2 (as 21B A/236B W model that matches GPT4 on coding) for as low as $0.28/M tokens (over 100X cheaper for the same quality output over the course of about 1.5 years). The just released Qwen2.5-Coder-7B-Instruct (Apache 2.0 licensed) also basically matches/beats GPT4 on coding benchmarks and quantized can not only can run at a decent speed on just about any consumer gaming GPU, but on most new CPUs/NPUs as well. This is about a 250X smaller model than GPT4. There are now a huge array of open weight (and open source) models that are very capable and that can be run locally/on the edge.

HPsquared 2y ago

It's called investment. You need to spend money to make money. Their costs will certainly come down.

JanSt 2y ago

1) The costs will go down over time, much of the cost is the margin of NVIDIA and training new models 2) Absolutely. Thats like one hour of an engineer salary for a whole month.

sofixa 2y ago
> The costs will go down over time, much of the cost is the margin of NVIDIA and training new models Isn't each new model bigger and heavier and thus requries more compute to train?
- JanSt 2y ago
  Yes, but 1) you only need to train the model once and the inference is way cheaper. Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it. 2) the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA
  - sofixa 2y ago
    > Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? > the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA Where is the hardware that can compete with NVIDIA going to come from? And if they don't have competition, which they don't, why would they bring down prices?
    - JanSt 2y ago
      The point is not that every lab will be profitable. There only needs to be one model in the end to increase our productivity massively, which is the point I'm making. Huge margins lead to a lot of competition trying to catch up, which is what makes market economies so successful.
    - ben_w 2y ago
      > Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? Eventually one of you runs out of money, but your customers keep getting better models until then; and if the loser in this race releases the weights on a suitable gratis license then your businesses can both lose. But that still leaves your customers with access to a model that's much cheaper to run than it was to create.
    - Workaccount2 2y ago
      Gemini models are trained and run on Google's in house TPU's, which frankly are incredible compared to H100's. In fact Claude was trained on TPUs. Google however does not sell these, you can only lease time on them via GCP.
- robrenaud 2y ago
  Then those new models get distilled into smaller ones. Raising the max intelligence of the models tends to raise the intelligence of all the models via distillation.

brookst 2y ago

Is there any reason to believe costs won’t come down with scale and hardware iteration, just like they did for everything else? Short term pricing inefficiency is not relevant to long term impact.

HarHarVeryFunny 2y ago
Of course, but every token generated by a 100B model is going to take minimally 100B FLOPS, and if this is being used as an IDE typing assistant then there is going to be a lot of tokens being generated. If there is a common shift to using additional runtime compute to improve quality of output, such as OpenAI's GPT-o1, then FLOPs required goes up massively (OpenAI has said it takes exponential increase in FLOPS/cost to generate linear gains in quality). So, while costs will of course decrease, those $20-30K NVIDEA chips are going to be kept burring, and are not going to pay for themselves ... This may end up like the shift to cloud computing that sounds good in theory (save the cost of running your own data center), but where corporate America balks when the bill comes in. It may well be that the endgame for corporate AI is to run free tools from the likes of Meta (or open source) in their own datacenter, or maybe even locally on "AI PCs".
- wongarsu 2y ago
  Which is why the work to improve the results of small models is so important. Running a 3B or even 1B model as typing assistant and reserving the 100B model for refactoring is a lot more viable.
- brookst 2y ago
  > but every token generated by a 100B model is going to take minimally 100B FLOPS Drop the S, I think. There’s no time dimension. And FLOP is a generalized capability meaning you can do any operation. Hardware optimizations for ML can deliver the same 100B computations faster and cheaper by not being completely generalized. Same way ray tracing acceleration works: it does not use the same amount of compute as ray tracing in general CPU’s.
  - 2y ago

archerx 2y ago

I can already pay $0 a month and use uncensored local models for both text and images. Llama, Mixtral, Stable diffusion and Flux are a lot of fun and free to run locally, you should try them out.

jsheard 2y ago
You can pay $0 for those models because a company paid $lots to train them and then released them for free. Those models aren't going away now of course, but lets not pretend that being able to download the product of millions of dollars worth of training completely free of charge is sustainable for future developments. Especially when most of the companies releasing these open models are wildly unprofitable and will inevitably bankrupt themselves when investments dry up unless they change their trajectory.
- likium 2y ago
  Much could be said about open source libraries that companies release for free to use (kubernetes, react, firecracker, etc). It might be strategically make sense for them so in the meantime we’ll just reap the benefits.
  - skydhash 2y ago
    All of these require maintenance, and mostly it's been a treadmill just applying updates to React codebases. Complex tools are brittle and often only makes sense at the original source.
- archerx 2y ago
  You’re acting as if computing power isn’t going to get better. With time training the models will get faster. Let me use CG rendering as an example. Back in the day only the big companies could afford to do photoreal 3D rendering because only they had access to the compute and even then it would take days to render a frame. Eventually people could do these renders at home with consumer hardware but it still took forever to render. Now we can render photoreal with path tracing at near realtime speeds. If you could go back twenty years and show CG artists the Unreal Engine 5 and show them it’s all realtime they would lose their minds. I see the same for A.I., now it’s only the big companies that can do it, then we will be able to do it at home but it will be slow and finally we will be able to train it at home for quick and cheap.

binocarlos 2y ago

I would pay hundreds of dollars per month for the combination of cursor and claude - I could not get my head around it when my beginner lever colleague said "I just coded this whole thing using cursor". It was an entire web app, with search filters, tree based drag and drop GUIs, the backend api server, database migrations, auth and everything else. Not once did he need to ask me a question. When I asked him "how long did this take" and expected him to say "a few weeks" (it would have taken me - a far more experienced engineer - 2 months minimum). His answer was "a few days". What I'm not saying is "AGI is close" but I've seen tangible evidence (only in the last 2 months), that my 20 year software engineering career is about to change and massively for the upside. Everyone is going to be so much more productive using these tools is how I see this.

threeseed 2y ago
> 20 year software engineering career is about to change I have also been developing for 20+ years. And have heard the exact same thing about IDEs, Search Engines, Stack Overflow, Github etc. But in my experience at least how fast I code has never been the limiting factor in my project's success. So LLMs are nice and all but isn't going to change the industry all that much.
- pluc 2y ago
  There will be a whole industry of people who fix what AI has created. I don't know if it will be faster to build the wrong thing and pay to have it fixed or to build the right thing from the get go, but after having seen some shit, like you, I have a little idea.
  - dumbfounder 2y ago
    Correction: a whole industry of AI that will fix what AI has created.
    - vocram 2y ago
      Will AI also be on call when things break in production?

gotaran 2y ago

$80 a month is a no brainer given the productivity multiplier.

ema 2y ago

If it makes software developers 10% more productive there sure would be many companies who'd pay $80 a month per seat.

apwell23 2y ago
It actually makes them less productive and creates havoc in codebases with hidden bugs and verbose code that ppl are copy pasting.
renegade-otter 2y ago
It's like saying "AI is going to replace book writers because they are so much more productive now". All you will get is more mediocre content that someone will have to fix later - the same with code. 10% more productive. What does that mean? If you mean lines of code, then it's an incredibly poor metric. They write more code, faster. Then what? What are the long-term consequences? Is it ultimately a wash, or even a detriment? https://stackoverflow.blog/2024/03/22/is-ai-making-your-code-worse/ https://stackoverflow.blog/2024/03/22/is-ai-making-your-code...
- ben_w 2y ago
  LLMs set a new minimum level; because of this they can fill in the gaps in a skillet — if I really suck at writing unit tests, they can bring me up from "none" to "it's a start". Likewise all the other specialities within software. Personally I am having a lot of fun, as an iOS developer, creating web games. No market in that, not really, but it's fun and I wouldn't have time to update my CSS and JS knowledge that was last up-to-date in 1998.
HarHarVeryFunny 2y ago
Maybe there are people out there working in coding sweatshops churning out boilerplate code 8 hours a day, 50 weeks a year - people who's job is 100% coding (not what I would call software engineers or developers - just coders). It's easy to imagine that for such people (but do they even exist?!) there could be large productivity gains. However, for a more typical software engineer, where every project is different, you have full lifecycle responsibility from design through coding, occasional production support, future enhancements, refactorings, updates for 3rd party library/OD updates, etc/etc, then how much of your time is actually spent pure coding (non-stop typing) ?! Probably closer to 10-25%, and certainly no-where near 100%. The potential overall time saving from a tool that saves, let's say, 10-25% of your code typing is going to be 1-5%, which is probably far less than gets wasted in meetings, chatting with your work buddies, or watching bullshit corporate training videos. IOW the savings is really just inconsequential noise. In many companies the work load is cyclic from one major project to the next, with intense periods of development interspersed with quieter periods in-between. Your productivity here certainly isn't limited by how fast you can type.

mklepaczewski 2y ago

I would, and I don't use chatgpt as much as other people. I would pay for it for each of my employees.

christkv 2y ago

Also at some point you can run the equivalent model locally. There is no long term moat here i think and facebook seems hellbent of ensuring there will be no new google from llms

KoolKat23 2y ago
I think physics at some point will get in the way, well at least for a while. An H100 costs like $20k-$30k and there's only so much compression/efficiency they can gain without beginning to lose intelligence, purely because you can't compute out of thin air.

ben_w 2y ago

> There's no accounting for taste, but keep in mind that all of these services are currently losing money, so how much would you actually be willing to pay for the service you're currently getting in order to let it break even Ok models already run locally; that aside, as the hosted ones are kinda similar quality to interns (though varying by field), the answer is "what you'd pay an intern". Could easily be £1500/month, depending on domain.

infecto 2y ago

Definitely. My time is valuable and I would spend multiples more on the current subscription costs.

fassssst 2y ago

Why do you assume they’re losing money on inference?

jejeyyy77 2y ago

- it won't work. - ok it works, but it won't be useful. - ok it's useful, but it won't scale. - ok it scales, but it won't make any money. - ok it makes money, but it's not going to last. etc etc

pdinny 2y ago
Retrospectively framing technologies that succeeded despite doubts at the time discounts those that failed. After all, you could have used the exact same response in defense of web3 tech. That doesn't mean LLMs are fated to be like web3, but similarly the outcome that the current expenditure can be recouped is far from a certainty just because there are doubters.
farts_mckensy 2y ago
There certainly has been some goal post moving over the past few months. A lot of the people in here have some kind of psychological block when it comes to technology that may potentially replace them one day.
- KoolKat23 2y ago
  Yeah currently the sentiment seems to be "okay fine it works for simple stuff but won't deal with my complex query so it can be dismissed outright." Save yourselves some time and use it for that simple stuff folks.

chmod775 2y ago

> There was a report that Microsoft is losing $20 for every $10 spent on Copilot subscriptions, with heavy users costing them as much as $80 per month. Assuming you're one of those heavy users, would you pay >$80 a month for it? I'm probably one of those "heavy users", though I've only been using it for a month to see how well it does. Here's my review: Large completions (10-15 lines): It will generally spit out near-working code for any codemonkey-level framework-user frontend code, but for anything more it'll be at best amusing and a waste of time. Small completions (complete current line): Usually nails it and saves me a few keystrokes. The downside is that it competes for my attention/screen space against good old auto-completion, which costs me productivity every time it fucks up. Having to go back and fix identifiers in which it messed up the capitalization/had typos, where basic auto-complete wouldn't have failed is also annoying. I'd pay about about $40 right now because at least it has some entertainment value, being technologically interesting.

jeremy151 2y ago
I find tools where I am manually shepherding the context into an LLM to work much better than Copilot at current. If I think thru the problem enough to articulate it and give the model a clear explanation, and choose the surrounding pieces of context (the same stuff I would open up and look at as a dev) I can be pretty sure the code generated (even larger outputs) will work and do what I wanted, and be stylistically good. I am still adding a lot in this scenario, but it's heavier on the analysis and requirements side, and less on the code creation side. If what I give it is too open ended, doesn't have enough info, etc, I'll still get a low quality output. Though I find I can steer it by asking it to ask clarifying questions. Asking it to build unit tests can help a lot too in bolstering, a few iterations getting the unit tests created and passing can really push the quality up.

jsemrau 2y ago

Just a thought exercise. If we would have an AI with the intellectual capabilities of a Ph.D holding professor in a hard science. How much would it be worth for you to have access to that AI? 100,000 ? 500,000 ?

salawat 2y ago
0 unless what I'm interested in is that Professor's very narrowly tailored niche. It's called Piled Higher and Deeper for a reason.

refulgentis 2y ago

CoT is not RL'ing over reasoning traces, costs have come down 87.5% since that article, and I agree generally that "free" is a bad price point

BaculumMeumEst 2y ago

I don't find this very compelling. Hardware is becoming more available and cheaper as production ramps up, and smaller models are constantly seeing dramatic improvements.

lupire 2y ago

People hate paying specifically for stuff. If Copilot came for free and Azure cost a tiny bit more, nobody would even blink.

CamperBob2 2y ago

lhl 2y ago

When was this profitability report, because the cost per token generation has dropped significantly. When GPT4 was launched last year, the API cost was about $36/M blended tokens, but you can now get GPT4o tokens for about $4.4/M tokens, Gemini 1.5 Pro for $2.2/M or DeepSeek-V2 (as 21B A/236B W model that matches GPT4 on coding) for as low as $0.28/M tokens (over 100X cheaper for the same quality output over the course of about 1.5 years). The just released Qwen2.5-Coder-7B-Instruct (Apache 2.0 licensed) also basically matches/beats GPT4 on coding benchmarks and quantized can not only can run at a decent speed on just about any consumer gaming GPU, but on most new CPUs/NPUs as well. This is about a 250X smaller model than GPT4. There are now a huge array of open weight (and open source) models that are very capable and that can be run locally/on the edge.