2 ms·

That is not my experience at all. Models are still leapfrogging each other every month in e.g. coding or research capability, or even in more mundane tasks suc

That is not my experience at all. Models are still leapfrogging each other every month in e.g. coding or research capability, or even in more mundane tasks such as summerizing long multi topic texts. Depending on which side of the issue you fall, you're hoping this will go on for a long time to come, or praying that it will end asap. I'm not using the cheapest in neither my own support, nor in my production systems.

nitwit005 2y ago
There are a ton of users that just want help generating emails, or adding some stock photo-like images to a blog post. If the choice is between something that costs $10 a month or $20 a month, and both solve those use cases, it's rational to pick the cheap one.
- Ekaros 2y ago
  And there will probably also be choice of something free and something trial... Which mean even less money spend.
senectus1 2y ago
from what I've seen the "leapfrogging" is very very incremental. They all seems to racing to the plateau... It doesn't look like there will ever be a "stand out" leader and the product that each company presents to the market appears to be essentially the same product that everyone else presents to the market. Maybe with some slight twist to it that is easily recruitable or exceedable within a few months. This is the issue really. at some point the investors are all going to realize that non of their investments are going to be market leaders. When they get to that stage the bubble will well and truly pop.
- raducu 2y ago
  > They all seems to racing to the plateau. To me it feels there is no plateau and the models are already very useful and impactful. I believe there is no plateau because there is nothing objectively special or magical about the human mind and it all can and will be eventually solved, one hack at a time.
xiphias2 2y ago
There seems to be some part of the LLM capabilities being lost by hacking some benchmarks. Claude 3.7 is a great example of a model clearly beating 3.5 in all benchmarks, but slowly destroying my code base by adding lots of extra lines or hacking around my instructions (adding ,,if'' statements when I want it to change the code to handle a case instead of understanding what change is really needed to be done for it). I still prefer o1 pro and a lot of those leapfrogging in benchmarks don't translate to being smarter anymore.

nitwit005 2y ago

There are a ton of users that just want help generating emails, or adding some stock photo-like images to a blog post. If the choice is between something that costs $10 a month or $20 a month, and both solve those use cases, it's rational to pick the cheap one.

Ekaros 2y ago
And there will probably also be choice of something free and something trial... Which mean even less money spend.

senectus1 2y ago

from what I've seen the "leapfrogging" is very very incremental. They all seems to racing to the plateau... It doesn't look like there will ever be a "stand out" leader and the product that each company presents to the market appears to be essentially the same product that everyone else presents to the market. Maybe with some slight twist to it that is easily recruitable or exceedable within a few months. This is the issue really. at some point the investors are all going to realize that non of their investments are going to be market leaders. When they get to that stage the bubble will well and truly pop.

raducu 2y ago
> They all seems to racing to the plateau. To me it feels there is no plateau and the models are already very useful and impactful. I believe there is no plateau because there is nothing objectively special or magical about the human mind and it all can and will be eventually solved, one hack at a time.

xiphias2 2y ago

There seems to be some part of the LLM capabilities being lost by hacking some benchmarks. Claude 3.7 is a great example of a model clearly beating 3.5 in all benchmarks, but slowly destroying my code base by adding lots of extra lines or hacking around my instructions (adding ,,if'' statements when I want it to change the code to handle a case instead of understanding what change is really needed to be done for it). I still prefer o1 pro and a lot of those leapfrogging in benchmarks don't translate to being smarter anymore.