7 ms·

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude

by Liquix 2y ago

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

iknownthing 2y ago
and even if there is another breakthrough all of these companies will implement it more or less simultaneously and they will remain in a dead heat
- llm_nerd 2y ago
  Presuming the breakthrough is openly shared. It remains surprising how transparent many of these companies are about new approaches that push the SoTa forward, and I suspect we're going to see a change. That companies won't reveal the secret sauce so readily. e.g. Almost the entire market relies upon Attention Is All You Need paper detailing transformers, and it would be an entirely different market if Google had held that as a trade secret.
  - talldayo 2y ago
    Given how absolutely pitiful the proprietary advancements in AI have been, I would posit we have little to worry about.
    - jsheard 2y ago
      OTOH the companies who are sharing their breakthroughs openly aren't yet making any money, so something has to give. Their research is currently being bankrolled by investors who assume there will be returns eventually, and eventually can only be kicked down the road for so long.
      - talldayo 2y ago
        Eventually can be (and has been) bankrolled by Nvidia. They did a lot of ground-floor research on GANs and training optimization, which only makes sense to release as public research. Similarly, Meta and Google are both well-incentivized to share their research through Pytorch and Tensorflow respectively. I really am not expecting Apple or Microsoft to discover AGI and ferret it away for profitability purposes. Strictly speaking, I don't think superhuman intelligence even exists in the domain of text generation.
      - thruway516 2y ago
        Well, that's because the potential reward from picking the right horse is MASSIVE and the cost of potentially missing out is lifelong regret. Investors are driven by FOMO more than anything else. They know most of these will be duds but one of these duds could turn out to be life changing. So they will keep bankrolling as long as they have the money.
      - michaelt 2y ago
        Sort of yes, sort of no. Of course, I agree that Stability AI made Stable Diffusion freely available and they're worth orders of magnitude less than OpenAI. To the point they're struggling to keep the lights on. But it doesn't necessarily make that much difference whether you openly share the inner technical details. When you've got a motivated and well financed competitor, merely demonstrating a given feature is possible, showing the output and performance and price, might be enough. If OpenAI adds a feature, who's to say Google and Facebook can't match it even though they can't access the code?
    - sebzim4500 2y ago
      Anthropic has been very secretive about the supposed synthetic data they used to train 3.5 Sonnet. Given how good the model is terms of the quality vs speed tradeoff, they must have something.
  - GaggiX 2y ago
    >Attention Is All You Need paper detailing transformers, and it would be an entirely different market if Google had held that as a trade secret. I would guess that in that timeline, Google would never have been able to learn about the incredible capabilities of transformer models outside of translation, at least not until much later.
happyhardcore 2y ago
I suspect this is why OpenAI is going more in the direction of optimising for price / latency / whatever with 4o-mini and whatnot. Presumably they found out long before the rest of us did that models can't really get all that much better than what we're approaching now, and once you're there the only thing you can compete on is how many parameters it takes and how cheaply you can serve that to users.
- __jl__ 2y ago
  Meta just claimed the opposite in their Llama 3.1 paper. Look at the conclusion. They say that their experience indicates significant gains for the next iteration of models. The current crop of benchmarks might not reflect these gains, by the way.
  - nathanasmith 2y ago
    They also said in the paper that 405B was only trained to "compute-optimal" unlike the smaller models that were trained well past that point indicating the larger model still had some runway so had they continued it would have kept getting stronger.
    - moffkalast 2y ago
      Makes sense right? Otherwise why make a model so large that nobody can conceivably run it if not to optimize for performance on a limited dataset/compute? It was always a distillation source model, not a production one.
  - imtringued 2y ago
    LLMs are reaching saturation on even some of the latest benchmarks and yet I am still a little disappointed by how they perform in practice. They are by no means bad, but I am now mostly interested in long context competency. We need benchmarks that force the LLM to complete multiple tasks simultaneously in one super long session.
    - 2y ago
deleted 2y ago
[deleted]
ActorNightly 2y ago
The thing I don't understand is why everyone is throwing money at LLMs for language, when there are much simpler use cases which are more useful? For example, has anyone ever attempted image -> html/css model? Seems like it be great if I can draw something on a piece of paper and have it generate a website view for me.
- ascorbic 2y ago
  All of the multi-modal LLMs are reasonably good at this.
- jacobn 2y ago
  I was under the impression that you could more or less do something like that with the existing LLMs? (May work poorly of course, and the sample I think I saw a year ago may well be cherry picked)
- GaggiX 2y ago
  >For example, has anyone ever attempted image -> html/css model? Have you tried upload the image to a LLM with vision capabilities like GPT-4o or Claude 3.5 Sonnet?
  - machiaweliczny 2y ago
    I tried and sonnet 3.5 can copy most of common UIs
- majiy 2y ago
  That's a thought I had. For example, could a model be trained to take a description, and create a Blender (or whatever other software) model from it? I have no idea how LLMs really work under the hood, so please tell me if this is nonsense.
  - eurekin 2y ago
    I'm waiting exactly for this, gpt4 trips up a lot with blender currently (nonsensical order of operations etc.)
swyx 2y ago
indeed. I pointed out in https://buttondown.email/ainews/archive/ainews-llama-31-the-synthetic-data-model/ https://buttondown.email/ainews/archive/ainews-llama-31-the-... that the frontier model curve is currently going down 1 OoM every 4 months, meaning every model release has a very short half life[0]. however this progress is still worth it if we can deploy it to improve millions and eventually billions of people's lives. a commenter pointed out that the amoutn spent on Llama 3.1 was only like 60% of the cost of Ant Man and the Wasp Quantumania, in which case I'd advocate for killing all Marvel slop and dumping all that budget on LLM progress. [0] not technically complete depreciation, since for example 4o mini is widely believed to be a distillation of 4o, so 4o's investment still carries over into 4o mini
- thierrydamiba 2y ago
  Agreed on everything, but calling the marvel movies slop…I think that word has gone too far.
  - ThrowawayTestr 2y ago
    The marvel movies are the genesis for this use of the word slop.
    - simonw 2y ago
      Can you back that claim up with a link or similar?
  - RUnconcerned 2y ago
    Not only are Marvel movies slop, they are very concentrated slop. The only way to increase the concentration of slop in a Marvel movie would be to ask ChatGPT to write the next one.
  - mattnewton 2y ago
    Not all Marvel films are slop. But, as a fan who comes from a family of fans and someone who has watched almost all of them; lets be real. That particular film, really and most of them, contain copious amounts of what is absolutely slop. I don't know if the utility is worse than an LLM that is SOTA for 2 months that no one even bothers switching to however - at least the marvel slop is being used for entertainment by someone. I think the market is definitely prioritizing the LLM researcher over Disney's latest slop sequel though so whoever made that comparison can rest easy, because we'll find out.
chipdart 2y ago
> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; I think you're just seeing the "make it work" stage of the combo "first make it work, then make it fast". Time to market is critical, as you can attest by the fact you framed the situation as "on par with GPT-4o and Claude Opus". You're seeing huge investments because being the first to get a working model stands to benefit greatly. You can only assess models that exist, and for that you need to train them at a huge computational cost.
- romeros 2y ago
  ChatGPT is like Google now. It is the default. Even if Claude becomes as good as ChatGPT or even slightly better it won't make me switch. It has to be like a lot better. Way better. It feels like ChatGPT won the time to market war already.
  - Tostino 2y ago
    Eh, with the degradation of coding performance in ChatGPT I made the switch. Seems much better to work with on problems, and I have to do way less hand holding to get good results. I'll switch again soon as something better is out.
  - brandall10 2y ago
    But plenty people switched to Claude, esp. with Sonnet 3.5. Many of them in this very thread. You may be right with the average person on the street, but I wonder how many have lost interest in LLM usage and cancelled their GPT plus sub.
  - asah 2y ago
    -1: I know many people who are switching to Claude. And Google makes it near-zero friction to adopt Gemini with Gsuite. And more still are using the top-N of them. This is similar to the early days of the search engine wars, the browser wars, and other categories where a user can easily adopt, switch between and use multiple. It's not like the cellphone OS/hardware war, PC war and database war where (most) users can only adopt one platform at a time and/or there's a heavy platform investment.
Workaccount2 2y ago
I think GPT5 will be the signal of whether or not we have hit a plateau. The space is still rapidly developing, and while large model gains are getting harder to pick apart, there have been enormous gains in the capabilities of light weight models.
- zainhoda 2y ago
  I’m waiting for the same signal. There are essentially 2 vastly different states of the world depending on whether GPT-5 is an incremental change vs a step change compared to GPT-4.
- chipdart 2y ago
  > I think GPT5 will be the signal of whether or not we have hit a plateau. I think GPT5 will tell if OpenAI hit a plateau. Sam Altman has been quoted as claiming "GPT-3 had the intelligence of a toddler, GPT-4 was more similar to a smart high-schooler, and that the next generation will look to have PhD-level intelligence (in certain tasks)" Notice the high degree of upselling based on vague claims of performance, and the fact that the jump from highschooler to PhD can very well be far less impressive than the jump from toddler to high schooler. In addition, notice the use of weasel words to frame expectations regarding "the next generation" to limit these gains to corner cases. There's some degree of salesmanship in the way these models are presented, but even between the hyperboles you don't see claims of transformative changes.
  - rvnx 2y ago
    PhD level-of-task-execution sounds like the LLM will debate whether the task is ethical instead of actually doing it
    - airspresso 2y ago
      lol! Producing academic papers for future training runs then.
    - throwadobe 2y ago
      I wish I could frame this comment
speed_spread 2y ago
Benchmarks scores aren't good because they apply to previous generations of LLMs. That 2.23% uptick can actually represent a world of difference in subjective tests and definitely be worth the investment. Progress is not slowing down but it gets harder to quantify.
satvikpendem 2y ago
This is already what the chinchilla paper surmised, it's no wonder that their prediction now comes to fruition. It is like an accelerated version of Moore's Law, because software development itself is more accelerated than hardware development.
genrilz 2y ago
For this model, it seems like the point is that it uses way less parameters than at least the large Llama model while having near identical performance. Given how large these models are getting, this is an important thing to do before making performance better again.
deleted 2y ago
[deleted]
skybrian 2y ago
I think it’s impressive that they’re doing it on a single (large) node. Costs matter. Efficiency improvements like this will probably increase capabilities eventually. I’m also optimistic about building better (rather than bigger) datasets to train on.
BoredPositron 2y ago
We always needed a tock to see real advancement, like with the last model generation. The tick we had with the h100 was enough to bring these models to market but that's it.
lossolo 2y ago
For some time, we have been at a plateau because everyone has caught up, which essentially means that everyone now has good training datasets and uses similar tweaks to the architecture. It seems that, besides new modalities, transformers might be a dead end as an architecture. Better scores on benchmarks result from better training data and fine-tuning. The so-called 'agents' and 'function calling' also boil down to training data and fine-tuning.
lolinder 2y ago
> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs Yes. This is exactly why I'm skeptical of AI doomerism/saviorism. Too many people have been looking at the pace of LLM development over the last two (2) years, modeled it as an exponential growth function, and come to the conclusion that AGI is inevitable in the next ${1-5} years and we're headed for ${(dys|u)topia}. But all that assumes that we can extrapolate a pattern of long-term exponential growth from less than two years of data. It's simply not possible to project in that way, and we're already seeing that OpenAI has pivoted from improving on GPT-4's benchmarks to reducing cost, while competitors (including free ones) catch up. All the evidence suggests we've been slowing the rate of growth in capabilities of SOTA LLMs for at least the past year, which means predictions based on exponential growth all need to be reevaluated.
- cjalmeida 2y ago
  Indeed.All exponential growth curves are sigmoids in disguise.
  - nicman23 2y ago
    except when it isn't and we ded :P
    - kridsdale3 2y ago
      I don't think Special Relativity would allow that.
  - ToValueFunfetti 2y ago
    This is something that is definitionally true in a finite universe, but doesn't carry a lot of useful predictive value in practice unless you can identify when the flattening will occur. If you have a machine that converts mass into energy and then uses that energy to increase the rate at which it operates, you could rightfully say that it will level off well before consuming all of the mass in the universe. You just can't say that next week after it has consumed all of the mass of Earth.
niemandhier 2y ago
The next iteration depends on NVIDIA & co, what we need is sparse libs. Most of the weights in llms are 0, once we deal with those more efficiently we will get to the next iteration.
- lawlessone 2y ago
  > Most of the weights in llms are 0, that's interesting. Do you have a rough percentage of this? Does this mean these connections have no influence at all on output?
  - machiaweliczny 2y ago
    My uneducated guess is that with many layers you can implement something akin to graph in brain by nulling lots of previous later outputs. I actually suspect that current models aren’t optimal with layers all of the same size but i know shit
    - kridsdale3 2y ago
      This is quite intuitive. We know that a biological neural net is a graph data structure. And ML systems on GPUs are more like layers of bitmaps in Photoshop (it's a graphics processor). So if most of the layers are akin to transparent pixels, in order to build a graph by stacking, that's hyper memory inefficient.
m3kw9 2y ago
There is different directions AI have lots to improve: multi modal which branch into robotics, single modal like image, video, and sound generation and understanding. Also would check back when openAI releases 5
swalsh 2y ago
And with the increasing parameter size, the main winner will be Nvidia. Frankly I just don't understand the economics of training a foundation model. I'd rather own an airline. At least I can get a few years out of the capital investment of a plane.
- machiaweliczny 2y ago
  But billionaires already have that, they want a chance of getting their own god.
mlsu 2y ago
What else can be done? If you are sitting on 1 billions $ of GPU capex, what's $50 million in energy/training cost for another incremental run that may beat the leaderboard? Over the last few years the market has placed its bets that this stuff will make gobs of money somehow. We're all not sure how. They're probably thinking -- it's likely that whoever has a few % is going to sweep and take most of this hypothetical value. What's another few million, especially if you already have the GPUs? I think you're right -- we are towards the right end of the sigmoid. And with no "killer app" in sight. It is great for all of us that they have created all this value, because I don't think anyone will be able to capture it. They certainly haven't yet.
sebzim4500 2y ago
I don't think we can conclude that until someone trains a model that is significantly bigger than GPT-4.

by Liquix 2y ago

iknownthing 2y ago

and even if there is another breakthrough all of these companies will implement it more or less simultaneously and they will remain in a dead heat

llm_nerd 2y ago
Presuming the breakthrough is openly shared. It remains surprising how transparent many of these companies are about new approaches that push the SoTa forward, and I suspect we're going to see a change. That companies won't reveal the secret sauce so readily. e.g. Almost the entire market relies upon Attention Is All You Need paper detailing transformers, and it would be an entirely different market if Google had held that as a trade secret.
- talldayo 2y ago
  Given how absolutely pitiful the proprietary advancements in AI have been, I would posit we have little to worry about.
  - jsheard 2y ago
    OTOH the companies who are sharing their breakthroughs openly aren't yet making any money, so something has to give. Their research is currently being bankrolled by investors who assume there will be returns eventually, and eventually can only be kicked down the road for so long.
    - talldayo 2y ago
      Eventually can be (and has been) bankrolled by Nvidia. They did a lot of ground-floor research on GANs and training optimization, which only makes sense to release as public research. Similarly, Meta and Google are both well-incentivized to share their research through Pytorch and Tensorflow respectively. I really am not expecting Apple or Microsoft to discover AGI and ferret it away for profitability purposes. Strictly speaking, I don't think superhuman intelligence even exists in the domain of text generation.
    - thruway516 2y ago
      Well, that's because the potential reward from picking the right horse is MASSIVE and the cost of potentially missing out is lifelong regret. Investors are driven by FOMO more than anything else. They know most of these will be duds but one of these duds could turn out to be life changing. So they will keep bankrolling as long as they have the money.
    - michaelt 2y ago
      Sort of yes, sort of no. Of course, I agree that Stability AI made Stable Diffusion freely available and they're worth orders of magnitude less than OpenAI. To the point they're struggling to keep the lights on. But it doesn't necessarily make that much difference whether you openly share the inner technical details. When you've got a motivated and well financed competitor, merely demonstrating a given feature is possible, showing the output and performance and price, might be enough. If OpenAI adds a feature, who's to say Google and Facebook can't match it even though they can't access the code?
  - sebzim4500 2y ago
    Anthropic has been very secretive about the supposed synthetic data they used to train 3.5 Sonnet. Given how good the model is terms of the quality vs speed tradeoff, they must have something.
- GaggiX 2y ago
  >Attention Is All You Need paper detailing transformers, and it would be an entirely different market if Google had held that as a trade secret. I would guess that in that timeline, Google would never have been able to learn about the incredible capabilities of transformer models outside of translation, at least not until much later.

happyhardcore 2y ago

I suspect this is why OpenAI is going more in the direction of optimising for price / latency / whatever with 4o-mini and whatnot. Presumably they found out long before the rest of us did that models can't really get all that much better than what we're approaching now, and once you're there the only thing you can compete on is how many parameters it takes and how cheaply you can serve that to users.

__jl__ 2y ago
Meta just claimed the opposite in their Llama 3.1 paper. Look at the conclusion. They say that their experience indicates significant gains for the next iteration of models. The current crop of benchmarks might not reflect these gains, by the way.
- nathanasmith 2y ago
  They also said in the paper that 405B was only trained to "compute-optimal" unlike the smaller models that were trained well past that point indicating the larger model still had some runway so had they continued it would have kept getting stronger.
  - moffkalast 2y ago
    Makes sense right? Otherwise why make a model so large that nobody can conceivably run it if not to optimize for performance on a limited dataset/compute? It was always a distillation source model, not a production one.
- imtringued 2y ago
  LLMs are reaching saturation on even some of the latest benchmarks and yet I am still a little disappointed by how they perform in practice. They are by no means bad, but I am now mostly interested in long context competency. We need benchmarks that force the LLM to complete multiple tasks simultaneously in one super long session.
  - 2y ago

deleted 2y ago

[deleted]

ActorNightly 2y ago

The thing I don't understand is why everyone is throwing money at LLMs for language, when there are much simpler use cases which are more useful? For example, has anyone ever attempted image -> html/css model? Seems like it be great if I can draw something on a piece of paper and have it generate a website view for me.

ascorbic 2y ago
All of the multi-modal LLMs are reasonably good at this.
jacobn 2y ago
I was under the impression that you could more or less do something like that with the existing LLMs? (May work poorly of course, and the sample I think I saw a year ago may well be cherry picked)
GaggiX 2y ago
>For example, has anyone ever attempted image -> html/css model? Have you tried upload the image to a LLM with vision capabilities like GPT-4o or Claude 3.5 Sonnet?
- machiaweliczny 2y ago
  I tried and sonnet 3.5 can copy most of common UIs
majiy 2y ago
That's a thought I had. For example, could a model be trained to take a description, and create a Blender (or whatever other software) model from it? I have no idea how LLMs really work under the hood, so please tell me if this is nonsense.
- eurekin 2y ago
  I'm waiting exactly for this, gpt4 trips up a lot with blender currently (nonsensical order of operations etc.)

swyx 2y ago

indeed. I pointed out in https://buttondown.email/ainews/archive/ainews-llama-31-the-synthetic-data-model/ https://buttondown.email/ainews/archive/ainews-llama-31-the-... that the frontier model curve is currently going down 1 OoM every 4 months, meaning every model release has a very short half life[0]. however this progress is still worth it if we can deploy it to improve millions and eventually billions of people's lives. a commenter pointed out that the amoutn spent on Llama 3.1 was only like 60% of the cost of Ant Man and the Wasp Quantumania, in which case I'd advocate for killing all Marvel slop and dumping all that budget on LLM progress. [0] not technically complete depreciation, since for example 4o mini is widely believed to be a distillation of 4o, so 4o's investment still carries over into 4o mini

thierrydamiba 2y ago
Agreed on everything, but calling the marvel movies slop…I think that word has gone too far.
- ThrowawayTestr 2y ago
  The marvel movies are the genesis for this use of the word slop.
  - simonw 2y ago
    Can you back that claim up with a link or similar?
- RUnconcerned 2y ago
  Not only are Marvel movies slop, they are very concentrated slop. The only way to increase the concentration of slop in a Marvel movie would be to ask ChatGPT to write the next one.
- mattnewton 2y ago
  Not all Marvel films are slop. But, as a fan who comes from a family of fans and someone who has watched almost all of them; lets be real. That particular film, really and most of them, contain copious amounts of what is absolutely slop. I don't know if the utility is worse than an LLM that is SOTA for 2 months that no one even bothers switching to however - at least the marvel slop is being used for entertainment by someone. I think the market is definitely prioritizing the LLM researcher over Disney's latest slop sequel though so whoever made that comparison can rest easy, because we'll find out.

chipdart 2y ago

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; I think you're just seeing the "make it work" stage of the combo "first make it work, then make it fast". Time to market is critical, as you can attest by the fact you framed the situation as "on par with GPT-4o and Claude Opus". You're seeing huge investments because being the first to get a working model stands to benefit greatly. You can only assess models that exist, and for that you need to train them at a huge computational cost.

romeros 2y ago
ChatGPT is like Google now. It is the default. Even if Claude becomes as good as ChatGPT or even slightly better it won't make me switch. It has to be like a lot better. Way better. It feels like ChatGPT won the time to market war already.
- Tostino 2y ago
  Eh, with the degradation of coding performance in ChatGPT I made the switch. Seems much better to work with on problems, and I have to do way less hand holding to get good results. I'll switch again soon as something better is out.
- brandall10 2y ago
  But plenty people switched to Claude, esp. with Sonnet 3.5. Many of them in this very thread. You may be right with the average person on the street, but I wonder how many have lost interest in LLM usage and cancelled their GPT plus sub.
- asah 2y ago
  -1: I know many people who are switching to Claude. And Google makes it near-zero friction to adopt Gemini with Gsuite. And more still are using the top-N of them. This is similar to the early days of the search engine wars, the browser wars, and other categories where a user can easily adopt, switch between and use multiple. It's not like the cellphone OS/hardware war, PC war and database war where (most) users can only adopt one platform at a time and/or there's a heavy platform investment.

Workaccount2 2y ago

I think GPT5 will be the signal of whether or not we have hit a plateau. The space is still rapidly developing, and while large model gains are getting harder to pick apart, there have been enormous gains in the capabilities of light weight models.

zainhoda 2y ago
I’m waiting for the same signal. There are essentially 2 vastly different states of the world depending on whether GPT-5 is an incremental change vs a step change compared to GPT-4.
chipdart 2y ago
> I think GPT5 will be the signal of whether or not we have hit a plateau. I think GPT5 will tell if OpenAI hit a plateau. Sam Altman has been quoted as claiming "GPT-3 had the intelligence of a toddler, GPT-4 was more similar to a smart high-schooler, and that the next generation will look to have PhD-level intelligence (in certain tasks)" Notice the high degree of upselling based on vague claims of performance, and the fact that the jump from highschooler to PhD can very well be far less impressive than the jump from toddler to high schooler. In addition, notice the use of weasel words to frame expectations regarding "the next generation" to limit these gains to corner cases. There's some degree of salesmanship in the way these models are presented, but even between the hyperboles you don't see claims of transformative changes.
- rvnx 2y ago
  PhD level-of-task-execution sounds like the LLM will debate whether the task is ethical instead of actually doing it
  - airspresso 2y ago
    lol! Producing academic papers for future training runs then.
  - throwadobe 2y ago
    I wish I could frame this comment

speed_spread 2y ago

Benchmarks scores aren't good because they apply to previous generations of LLMs. That 2.23% uptick can actually represent a world of difference in subjective tests and definitely be worth the investment. Progress is not slowing down but it gets harder to quantify.

satvikpendem 2y ago

This is already what the chinchilla paper surmised, it's no wonder that their prediction now comes to fruition. It is like an accelerated version of Moore's Law, because software development itself is more accelerated than hardware development.

genrilz 2y ago

For this model, it seems like the point is that it uses way less parameters than at least the large Llama model while having near identical performance. Given how large these models are getting, this is an important thing to do before making performance better again.

deleted 2y ago

[deleted]

skybrian 2y ago

I think it’s impressive that they’re doing it on a single (large) node. Costs matter. Efficiency improvements like this will probably increase capabilities eventually. I’m also optimistic about building better (rather than bigger) datasets to train on.

BoredPositron 2y ago

We always needed a tock to see real advancement, like with the last model generation. The tick we had with the h100 was enough to bring these models to market but that's it.

lossolo 2y ago

For some time, we have been at a plateau because everyone has caught up, which essentially means that everyone now has good training datasets and uses similar tweaks to the architecture. It seems that, besides new modalities, transformers might be a dead end as an architecture. Better scores on benchmarks result from better training data and fine-tuning. The so-called 'agents' and 'function calling' also boil down to training data and fine-tuning.

lolinder 2y ago

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs Yes. This is exactly why I'm skeptical of AI doomerism/saviorism. Too many people have been looking at the pace of LLM development over the last two (2) years, modeled it as an exponential growth function, and come to the conclusion that AGI is inevitable in the next ${1-5} years and we're headed for ${(dys|u)topia}. But all that assumes that we can extrapolate a pattern of long-term exponential growth from less than two years of data. It's simply not possible to project in that way, and we're already seeing that OpenAI has pivoted from improving on GPT-4's benchmarks to reducing cost, while competitors (including free ones) catch up. All the evidence suggests we've been slowing the rate of growth in capabilities of SOTA LLMs for at least the past year, which means predictions based on exponential growth all need to be reevaluated.

cjalmeida 2y ago
Indeed.All exponential growth curves are sigmoids in disguise.
- nicman23 2y ago
  except when it isn't and we ded :P
  - kridsdale3 2y ago
    I don't think Special Relativity would allow that.
- ToValueFunfetti 2y ago
  This is something that is definitionally true in a finite universe, but doesn't carry a lot of useful predictive value in practice unless you can identify when the flattening will occur. If you have a machine that converts mass into energy and then uses that energy to increase the rate at which it operates, you could rightfully say that it will level off well before consuming all of the mass in the universe. You just can't say that next week after it has consumed all of the mass of Earth.

niemandhier 2y ago

The next iteration depends on NVIDIA & co, what we need is sparse libs. Most of the weights in llms are 0, once we deal with those more efficiently we will get to the next iteration.

lawlessone 2y ago
> Most of the weights in llms are 0, that's interesting. Do you have a rough percentage of this? Does this mean these connections have no influence at all on output?
- machiaweliczny 2y ago
  My uneducated guess is that with many layers you can implement something akin to graph in brain by nulling lots of previous later outputs. I actually suspect that current models aren’t optimal with layers all of the same size but i know shit
  - kridsdale3 2y ago
    This is quite intuitive. We know that a biological neural net is a graph data structure. And ML systems on GPUs are more like layers of bitmaps in Photoshop (it's a graphics processor). So if most of the layers are akin to transparent pixels, in order to build a graph by stacking, that's hyper memory inefficient.

m3kw9 2y ago

There is different directions AI have lots to improve: multi modal which branch into robotics, single modal like image, video, and sound generation and understanding. Also would check back when openAI releases 5

swalsh 2y ago

And with the increasing parameter size, the main winner will be Nvidia. Frankly I just don't understand the economics of training a foundation model. I'd rather own an airline. At least I can get a few years out of the capital investment of a plane.

machiaweliczny 2y ago
But billionaires already have that, they want a chance of getting their own god.

mlsu 2y ago

What else can be done? If you are sitting on 1 billions $ of GPU capex, what's $50 million in energy/training cost for another incremental run that may beat the leaderboard? Over the last few years the market has placed its bets that this stuff will make gobs of money somehow. We're all not sure how. They're probably thinking -- it's likely that whoever has a few % is going to sweep and take most of this hypothetical value. What's another few million, especially if you already have the GPUs? I think you're right -- we are towards the right end of the sigmoid. And with no "killer app" in sight. It is great for all of us that they have created all this value, because I don't think anyone will be able to capture it. They certainly haven't yet.

sebzim4500 2y ago

I don't think we can conclude that until someone trains a model that is significantly bigger than GPT-4.