5 ms·

Quite a few people saw this coming. It is still early to tell if we reached AI winter again or not, but at least we can see that news are slowing down.

cs702 2y ago
If there are no breakthroughs and funding dries up in the near future, it will feel to many like going off a precipice at high speed. Only the poor souls who survive the fall, at the very bottom of the precipice, will get to experience the AI winter.
pas 2y ago
RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?
- cs702 2y ago
  I think the grandparent comment is about AI research, driven by the quest for AGI. Incremental improvement of proven approaches, driven by profit motive, will surely continue regardless of whether there is an AI winter or not.
  - pas 2y ago
    Indeed, but there's probably a very real funding and attention (heh) issue, if there's ton of both then there will be progress. But. Usually the bigger the hype the more progress is expected so the faster the fund-tention will dissipate.
- nkozyra 2y ago
  RAG takes the current limit of LLM and focuses it on specific problems using custom data. It's not exactly magic, it just finds a way to produce something tangible and usable from an otherwise broadly focused model. It's the rage because it's a way to practically _do work_ from LLMs that generally provide wow from conversationally accurate, often factually accurate responses.
- PheonixPharts 2y ago
  There's tons of optimization work left, but by it's nature optimization tends to push the limits of what we currently have, and rarely allows for substantial improvements. There are some major limitations to LLMs that aren't going to be "optimized" away. At the end of the day LLMs are Monte Carlo samplers over a latent, compressed representation existing human text data. Many of the tasks people hope LLMs will achieve require major leaps in our current understanding of language modeling. One great example limitation (which shocks me sometimes when I think about it): generating output is still ultimately stuck in looking at the probability of the next token rather than the much more useful, probability of the generated statement. There are techniques to improve this, but we're missing a major piece of generating highly probable statements with no hint about how to really get there. Consider how you might write SQL. You conceive of the high level query first and start sketching out the pieces. An LLM can only look at each token and can't, statistically speaking, think in terms of the entire query. Personally I think LLMs are very underutilized/exploited for what they are good at, and there is way too much focus on what they can't do. Hopefully we'll dodge an AI winter by using LLMs to solve the wide range of classical NLP problems that make many tasks that just a few years ago nearly impossible, rather simple today. Unfortunately the irrational hype around these models makes me skeptical of that scenario.
  - mjburgess 2y ago
    The issue with predicting the next statement is the combinatorial infinity which is improbable to model with historical frequencies, ie., P(mat|the cat sat on the) is a distribution over, say, 100k words. Whereas, P(the cat sat|on the mat) is a distribution over 100k^3 words. Part of the illusion of an LLM is that we produce create text in such a highly regular way that a mere distribution over 100k iterated, say 500 times, gives you a page of text as if modelling 100k^500.
  - pas 2y ago
    > An LLM can only look at each token and can't, statistically speaking, think in terms of the entire query. There's always some space in high-dimensions to put that extra context somewhere. So yes, only one token is predicted, but there's a lot of information squeezed into the "condition" (as in conditional probability, as mjburgess' comment shows) > [hype instead of NLP] yes, exactly. we'll see what remains after the bubble bursts.
loudmax 2y ago
I think that depends what your expectations are, and what you mean by another AI winter. We're just scratching the surface of what's possible with the current state of the art. Even if there are no major advances or breakthroughs in the near future, LLMs and associated technologies are already useful in many use cases. Or close enough to useful where engineering rather than science will be sufficient to overcome many (though not all) of the shortcomings of current AI models. Never mind grandiose claims about AGI, there's enough utility to be gotten out of the limited LLMs that we have today to keep engineers and entrepreneurs busy for years to come.
- swatcoder 2y ago
  You're right, and I suspect the GP would agree with you about there being real engineering applications for LLMs, diffusion models, etc But I think the term "AI Winter" usually refers to the underlying research programme and the economics around it. Soaking up many billions of dollars of industry money and public grants on the pitch that AGI might be just around the corner, and then being unable to deliver on that pitch can induce a hangover effect that makes it much harder to raise money for anything that smells even a little bit like the failed pitch. Investors and administrators feel burnt and turn very skeptical for a very long time. Meanwhile, the actual productive applications which shook out of the initial boom just get renamed to something else so that they don't carry that smell. We'll see how it goes here, but that's the familiar road and where the terminology comes from.
  - ijustlovemath 2y ago
    The minor irony of this is that current efforts at AGI are focused on the data scaling laws, which we have showed no signs of slowing on, but if funding dries up these crazy expensive training runs wont be allowed anymore.
- nicklecompte 2y ago
  The whole point of past AI winters is that genuinely useful technologies were preposterously overhyped as representing "human cognition," despite overwhelming evidence to the contrary. The bubble bursts - and the money evaporates - because expectations come crashing down, not because the usefulness was a mirage. Lisp was hardly the key to human symbolic thought like some hoped it would be, but it helped improve programming languages across the board. Perceptrons really are quite limited and there's absolutely no way you could emulate a human brain with them alone, but they're still essential for more advanced ML. But you would think after 70 years AI practitioners would learn some humility! It is very obvious that GPT-4 is dumber than a honeybee, let alone a cat, let alone a crow. But for over a year I've heard dozens of tech folks insist it's smarter than most people.
squigz 2y ago
> reached AI winter again Again?
- pocketarc 2y ago
  It's a term for when AI hype dries up, has happened multiple times since the 60s[0]. [0]: https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
- swatcoder 2y ago
  This was not the technology industry's first encounter with AI hype. The term was coined 40 years ago, and has been suggested as a description for almost a dozen periods in the field's history: https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
msp26 2y ago
I hope this means that people actually start curating their training sets. The quality control is horrible on all of the datasets I've looked through. Especially image captions. Yes sheer quantity has a quality of its own but that won't produce optimal results.
- RodgerTheGreat 2y ago
  From a very cynical marketing/VC-pitch perspective, un-curated datasets full of random crap have the benefit of sometimes producing totally surprising, unplanned "features", which helps sell the idea that one-size-fits-all black-box machine learning can solve any problem.
godelski 2y ago
I think we're in the beginning of a warm winter. The funding is there but we're letting hype drive everything and not calling out the con artists. The problem is we've had recent great success but still don't know how to get to AGI. But because we're afraid of winter we're not willing to try new things. We want to only compare to sota and think it's fair to compare a new method with a handful of papers to the status quo. That's not how the S-curves of technology work. Sure, maybe things don't scale but that doesn't mean they don't have merit or can't scale if someone finds some modification. The problem is we treat research like products, not academic work. You need to produce everything from hard core theory to robust products to have an effective chain. But we seem hyper focused on the middle area. And for some reason people think products can be placing a nice interface around research code. There's still a lot of work you need to do and all those models can be optimized. They absolutely do not have optimal hyper parameters or even parameters.
darepublic 2y ago
Dunno exactly what you mean by ai winter, but the whole gen AI thing has kind of got all the attention and even if that route slows down there are a ton of other branches fruitful for development.
- stephc_int13 2y ago
  The issue, I think, is that if results don't follow expectations, given the very high costs, investors might get cold feet. Sure we already have some interesting applications, but they are not exactly printing money.
CuriouslyC 2y ago
Definitely not an AI winter, since the tools are so useful. People who think AGI is right around the corner might be disappointed though, since we can't really even accurately define AGI.
trgn 2y ago
Commercial adoption has only started to ramp up. We are going to see a floodwave of LLM-powered bots/UX-wizards. Even if academic progress stalls, we're going to be inundated for at least another few years.
refulgentis 2y ago
Uh, what?
ertgbnm 2y ago
I'm not even going to discuss whether or not things are actually slowing down since I disagree with that. But I do feel confident that an AI winter is not on the horizon solely due to the overhead of implementation that we currently have. Just with currently existing AI, it would take years for the economy to fully leverage the abilities that are available. I'm confident we have transformative AI. So it won't feel like a winter for several years as we actually succeed in optimizing, implementing, and productizing current technology in other industries.

cs702 2y ago

If there are no breakthroughs and funding dries up in the near future, it will feel to many like going off a precipice at high speed. Only the poor souls who survive the fall, at the very bottom of the precipice, will get to experience the AI winter.

pas 2y ago

RAG seems to be all the rage. Not to mention the quest for the cooking up the correct cocktail of smaller MoE/ensemble models, and ... there's decades' worth of optimization work ahead (a few years of it seems to be already VC and edu grants funded), no?

cs702 2y ago
I think the grandparent comment is about AI research, driven by the quest for AGI. Incremental improvement of proven approaches, driven by profit motive, will surely continue regardless of whether there is an AI winter or not.
- pas 2y ago
  Indeed, but there's probably a very real funding and attention (heh) issue, if there's ton of both then there will be progress. But. Usually the bigger the hype the more progress is expected so the faster the fund-tention will dissipate.
nkozyra 2y ago
RAG takes the current limit of LLM and focuses it on specific problems using custom data. It's not exactly magic, it just finds a way to produce something tangible and usable from an otherwise broadly focused model. It's the rage because it's a way to practically _do work_ from LLMs that generally provide wow from conversationally accurate, often factually accurate responses.
PheonixPharts 2y ago
There's tons of optimization work left, but by it's nature optimization tends to push the limits of what we currently have, and rarely allows for substantial improvements. There are some major limitations to LLMs that aren't going to be "optimized" away. At the end of the day LLMs are Monte Carlo samplers over a latent, compressed representation existing human text data. Many of the tasks people hope LLMs will achieve require major leaps in our current understanding of language modeling. One great example limitation (which shocks me sometimes when I think about it): generating output is still ultimately stuck in looking at the probability of the next token rather than the much more useful, probability of the generated statement. There are techniques to improve this, but we're missing a major piece of generating highly probable statements with no hint about how to really get there. Consider how you might write SQL. You conceive of the high level query first and start sketching out the pieces. An LLM can only look at each token and can't, statistically speaking, think in terms of the entire query. Personally I think LLMs are very underutilized/exploited for what they are good at, and there is way too much focus on what they can't do. Hopefully we'll dodge an AI winter by using LLMs to solve the wide range of classical NLP problems that make many tasks that just a few years ago nearly impossible, rather simple today. Unfortunately the irrational hype around these models makes me skeptical of that scenario.
- mjburgess 2y ago
  The issue with predicting the next statement is the combinatorial infinity which is improbable to model with historical frequencies, ie., P(mat|the cat sat on the) is a distribution over, say, 100k words. Whereas, P(the cat sat|on the mat) is a distribution over 100k^3 words. Part of the illusion of an LLM is that we produce create text in such a highly regular way that a mere distribution over 100k iterated, say 500 times, gives you a page of text as if modelling 100k^500.
- pas 2y ago
  > An LLM can only look at each token and can't, statistically speaking, think in terms of the entire query. There's always some space in high-dimensions to put that extra context somewhere. So yes, only one token is predicted, but there's a lot of information squeezed into the "condition" (as in conditional probability, as mjburgess' comment shows) > [hype instead of NLP] yes, exactly. we'll see what remains after the bubble bursts.

loudmax 2y ago

I think that depends what your expectations are, and what you mean by another AI winter. We're just scratching the surface of what's possible with the current state of the art. Even if there are no major advances or breakthroughs in the near future, LLMs and associated technologies are already useful in many use cases. Or close enough to useful where engineering rather than science will be sufficient to overcome many (though not all) of the shortcomings of current AI models. Never mind grandiose claims about AGI, there's enough utility to be gotten out of the limited LLMs that we have today to keep engineers and entrepreneurs busy for years to come.

swatcoder 2y ago
You're right, and I suspect the GP would agree with you about there being real engineering applications for LLMs, diffusion models, etc But I think the term "AI Winter" usually refers to the underlying research programme and the economics around it. Soaking up many billions of dollars of industry money and public grants on the pitch that AGI might be just around the corner, and then being unable to deliver on that pitch can induce a hangover effect that makes it much harder to raise money for anything that smells even a little bit like the failed pitch. Investors and administrators feel burnt and turn very skeptical for a very long time. Meanwhile, the actual productive applications which shook out of the initial boom just get renamed to something else so that they don't carry that smell. We'll see how it goes here, but that's the familiar road and where the terminology comes from.
- ijustlovemath 2y ago
  The minor irony of this is that current efforts at AGI are focused on the data scaling laws, which we have showed no signs of slowing on, but if funding dries up these crazy expensive training runs wont be allowed anymore.
nicklecompte 2y ago
The whole point of past AI winters is that genuinely useful technologies were preposterously overhyped as representing "human cognition," despite overwhelming evidence to the contrary. The bubble bursts - and the money evaporates - because expectations come crashing down, not because the usefulness was a mirage. Lisp was hardly the key to human symbolic thought like some hoped it would be, but it helped improve programming languages across the board. Perceptrons really are quite limited and there's absolutely no way you could emulate a human brain with them alone, but they're still essential for more advanced ML. But you would think after 70 years AI practitioners would learn some humility! It is very obvious that GPT-4 is dumber than a honeybee, let alone a cat, let alone a crow. But for over a year I've heard dozens of tech folks insist it's smarter than most people.

squigz 2y ago

> reached AI winter again Again?

pocketarc 2y ago
It's a term for when AI hype dries up, has happened multiple times since the 60s[0]. [0]: https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
swatcoder 2y ago
This was not the technology industry's first encounter with AI hype. The term was coined 40 years ago, and has been suggested as a description for almost a dozen periods in the field's history: https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter

msp26 2y ago

I hope this means that people actually start curating their training sets. The quality control is horrible on all of the datasets I've looked through. Especially image captions. Yes sheer quantity has a quality of its own but that won't produce optimal results.

RodgerTheGreat 2y ago
From a very cynical marketing/VC-pitch perspective, un-curated datasets full of random crap have the benefit of sometimes producing totally surprising, unplanned "features", which helps sell the idea that one-size-fits-all black-box machine learning can solve any problem.

godelski 2y ago

I think we're in the beginning of a warm winter. The funding is there but we're letting hype drive everything and not calling out the con artists. The problem is we've had recent great success but still don't know how to get to AGI. But because we're afraid of winter we're not willing to try new things. We want to only compare to sota and think it's fair to compare a new method with a handful of papers to the status quo. That's not how the S-curves of technology work. Sure, maybe things don't scale but that doesn't mean they don't have merit or can't scale if someone finds some modification. The problem is we treat research like products, not academic work. You need to produce everything from hard core theory to robust products to have an effective chain. But we seem hyper focused on the middle area. And for some reason people think products can be placing a nice interface around research code. There's still a lot of work you need to do and all those models can be optimized. They absolutely do not have optimal hyper parameters or even parameters.

darepublic 2y ago

Dunno exactly what you mean by ai winter, but the whole gen AI thing has kind of got all the attention and even if that route slows down there are a ton of other branches fruitful for development.

stephc_int13 2y ago
The issue, I think, is that if results don't follow expectations, given the very high costs, investors might get cold feet. Sure we already have some interesting applications, but they are not exactly printing money.

CuriouslyC 2y ago

Definitely not an AI winter, since the tools are so useful. People who think AGI is right around the corner might be disappointed though, since we can't really even accurately define AGI.

trgn 2y ago

Commercial adoption has only started to ramp up. We are going to see a floodwave of LLM-powered bots/UX-wizards. Even if academic progress stalls, we're going to be inundated for at least another few years.

refulgentis 2y ago

Uh, what?

ertgbnm 2y ago

I'm not even going to discuss whether or not things are actually slowing down since I disagree with that. But I do feel confident that an AI winter is not on the horizon solely due to the overhead of implementation that we currently have. Just with currently existing AI, it would take years for the economy to fully leverage the abilities that are available. I'm confident we have transformative AI. So it won't feel like a winter for several years as we actually succeed in optimizing, implementing, and productizing current technology in other industries.