3 ms·

AI has always had the concept of a “winter” or bust. [1] As in computer science in general we have had people shift concentration on what can make the most and

AI has always had the concept of a “winter” or bust. [1] As in computer science in general we have had people shift concentration on what can make the most and that can occur during a PhD. The thing about the AI winter this time is that ML and Deep learning and LLMs advanced so quickly that a large bust in AI didn’t really occur. Also, I have heard about people getting the AI jobs and then complaining it’s all about cleaning data and nothing about applying new AI techniques to solve a big challenging problem. For CS in general and IT I can sort of see this as a business cycle and LLMs doing some replacement of jobs or it could just be a business cycle. [1] https://en.wikipedia.org/wiki/AI_winter?wprov=sfti1# https://en.wikipedia.org/wiki/AI_winter?wprov=sfti1#

janalsncm 2y ago
I think you two are talking about different things. The AI winter as I understand it refers to slowing technological advancement in “expert systems” for example. But technological development is moving faster than ever. This is different. It’s a weird combination of huge amounts of capital chasing a very specific idea of next token prediction, and a slowdown in SWE hiring that may be related. It’s the first time “AI” as an industry has eaten itself.
- est31 2y ago
  From what I read, AI "winters" have not just had less progress but also have had less research activity in general: during them, AI was seen as a dead end that didn't provide value, which translates to less funding, which translates to less people doing research on it (or introducing AI to the research they wanted to do already).
- throwup238 2y ago
  The AI winter refers to the collapse in government and industry funding. It came as a result of a lot of hype like expert systems not panning out but it was about the money that dried up. This time “it’s different” but VCs are just as fickle as grant reviewers and executives - that funding could dry up at any moment if one of the big AI companies has a down round or goes bankrupt altogether. I don’t think a second AI winter is likely because of how useful LLMs have proven this time around, but we’re probably going to have a correction sooner or later. I don’t think the hiring slowdown and AI are related. Some companies are using rhetoric about AI to save face but the collapse in the job market was due to an end to ZIRP.
  - timy2shoes 2y ago
    The down rounds have started for those companies that either haven’t found product market fit, or aren’t in contention on leaderboards. Especially those companies that are a OpenAI wrapper and can be easily replaced by a new, more broadly applicable, foundation model or a new modality model. https://www.businessinsider.com/ai-down-rounds-rise-valuations-investors-2024-9 https://www.businessinsider.com/ai-down-rounds-rise-valuatio...
godelski 2y ago
> Deep learning and LLMs advanced so quickly that a large bust in AI didn’t really occur. I think the problem is we railroad too much. It’s LLMs/large models or bust. It can be hard to even publish if you aren’t using one (e.g. build on a pertained model). The problem is putting all our eggs in one basket when we should diversify. The big tech companies hired a lot of people to freely research but research just narrowed. We ignored the limitations that many discussed for a long time and we could have solved them by now if we just spent a small portion of the time and money we have on railroad topics. What I’ve seen is research become very product focused. Fine for industry research but we have to also have the more academic research. Academic is supposed to do the low TRL (1-4) while industry does higher (like 4-6). But when it’s all mid or high you got nothing in the pipeline. Even if the railroad will get us there there’s always other ways and ways to increase efficiency. This is a good time to have this conversation as we haven’t finished CVPR reviews. So if you’re a reviewer, an AC, or meta, remember this when evaluating. Evaluate beyond benchmarks and remember the context of the compute power people have. If you require more than an A100 node you only allow industry. Most universities don’t even have a full node without industry support. https://www.nasa.gov/directorates/somd/space-communications-navigation-program/technology-readiness-levels/ https://www.nasa.gov/directorates/somd/space-communications-...
- bossyTeacher 2y ago
  Problem is that people controlling the budgets have tunnel vision and little patience. They see a direct line from research to product, they see the line hitting revenue generating milestones. Nobody wants to hear that research is uncertain, that dead-ends happen, that having everyone researching a single topic is not wise, that you can't predict what findings you will get.
- selimthegrim 2y ago
  Somebody from Nvidia donated some money to our university to buy a few H100s but it’s tied up in procurement hell between the CS/DS department and IT for almost a year now
  - godelski 2y ago
PaulHoule 2y ago
If you want to to "solve a big challenging problem" somebody has to do the data cleaning. As Edison put it: genius is 1% inspiration and 99% perspiration.

janalsncm 2y ago

I think you two are talking about different things. The AI winter as I understand it refers to slowing technological advancement in “expert systems” for example. But technological development is moving faster than ever. This is different. It’s a weird combination of huge amounts of capital chasing a very specific idea of next token prediction, and a slowdown in SWE hiring that may be related. It’s the first time “AI” as an industry has eaten itself.

est31 2y ago
From what I read, AI "winters" have not just had less progress but also have had less research activity in general: during them, AI was seen as a dead end that didn't provide value, which translates to less funding, which translates to less people doing research on it (or introducing AI to the research they wanted to do already).
throwup238 2y ago
The AI winter refers to the collapse in government and industry funding. It came as a result of a lot of hype like expert systems not panning out but it was about the money that dried up. This time “it’s different” but VCs are just as fickle as grant reviewers and executives - that funding could dry up at any moment if one of the big AI companies has a down round or goes bankrupt altogether. I don’t think a second AI winter is likely because of how useful LLMs have proven this time around, but we’re probably going to have a correction sooner or later. I don’t think the hiring slowdown and AI are related. Some companies are using rhetoric about AI to save face but the collapse in the job market was due to an end to ZIRP.
- timy2shoes 2y ago
  The down rounds have started for those companies that either haven’t found product market fit, or aren’t in contention on leaderboards. Especially those companies that are a OpenAI wrapper and can be easily replaced by a new, more broadly applicable, foundation model or a new modality model. https://www.businessinsider.com/ai-down-rounds-rise-valuations-investors-2024-9 https://www.businessinsider.com/ai-down-rounds-rise-valuatio...

godelski 2y ago

> Deep learning and LLMs advanced so quickly that a large bust in AI didn’t really occur. I think the problem is we railroad too much. It’s LLMs/large models or bust. It can be hard to even publish if you aren’t using one (e.g. build on a pertained model). The problem is putting all our eggs in one basket when we should diversify. The big tech companies hired a lot of people to freely research but research just narrowed. We ignored the limitations that many discussed for a long time and we could have solved them by now if we just spent a small portion of the time and money we have on railroad topics. What I’ve seen is research become very product focused. Fine for industry research but we have to also have the more academic research. Academic is supposed to do the low TRL (1-4) while industry does higher (like 4-6). But when it’s all mid or high you got nothing in the pipeline. Even if the railroad will get us there there’s always other ways and ways to increase efficiency. This is a good time to have this conversation as we haven’t finished CVPR reviews. So if you’re a reviewer, an AC, or meta, remember this when evaluating. Evaluate beyond benchmarks and remember the context of the compute power people have. If you require more than an A100 node you only allow industry. Most universities don’t even have a full node without industry support. https://www.nasa.gov/directorates/somd/space-communications-navigation-program/technology-readiness-levels/ https://www.nasa.gov/directorates/somd/space-communications-...

bossyTeacher 2y ago
Problem is that people controlling the budgets have tunnel vision and little patience. They see a direct line from research to product, they see the line hitting revenue generating milestones. Nobody wants to hear that research is uncertain, that dead-ends happen, that having everyone researching a single topic is not wise, that you can't predict what findings you will get.
selimthegrim 2y ago
Somebody from Nvidia donated some money to our university to buy a few H100s but it’s tied up in procurement hell between the CS/DS department and IT for almost a year now
- godelski 2y ago

PaulHoule 2y ago

If you want to to "solve a big challenging problem" somebody has to do the data cleaning. As Edison put it: genius is 1% inspiration and 99% perspiration.