3 ms·

> Deep learning and LLMs advanced so quickly that a large bust in AI didn’t really occur. I think the problem is we railroad too much. It’s LLMs/large models o

> Deep learning and LLMs advanced so quickly that a large bust in AI didn’t really occur. I think the problem is we railroad too much. It’s LLMs/large models or bust. It can be hard to even publish if you aren’t using one (e.g. build on a pertained model). The problem is putting all our eggs in one basket when we should diversify. The big tech companies hired a lot of people to freely research but research just narrowed. We ignored the limitations that many discussed for a long time and we could have solved them by now if we just spent a small portion of the time and money we have on railroad topics. What I’ve seen is research become very product focused. Fine for industry research but we have to also have the more academic research. Academic is supposed to do the low TRL (1-4) while industry does higher (like 4-6). But when it’s all mid or high you got nothing in the pipeline. Even if the railroad will get us there there’s always other ways and ways to increase efficiency. This is a good time to have this conversation as we haven’t finished CVPR reviews. So if you’re a reviewer, an AC, or meta, remember this when evaluating. Evaluate beyond benchmarks and remember the context of the compute power people have. If you require more than an A100 node you only allow industry. Most universities don’t even have a full node without industry support. https://www.nasa.gov/directorates/somd/space-communications-navigation-program/technology-readiness-levels/ https://www.nasa.gov/directorates/somd/space-communications-...

bossyTeacher 2y ago
Problem is that people controlling the budgets have tunnel vision and little patience. They see a direct line from research to product, they see the line hitting revenue generating milestones. Nobody wants to hear that research is uncertain, that dead-ends happen, that having everyone researching a single topic is not wise, that you can't predict what findings you will get.
selimthegrim 2y ago
Somebody from Nvidia donated some money to our university to buy a few H100s but it’s tied up in procurement hell between the CS/DS department and IT for almost a year now
- godelski 2y ago
  I had gotten a contract with an unspecified California energy lab because they wanted to continue my summer internship research and it after 2 terms it still wasn’t solidified… so we moved on. People say government is slow but it wasn’t the government in my case… All these administrators drive up the cost but I’ve failed to see the value add
  - prpl 2y ago
    east bay or peninsula?
    - godelski 2y ago
      The one with the bigger computer ;)
zitterbewegung 2y ago
Other machine learning techniques are still being researched . Deep learning / lLMs are just more popular in general due to being in vogue and the public can understand GenAI and what it does.
- godelski 2y ago
  No one says they aren’t being researched. But there sure is an extremely disproportionate amount of time and money being put into a few things. Try publishing something in like classification or detection using a pretrained huge model or without lots of compute (more than 1 node of A100s). You’ll just get asked for more datasets and why your 10M parameter model isn’t better than a 1B model even if it’s better than their 100M model. There’s always reason go criticize and so there’s a compete arms race which is unnecessary

bossyTeacher 2y ago

Problem is that people controlling the budgets have tunnel vision and little patience. They see a direct line from research to product, they see the line hitting revenue generating milestones. Nobody wants to hear that research is uncertain, that dead-ends happen, that having everyone researching a single topic is not wise, that you can't predict what findings you will get.

selimthegrim 2y ago

Somebody from Nvidia donated some money to our university to buy a few H100s but it’s tied up in procurement hell between the CS/DS department and IT for almost a year now

godelski 2y ago
I had gotten a contract with an unspecified California energy lab because they wanted to continue my summer internship research and it after 2 terms it still wasn’t solidified… so we moved on. People say government is slow but it wasn’t the government in my case… All these administrators drive up the cost but I’ve failed to see the value add
- prpl 2y ago
  east bay or peninsula?
  - godelski 2y ago
    The one with the bigger computer ;)

zitterbewegung 2y ago

Other machine learning techniques are still being researched . Deep learning / lLMs are just more popular in general due to being in vogue and the public can understand GenAI and what it does.

godelski 2y ago
No one says they aren’t being researched. But there sure is an extremely disproportionate amount of time and money being put into a few things. Try publishing something in like classification or detection using a pretrained huge model or without lots of compute (more than 1 node of A100s). You’ll just get asked for more datasets and why your 10M parameter model isn’t better than a 1B model even if it’s better than their 100M model. There’s always reason go criticize and so there’s a compete arms race which is unnecessary