6 ms·

The "Foundation models" term has sparked some fiery debate at that workshop and at the core the issue isn't about general or strong AI at all. The Foundation M

The "Foundation models" term has sparked some fiery debate at that workshop and at the core the issue isn't about general or strong AI at all. The Foundation Models paper is a strategic attempt of the Stanford AI researchers to coopt the field and define a research program for large-scale pre-trained models. Propagating a new term that is slightly hyperbolic is one way to differentiate your work from other's. But it's only marketing. Watch how all Stanford-adjacent research will now cite the Foundation Models paper instead of the original BERT paper, other surveys of pre-training, or field-specific papers. There are over a 100 authors on that manuscript, it's a cheap and easy way to increase their researcher impact. Other researchers naturally aren't too happy with this cheap marketing approach.

anfilt 5y ago
So your saying basically they basically did the equivalent of this colloquialism: "If you can't win, change the rules" Just to increase citation count of their own papers. That does sound pretty dodgy.
- gillesjacobs 5y ago
  Not primarily to increase citation count only (though it certainly will help them). The Foundation Models paper reads more like a grant proposal instead of a survey paper because it basically spells out the author's research program. What they are doing is coopting field of methodology under a new name and starting a new lab with that name. This has expectedly left a foul taste in the mouth of many colleagues, hence the controversy.
lmeyerov 5y ago
Much less than citations, it's a basic empirical observation of SOTA for many NLP+vision tasks that, crucially, supports asking for a _lot_ of funding: $$$$ for hw, and skim $$$ off for a big lab of people for a decent amount of time. Less cynically... they have to do it for going after low-hanging fruit around the current local maxima. I'd expect similar writing happening in grant proposals at other big AI labs too.
- gillesjacobs 5y ago
  Yes, exactly, the Foundation Models paper is basically a grant proposal of research program. Still many object to coining a new (hyperbolic) term for an established field for own gain. Personally, I will stick to the established 'larte-scale pre-trained models' instead of 'foundation models' in my publications.
trextrex 5y ago
This. They want to create hype around large scale model so that they can ask for lots of money to fund this line of research. Except that I don't think it's targeted at the general public. Rather it's targeted at grant agencies and University administration who are usually extremely myopic and conservative about funding projects that require a large upfront investment (unlike companies like openai). In fact, I'm sure tons of people in academia are going to be pointing to this paper from now on when writing grants and asking for funding for large scale machine learning projects. Especially because of the credibility lent to it by Stanford and the people who put their names on the paper. So it's more like a research manifesto for the field.
- kevinventullo 5y ago
  Taking the politics of grant funding as an immutable prior, wouldn’t this then be a good thing for researchers?
dr_dshiv 5y ago
Speaking of "cheap marketing," did you notice how researchers have become willing to continually mislead the public by calling everything they do "Artificial Intelligence"? When I was in grad school less than 10 years ago, there was an explicit effort to be specific. Call it machine learning, deep learning, adaptive, etc. The field can be called AI and we can publish in AI journals, but you don't make the mistake of thinking that there is such a "thing" as AI. All that changed after the dump trucks full of money arrived.
- gillesjacobs 5y ago
  Artificial Intelligence is simply a broad term meaning 'the simulation of human intelligence processes by machines'. The term accurately applies to any subtask no matter how small that involves the automation of human intelligence, even if it is just recognizing dogs vs. cat images. Colloquially there is a false equivalence between A.I. and Artificial General Intelligence (AGI), 'full' or 'strong' AI. AGI specifically denotes a system that can learn any intellectual task a human could learn. And such integrated intelligence is purely hypothetical as of now. I am active in NLP research and I have not encountered an instance of any academic claiming that their work is AGI. And terms and descriptions are usually domain- and task-specific, so not much has changed since your grad school time. Only those that work on grounding and multi-modality make the claim that their work is a step towards AGI. Which is true, as these researchers usually only claim that it is an exploration, an increment towards AGI. What you are seeing is commercial entities marketing their solutions as AGI. Mass media has given more exposure to that idea because it sounds scifi. Also often journalists let their imagination run a little when describing papers. I agree that is misleading but academics still have the same rigor.
  - dr_dshiv 5y ago
    I like the definition you provide: "the simulation of human intelligence processes by machines" It would, of course, allow AI to exist that predates the invention of computers. Such as autopilot (1914) or the steam governor (1788), etc.
    - 5y ago
PeterisP 5y ago
Arguably as soon as you go beyond English, this term starts making more sense. For many languages - especially smaller ones - having a good universal model pretrained on large quantities of good data is literally quite "foundational"; it's very useful as a foundation to build other things on top of it, and it's sufficiently resource-intensive (and also getting appropriate data is labor-intensive and sometimes legally tricky) to make sense to make such models rarely, not forcing everyone to recreate one from scratch. Having a GPT-3 clone for a new language isn't really novel, but it's foundational work that helps a lot of other research.
mollerhoj 5y ago
Ironically, google kina did the same thing with BERT.. The original insight came from Jeremy Howards ulmfit. The first transfer learning model for nlp. Tranformers are great an all, but everything after ulmfit has pretty much just been the result of increased model size

anfilt 5y ago

So your saying basically they basically did the equivalent of this colloquialism: "If you can't win, change the rules" Just to increase citation count of their own papers. That does sound pretty dodgy.

gillesjacobs 5y ago
Not primarily to increase citation count only (though it certainly will help them). The Foundation Models paper reads more like a grant proposal instead of a survey paper because it basically spells out the author's research program. What they are doing is coopting field of methodology under a new name and starting a new lab with that name. This has expectedly left a foul taste in the mouth of many colleagues, hence the controversy.

lmeyerov 5y ago

Much less than citations, it's a basic empirical observation of SOTA for many NLP+vision tasks that, crucially, supports asking for a _lot_ of funding: $$$$ for hw, and skim $$$ off for a big lab of people for a decent amount of time. Less cynically... they have to do it for going after low-hanging fruit around the current local maxima. I'd expect similar writing happening in grant proposals at other big AI labs too.

gillesjacobs 5y ago
Yes, exactly, the Foundation Models paper is basically a grant proposal of research program. Still many object to coining a new (hyperbolic) term for an established field for own gain. Personally, I will stick to the established 'larte-scale pre-trained models' instead of 'foundation models' in my publications.

trextrex 5y ago

This. They want to create hype around large scale model so that they can ask for lots of money to fund this line of research. Except that I don't think it's targeted at the general public. Rather it's targeted at grant agencies and University administration who are usually extremely myopic and conservative about funding projects that require a large upfront investment (unlike companies like openai). In fact, I'm sure tons of people in academia are going to be pointing to this paper from now on when writing grants and asking for funding for large scale machine learning projects. Especially because of the credibility lent to it by Stanford and the people who put their names on the paper. So it's more like a research manifesto for the field.

kevinventullo 5y ago
Taking the politics of grant funding as an immutable prior, wouldn’t this then be a good thing for researchers?

dr_dshiv 5y ago

Speaking of "cheap marketing," did you notice how researchers have become willing to continually mislead the public by calling everything they do "Artificial Intelligence"? When I was in grad school less than 10 years ago, there was an explicit effort to be specific. Call it machine learning, deep learning, adaptive, etc. The field can be called AI and we can publish in AI journals, but you don't make the mistake of thinking that there is such a "thing" as AI. All that changed after the dump trucks full of money arrived.

gillesjacobs 5y ago
Artificial Intelligence is simply a broad term meaning 'the simulation of human intelligence processes by machines'. The term accurately applies to any subtask no matter how small that involves the automation of human intelligence, even if it is just recognizing dogs vs. cat images. Colloquially there is a false equivalence between A.I. and Artificial General Intelligence (AGI), 'full' or 'strong' AI. AGI specifically denotes a system that can learn any intellectual task a human could learn. And such integrated intelligence is purely hypothetical as of now. I am active in NLP research and I have not encountered an instance of any academic claiming that their work is AGI. And terms and descriptions are usually domain- and task-specific, so not much has changed since your grad school time. Only those that work on grounding and multi-modality make the claim that their work is a step towards AGI. Which is true, as these researchers usually only claim that it is an exploration, an increment towards AGI. What you are seeing is commercial entities marketing their solutions as AGI. Mass media has given more exposure to that idea because it sounds scifi. Also often journalists let their imagination run a little when describing papers. I agree that is misleading but academics still have the same rigor.
- dr_dshiv 5y ago
  I like the definition you provide: "the simulation of human intelligence processes by machines" It would, of course, allow AI to exist that predates the invention of computers. Such as autopilot (1914) or the steam governor (1788), etc.
  - 5y ago

PeterisP 5y ago

Arguably as soon as you go beyond English, this term starts making more sense. For many languages - especially smaller ones - having a good universal model pretrained on large quantities of good data is literally quite "foundational"; it's very useful as a foundation to build other things on top of it, and it's sufficiently resource-intensive (and also getting appropriate data is labor-intensive and sometimes legally tricky) to make sense to make such models rarely, not forcing everyone to recreate one from scratch. Having a GPT-3 clone for a new language isn't really novel, but it's foundational work that helps a lot of other research.

mollerhoj 5y ago

Ironically, google kina did the same thing with BERT.. The original insight came from Jeremy Howards ulmfit. The first transfer learning model for nlp. Tranformers are great an all, but everything after ulmfit has pretty much just been the result of increased model size