7 ms·

ChatGPT is not all you need. A SOTA Review of large Generative AI models

157 points by georgehill 4y ago

Xeoncross 4y ago
A correct anticipation of the future is a big part of all successful projects. However, that is mostly do the effort required to change course. If we reach a point where it's easy enough to regenerate everything from scratch, will it be so important to correctly plan ahead?
- matchagaucho 4y ago
  Depends on the temperature of the prompt model. Getting random results with each generation is a feature of GPT. Determinism still requires planning. (this response generated by GPT ;-) )
amelius 4y ago
> This work consists on an attempt to describe in a concise way the main models are sectors that are affected by generative AI and to provide a taxonomy of the main generative models published recently. I had trouble parsing this sentence. And why doesn't the abstract provide at least some basic explanation of the title?
- posharma 4y ago
  May be because this paper was generated by ChatGPT :-)
- csande17 4y ago
  "[X] Is All You Need" is a popular trope in AI paper titles, sort of like "[X] Considered Harmful".
  - amelius 4y ago
    Yeah I know that, but the abstract doesn't explain why I (don't) need it.
- basch 4y ago
  How it _should_ read if it were concise and clear? >This work concisely describes the main sectors affected by generative AI, along with a taxonomy of recently published models.
- tehsauce 4y ago
  I also didn't like the abstract. It feels more like what the introduction section of the paper should be. Instead here they should summarize what their key finds or results were.
  - musicale 4y ago
    Academic papers shouldn't read like mystery novels or vague, pointless ramblings. If the abstract doesn't describe or summarize any notable results or contributions, a reader might (very reasonably) assume that there aren't any. The last line of the abstract does seem to be relevant: > This work consists on (sic) an attempt to describe in a concise way the main models are (sic) sectors that are affected by generative AI and to provide a taxonomy of the main generative models published recently. I think the authors may have meant something like: "This work presents a concise overview of generative AI models and their application areas, and provides a taxonomy of recently published generative models." They could have gone on to explain that the models are categorized by input and output type.
haldujai 4y ago
Is it just me or does anyone else cringe when they read "is/is not all you need" in the title of an AI-related paper? Also, what is "SOTA" for review? There isn't exactly a benchmark to compare... In terms of comprehensiveness they don't mention PaLM + variants, which probably should be mentioned as it is currently the largest LLM with SOTA on several benchmarks (e.g. MedQA-USMLE). In terms of correctness, I admittedly skipped to the sections I'm familiar with (LLMs) but I don't understand why they are distinguishing 'text-science' from 'text-text', they're both text-text and there is no reason why you can't, for example, adapt GPT3.5 to a scientific domain domain (some people even argue this is a better approach). A lot of powerful language models in the biomedical domain were initialized from general language models and use out-of-domain tokenizers/vocabularies (e.g. BioBERT). The authors also make this statement regarding Galactica: "The main advantage of [Galactica] is the ability to train on it for multiple epochs without overfitting" This is not a unique feature of Galactica and has been done before. You're allowed to train LLMs for more than 1 epoch and in fact it can be very beneficial (see BioBERT as an example of increasing training length). People GENERALLY don't do this because the corpus used during self-supervised training is filled with garbage/noise, so the model starts to fit to that instead of what you desire. There is nothing special about Galactica's architecture that specifically allows/encourages longer training cycles but rather they curated the dataset to minimize garbage. As another example, my research involves radiology NLP and when doing domain adaptive pretraining on a highly curated dataset we have been going up to 8 epochs without overfitting.
- azinman2 4y ago
  > Is it just me or does anyone else cringe when they read "is/is not all you need" in the title of an AI-related paper? I believe it's a reference to 'attention is all you need,' which is a very famous paper now. I'm guessing you probably already know that -- did you cringe with it? It was a landmark paper so at least it's worth hyperbole in the title. Maybe I'm missing that its usage started before that. Reminds me of my ML prof who would complain about people using the word 'optimum' or 'optimized' in titles of papers... one could almost always optimize more, and with respect to what is not specified by the title.
  - 4y ago
coolspot 4y ago
No OpenScience’s Bloom (opensource, 175B params), no Google’s T5 (opensource, 11B). The AI/ML space moves so fast that this review is already outdated.
- anticensor 4y ago
  No Riffusion, no VALL-E, no Point-E either.
simonw 4y ago
The diagrams in this paper did not fill me with confidence: there doesn't appear to be any reason for the layout of the boxes and lines in them other than to fill some space.
la64710 4y ago
I think the simple chat interface is why chatGPT became popular. If other people build relatively simple interfaces to leverage other models they might be successful as well.
- winddude 4y ago
  I think it's the chat interface combined with memory. The second not many people had seen used in practise. The chat bot also generates much longer prompts, most other chat models seem to be trained for short prompts, and purely conversational.
- broast 4y ago
  As a user of both, I tend to prefer the playground for most productive uses. I think chatgpt is a very specific use-case of language models that's mostly just for fun, and Google searches- but even for that the playground gives you greater control.
  - ilaksh 4y ago
    Would you consider checking out my site aidev.codes? You can use text-davinci-003 or code-davinci-002 and it will immediately create and host pages on the site. Or you can just collect whatever files and there is a download button. I plan to add GitHub and a lot of other stuff.
    - la64710 4y ago
      I am not sure what is the framework you are using for your site but it is not very mobile friendly. There are many frameworks available that can make a responsive site without much effort and also set up SSO authentication for you (like one can use google to login). I will try it later from a laptop.Good luck!
      - ilaksh 4y ago
        It uses Bootstrap.
deleted 4y ago
[deleted]
soperj 4y ago
I really haven't found ChatGTP to be very good. Ask it the lyrics to a song and it'll come up with some of the lyrics but not all of the lyrics. Like this is what I got for "In Bloom" by Nirvana: Verse 1: Sell the kids for food Weather changes moods Spring is here again Reproductive glands Verse 2: A country battle song Multiply, exciting people Come on, join the party Come on, everybody Chorus: In bloom In bloom In bloom In bloom Verse 3: Subhumanity is fun You have one, you have none A soap impression of his wife Which he ate and donated to the National Trust Verse 4: I'm not like them But I can pretend The sun is gone But I have a light Chorus: In bloom In bloom In bloom In bloom
- simonw 4y ago
  It's not a search engine. Asking it for direct lookup of things plays to its weaknesses, not to its strengths. Similarly, don't judge it based on its ability to solve math problems. When you use it for the many tasks it IS suitable for it's really, really impressive.
  - soperj 4y ago
    There's a vast difference between confidently stating the answer and "something like this: ..." When you don't know what it's talking about, it seems very impressive, when you do, it looks very foolish.
    - simonw 4y ago
      Try "Write me a Nirvana song about angry weasels invading the French alps" if you want to see it doing something impressive.
      - dinkumthinkum 4y ago
        I don’t know, people have asked to compose a Shakespearean work and what it produces is puffery that doesn’t sound like Shakespeare.
        4y ago
known 4y ago
[dead]

157 points by georgehill 4y ago

A correct anticipation of the future is a big part of all successful projects. However, that is mostly do the effort required to change course. If we reach a point where it's easy enough to regenerate everything from scratch, will it be so important to correctly plan ahead?

matchagaucho 4y ago
Depends on the temperature of the prompt model. Getting random results with each generation is a feature of GPT. Determinism still requires planning. (this response generated by GPT ;-) )

amelius 4y ago

> This work consists on an attempt to describe in a concise way the main models are sectors that are affected by generative AI and to provide a taxonomy of the main generative models published recently. I had trouble parsing this sentence. And why doesn't the abstract provide at least some basic explanation of the title?

posharma 4y ago
May be because this paper was generated by ChatGPT :-)
csande17 4y ago
"[X] Is All You Need" is a popular trope in AI paper titles, sort of like "[X] Considered Harmful".
- amelius 4y ago
  Yeah I know that, but the abstract doesn't explain why I (don't) need it.
basch 4y ago
How it _should_ read if it were concise and clear? >This work concisely describes the main sectors affected by generative AI, along with a taxonomy of recently published models.
tehsauce 4y ago
I also didn't like the abstract. It feels more like what the introduction section of the paper should be. Instead here they should summarize what their key finds or results were.
- musicale 4y ago
  Academic papers shouldn't read like mystery novels or vague, pointless ramblings. If the abstract doesn't describe or summarize any notable results or contributions, a reader might (very reasonably) assume that there aren't any. The last line of the abstract does seem to be relevant: > This work consists on (sic) an attempt to describe in a concise way the main models are (sic) sectors that are affected by generative AI and to provide a taxonomy of the main generative models published recently. I think the authors may have meant something like: "This work presents a concise overview of generative AI models and their application areas, and provides a taxonomy of recently published generative models." They could have gone on to explain that the models are categorized by input and output type.

haldujai 4y ago

Is it just me or does anyone else cringe when they read "is/is not all you need" in the title of an AI-related paper? Also, what is "SOTA" for review? There isn't exactly a benchmark to compare... In terms of comprehensiveness they don't mention PaLM + variants, which probably should be mentioned as it is currently the largest LLM with SOTA on several benchmarks (e.g. MedQA-USMLE). In terms of correctness, I admittedly skipped to the sections I'm familiar with (LLMs) but I don't understand why they are distinguishing 'text-science' from 'text-text', they're both text-text and there is no reason why you can't, for example, adapt GPT3.5 to a scientific domain domain (some people even argue this is a better approach). A lot of powerful language models in the biomedical domain were initialized from general language models and use out-of-domain tokenizers/vocabularies (e.g. BioBERT). The authors also make this statement regarding Galactica: "The main advantage of [Galactica] is the ability to train on it for multiple epochs without overfitting" This is not a unique feature of Galactica and has been done before. You're allowed to train LLMs for more than 1 epoch and in fact it can be very beneficial (see BioBERT as an example of increasing training length). People GENERALLY don't do this because the corpus used during self-supervised training is filled with garbage/noise, so the model starts to fit to that instead of what you desire. There is nothing special about Galactica's architecture that specifically allows/encourages longer training cycles but rather they curated the dataset to minimize garbage. As another example, my research involves radiology NLP and when doing domain adaptive pretraining on a highly curated dataset we have been going up to 8 epochs without overfitting.

azinman2 4y ago
> Is it just me or does anyone else cringe when they read "is/is not all you need" in the title of an AI-related paper? I believe it's a reference to 'attention is all you need,' which is a very famous paper now. I'm guessing you probably already know that -- did you cringe with it? It was a landmark paper so at least it's worth hyperbole in the title. Maybe I'm missing that its usage started before that. Reminds me of my ML prof who would complain about people using the word 'optimum' or 'optimized' in titles of papers... one could almost always optimize more, and with respect to what is not specified by the title.
- 4y ago

coolspot 4y ago

No OpenScience’s Bloom (opensource, 175B params), no Google’s T5 (opensource, 11B). The AI/ML space moves so fast that this review is already outdated.

anticensor 4y ago
No Riffusion, no VALL-E, no Point-E either.

simonw 4y ago

The diagrams in this paper did not fill me with confidence: there doesn't appear to be any reason for the layout of the boxes and lines in them other than to fill some space.

la64710 4y ago

I think the simple chat interface is why chatGPT became popular. If other people build relatively simple interfaces to leverage other models they might be successful as well.

winddude 4y ago
I think it's the chat interface combined with memory. The second not many people had seen used in practise. The chat bot also generates much longer prompts, most other chat models seem to be trained for short prompts, and purely conversational.
broast 4y ago
As a user of both, I tend to prefer the playground for most productive uses. I think chatgpt is a very specific use-case of language models that's mostly just for fun, and Google searches- but even for that the playground gives you greater control.
- ilaksh 4y ago
  Would you consider checking out my site aidev.codes? You can use text-davinci-003 or code-davinci-002 and it will immediately create and host pages on the site. Or you can just collect whatever files and there is a download button. I plan to add GitHub and a lot of other stuff.
  - la64710 4y ago
    I am not sure what is the framework you are using for your site but it is not very mobile friendly. There are many frameworks available that can make a responsive site without much effort and also set up SSO authentication for you (like one can use google to login). I will try it later from a laptop.Good luck!
    - ilaksh 4y ago
      It uses Bootstrap.

deleted 4y ago

[deleted]

soperj 4y ago

I really haven't found ChatGTP to be very good. Ask it the lyrics to a song and it'll come up with some of the lyrics but not all of the lyrics. Like this is what I got for "In Bloom" by Nirvana: Verse 1: Sell the kids for food Weather changes moods Spring is here again Reproductive glands Verse 2: A country battle song Multiply, exciting people Come on, join the party Come on, everybody Chorus: In bloom In bloom In bloom In bloom Verse 3: Subhumanity is fun You have one, you have none A soap impression of his wife Which he ate and donated to the National Trust Verse 4: I'm not like them But I can pretend The sun is gone But I have a light Chorus: In bloom In bloom In bloom In bloom

simonw 4y ago
It's not a search engine. Asking it for direct lookup of things plays to its weaknesses, not to its strengths. Similarly, don't judge it based on its ability to solve math problems. When you use it for the many tasks it IS suitable for it's really, really impressive.
- soperj 4y ago
  There's a vast difference between confidently stating the answer and "something like this: ..." When you don't know what it's talking about, it seems very impressive, when you do, it looks very foolish.
  - simonw 4y ago
    Try "Write me a Nirvana song about angry weasels invading the French alps" if you want to see it doing something impressive.
    - dinkumthinkum 4y ago
      I don’t know, people have asked to compose a Shakespearean work and what it produces is puffery that doesn’t sound like Shakespeare.
      - 4y ago

known 4y ago

[dead]