5 ms·

Key point is vaguely good. Scale is still important and that manifests in the difference between gpt3.5 and gpt4 based chatgpts. It's qualitatively and quantita

by typon 4y ago

Key point is vaguely good. Scale is still important and that manifests in the difference between gpt3.5 and gpt4 based chatgpts. It's qualitatively and quantitatively so much better in pretty much every benchmark. There is no way around the bitter lesson.

feanaro 4y ago
Isn't it the case that we literally have no clue how GPT4 and GPT3.5 are different in terms of training, given OpenAI doesn't want to disclose anything at all?
- computerex 4y ago
  We don' have the details, it is true. But empirically and based on their report gpt-4 is notably better than chatgpt.
  - feanaro 4y ago
    Better, yes, and for that we have evidence. But is the improvement stemming simply from even more data? That's what I'm questioning.
    - stevenhuang 4y ago
      It's speculated it has same number of parameters, but more compute and is multi modal.
    - computerex 4y ago
      This paper is pretty approachable and goes over the "scaling laws" in detail: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 In short, yes. More data, higher quality data, more epochs on the data. That is the name of the game.
      - feanaro 4y ago
        That paper doesn't discuss GPT-4 at all. It does however contain this interesting excerpt (emphasis mine): > Although we may observe an emergent ability to occur at a certain scale, it is possible that the ability could be later achieved at a smaller scale—in other words, model scale is not the singular factor for unlocking an emergent ability. As the science of training large language models progresses, certain abilities may be unlocked for smaller models with new architectures, higher-quality data, or improved training procedures. For example, there are 14 BIG-Bench tasks5 for which LaMDA 137B and GPT-3 175B models perform at near-random, but PaLM 62B in fact achieves above-random performance, despite having fewer model parameters and training FLOPs. So it's not obvious that it should be so straightforward.
- typon 4y ago
  It's not true we know nothing. We know a little bit by using the two models from their API. Given the time per inference and the limit on messages per day for GPT4, I'm willing to bet it's doing around 10x more compute than GPT3.5. If that's because it has 10x more weights, I don't know. But it wouldn't be a terrible guess.
  - feanaro 4y ago
    So your estimate is that GPT4 has 1.75 trillion weights?
  - dwaltrip 4y ago
    Is there anything that affects inference compute time besides the number of parameters? Assuming same hardware, etc.
    - typon 4y ago
      Yes - for example adding memory to the attention mechanism (similar to RETRO or Memorizing Transformers paper)
UncleEntity 4y ago
Free is better than $$/token imho. If you have a use case or a bunch of disposable income then go with the “bitter” one.
bodyfour 4y ago
> There is no way around the bitter lesson. Isn't there? I'm certainly not sure, based on the results published over the last weeks and months. The giant GPT-{3.5,4} models show that if you make the model big enough and throw enough data at it you can produce an AI capable of conversing on basically any topic, in dozens of languages. There are plenty of different takes on how near-human its abilities are on specific tasks, but it's worth stepping back and appreciating how super-human the breadth of this knowledge is. But it's also not clear if a mega-model is anything close to the most efficient way of storing knowledge. After all, you don't need to memorize every fact in Wikipedia if you know how to effectively search it. And we're currently seeing a daily explosion in these capabilities. Today's flavor is interfacing with Wolfram, but we've also seen web searches, python coding, etc. That, I think, it the real superpower that comes out of this: you or I can answer a question by "doing a web search" or "query a database" or "use wolfram" or "develop a python program that finds the answer" However, an AI could do tasks like this just by "thinking" about it. Maybe it would be as natural as we find blinking. That to me is the real breakthrough in stuff like Alpaca -- start with a mega-model and prompt it with something like: "After this paragraph, you are going to be speaking to a AI model similar to yourself but much more primitive. Its task will involve interfacing with English speakers, so converse with it only in that language. It has access to the same {X,Y,Z} APIs you have so any time it has trouble answering a question, prefer to give hints about how it could find the answer using those APIs rather than providing the answer directly yourself. Only give an answer directly if it repeatedly fails to be able to answer it by using an API. I've provided a large set of standardized tests used by humans at this URL -- start by asking it questions intended for a preschool-aged child. Each time it is able to answer new questions at a given level correctly 99% of the time increase the material's level until it is able to achieve that score on a test designed for a Computer Science PhD candidate" How large would the "student" model have to be to succeed at this deep but narrower task? I think the answer right now "we have no idea". However if the model has the advantage that it can rely on external knowledge and tools from the start (and is rewarded by the "teacher" for doing just that) I bet it'll be a lot smaller than these mega-models. Sure, you wouldn't be able to disconnect the "student-AI" from its APIs and expect it to converse with you in Hungarian about the history of yacht design, but that might not be a capability it needs to have. My personal hunch is that we're going to find these "AI-taught specialist AI, with API access" models will be a lot smaller than most people are expecting. That's the moment when things REALLY change: instead of pairing a human with a mega-model AI, if specialized models are cheap someone can say "spin up 100K expert-programmer AIs and have them supervized by 5K expert-manager AIs and have them build XYZ" Or if you need it to work on an existing task you'd specialize further -- you'd go to your AI vendor and say "I'd like to license the weights for your expert-programmer model, but first have it read these 200 books I consider important to my problem domain and then show it every commit ever made by a human to my git repo and every design document I have"

by typon 4y ago

feanaro 4y ago

Isn't it the case that we literally have no clue how GPT4 and GPT3.5 are different in terms of training, given OpenAI doesn't want to disclose anything at all?

computerex 4y ago
We don' have the details, it is true. But empirically and based on their report gpt-4 is notably better than chatgpt.
- feanaro 4y ago
  Better, yes, and for that we have evidence. But is the improvement stemming simply from even more data? That's what I'm questioning.
  - stevenhuang 4y ago
    It's speculated it has same number of parameters, but more compute and is multi modal.
  - computerex 4y ago
    This paper is pretty approachable and goes over the "scaling laws" in detail: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 In short, yes. More data, higher quality data, more epochs on the data. That is the name of the game.
    - feanaro 4y ago
      That paper doesn't discuss GPT-4 at all. It does however contain this interesting excerpt (emphasis mine): > Although we may observe an emergent ability to occur at a certain scale, it is possible that the ability could be later achieved at a smaller scale—in other words, model scale is not the singular factor for unlocking an emergent ability. As the science of training large language models progresses, certain abilities may be unlocked for smaller models with new architectures, higher-quality data, or improved training procedures. For example, there are 14 BIG-Bench tasks5 for which LaMDA 137B and GPT-3 175B models perform at near-random, but PaLM 62B in fact achieves above-random performance, despite having fewer model parameters and training FLOPs. So it's not obvious that it should be so straightforward.
typon 4y ago
It's not true we know nothing. We know a little bit by using the two models from their API. Given the time per inference and the limit on messages per day for GPT4, I'm willing to bet it's doing around 10x more compute than GPT3.5. If that's because it has 10x more weights, I don't know. But it wouldn't be a terrible guess.
- feanaro 4y ago
  So your estimate is that GPT4 has 1.75 trillion weights?
- dwaltrip 4y ago
  Is there anything that affects inference compute time besides the number of parameters? Assuming same hardware, etc.
  - typon 4y ago
    Yes - for example adding memory to the attention mechanism (similar to RETRO or Memorizing Transformers paper)

UncleEntity 4y ago

Free is better than $$/token imho. If you have a use case or a bunch of disposable income then go with the “bitter” one.

bodyfour 4y ago

> There is no way around the bitter lesson. Isn't there? I'm certainly not sure, based on the results published over the last weeks and months. The giant GPT-{3.5,4} models show that if you make the model big enough and throw enough data at it you can produce an AI capable of conversing on basically any topic, in dozens of languages. There are plenty of different takes on how near-human its abilities are on specific tasks, but it's worth stepping back and appreciating how super-human the breadth of this knowledge is. But it's also not clear if a mega-model is anything close to the most efficient way of storing knowledge. After all, you don't need to memorize every fact in Wikipedia if you know how to effectively search it. And we're currently seeing a daily explosion in these capabilities. Today's flavor is interfacing with Wolfram, but we've also seen web searches, python coding, etc. That, I think, it the real superpower that comes out of this: you or I can answer a question by "doing a web search" or "query a database" or "use wolfram" or "develop a python program that finds the answer" However, an AI could do tasks like this just by "thinking" about it. Maybe it would be as natural as we find blinking. That to me is the real breakthrough in stuff like Alpaca -- start with a mega-model and prompt it with something like: "After this paragraph, you are going to be speaking to a AI model similar to yourself but much more primitive. Its task will involve interfacing with English speakers, so converse with it only in that language. It has access to the same {X,Y,Z} APIs you have so any time it has trouble answering a question, prefer to give hints about how it could find the answer using those APIs rather than providing the answer directly yourself. Only give an answer directly if it repeatedly fails to be able to answer it by using an API. I've provided a large set of standardized tests used by humans at this URL -- start by asking it questions intended for a preschool-aged child. Each time it is able to answer new questions at a given level correctly 99% of the time increase the material's level until it is able to achieve that score on a test designed for a Computer Science PhD candidate" How large would the "student" model have to be to succeed at this deep but narrower task? I think the answer right now "we have no idea". However if the model has the advantage that it can rely on external knowledge and tools from the start (and is rewarded by the "teacher" for doing just that) I bet it'll be a lot smaller than these mega-models. Sure, you wouldn't be able to disconnect the "student-AI" from its APIs and expect it to converse with you in Hungarian about the history of yacht design, but that might not be a capability it needs to have. My personal hunch is that we're going to find these "AI-taught specialist AI, with API access" models will be a lot smaller than most people are expecting. That's the moment when things REALLY change: instead of pairing a human with a mega-model AI, if specialized models are cheap someone can say "spin up 100K expert-programmer AIs and have them supervized by 5K expert-manager AIs and have them build XYZ" Or if you need it to work on an existing task you'd specialize further -- you'd go to your AI vendor and say "I'd like to license the weights for your expert-programmer model, but first have it read these 200 books I consider important to my problem domain and then show it every commit ever made by a human to my git repo and every design document I have"