10 ms·

Left unsaid in this piece is that OpenAI likely would have to increase parameters and compute by an order of magnitude (~10x) to train a new model that offers n

by cs702 4y ago

Left unsaid in this piece is that OpenAI likely would have to increase parameters and compute by an order of magnitude (~10x) to train a new model that offers noticeable improvements over GPT-4, due to the diminishing returns seen in "transformer scaling laws." Also, it's possible that OpenAI is still training GPT-4, perhaps with additional modalities, and will make future snapshots available as public releases.

version_five 4y ago
They could clean up the training data I bet. That would be where I'd focus next.
- mathematicaster 4y ago
  Is there any indication from OpenAI people that there are low hanging fruits to be picked in this direction?
  - summarity 4y ago
    It’s the indication data of current research: train more and better, current models are oversized and undertrained. A good foundation model can exhibit massive quality differences with just a tiny bit of quality fine tuning (e.g. Alpaca vs Koala) Personal opinion, not OAI/GH/MSFT’s
iliane5 4y ago
I bet they’re not saying how big of a model GPT-4 is because it’s actually much smaller we would expect. ChatGPT is IMO a heavily fine-tuned Curie sized model (same price via API + less cognitive capacity than even text davinci-003) so it would make sense that a heavily fine-tuned Davinci sized model would yield similar results to GPT-4.
- kyle_grove 4y ago
  Yannic Kilcher makes a similar supposition based on results from the tech report https://www.youtube.com/watch?v=2zW33LfffPc&pp=ygUOeWFubmljIGtpbGNoZXI%3D https://www.youtube.com/watch?v=2zW33LfffPc&pp=ygUOeWFubmljI... . It’s about 3/4 of the way through the video if memory serves.
- qumpis 4y ago
  I wonder why it's slower at inference time then (for members using their web UI), or rather, if it's similar in size to gpt3, how gpt3 is optimized in a way that gpt4 isn't or can't be? I'd expect that by now we would enjoy similar speeds but this hasn't yet happened.
  - MacsHeadroom 4y ago
    GPT-4 is the same speed as legacy GPT-3 ChatGPT for me. It's only occasionally slower, which I expect is due to load and not it being larger.
    - qumpis 4y ago
      Interesting. I remember when the speedup of chat-gpt happened, the API prices dropped by around 10x, so I'd imagine there were some tricks of making them run faster. If they still haven't implemented these, it would be positively surprising (to me) to see the model run at similar speeds as chatgpt now. It'd be a great achievement if they really packed such performance on similar architecture (say by just training longer)
      - MacsHeadroom 4y ago
        The speed-up of the free and default "chatGPT" happened because they switched it from the full size GPT-3.5 to "GPT-3.5-Turbo", which is likely a finetune of the 10x smaller GPT-3 Curie. If you have chatGPT Plus you can choose "Legacy" from the drop-down to get the smarter (and slower) 175B Parameter version of GPT-3.5. That version is the same speed as GPT-4 when load is low (early morning EST), which lends credence to the theory that GPT-4 is the same size as overparametrized GPT-3.
        qumpis 4y ago
        Oh this explains a lot, thank you for the information!
- KeplerBoy 4y ago
  I wouldn't bet on their pricing being indicative of their costs. If MSFT wants the ChatGPT-API to be a success and is willing to subsidize it, that's just how it is.
  - iliane5 4y ago
    It’s not only 10x cheaper, it’s also way faster at inference and not as smart as Davinci. IMO the only logical answer is that the model is just smaller.
ilaksh 4y ago
Actually what he has said is that the biggest performance gains were from the human feedback reinforcement learning. There are also all of the quantization and other tricks out there. Also they have demonstrated that the model already understands images but just haven't completed the API for this. So they use quantization to increase the speed by a factor of 3 while slightly increasing the parameter count. Maybe find a way to make the network more sparse and efficient so in the end with the quantization the model actually uses significantly less memory. and continue with the RHLF focusing on even more difficult tasks and those that incorporate visual data. Then instead of calling it GPT-5 they just call it GPT-4.5. Twice as fast as GPT-4, IQ goes from 130 to 155. And the API now allows images to be passed in and analyzed.
- brianjking 4y ago
  There is an API for multimodal computer vision and visual reasoning/VQA, and it's available, just not for normies. It's exclusively for their test group and then the Be My Eyes project at https://www.bemyeyes.com/ https://www.bemyeyes.com/.
  - ilaksh 4y ago
    I assume they will release this API publicly at some point? It's amazing the extreme levels of advantage that groups have depending on funding and connections.
    - hn_throwaway_99 4y ago
      > It's amazing the extreme levels of advantage that groups have depending on funding and connections. It's actually not.
    - brianjking 4y ago
      The multi-modal vision support? Yes. It's just temporarily available only to BeMyEyes. For now I'm using models like Salesforce/blip2 and OVF and Meta's Segment Anything for visual questioning.
  - 4y ago
ftxbro 4y ago
> Left unsaid in this piece is that OpenAI likely would have to increase parameters Maybe true, but he also said "We are not here to jerk ourselves off about parameter count" https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wont-matter-as-much-moving-forward/ https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo... Also, who says that the "transformer scaling laws" are the ultimate arbiter of LLM scaling? They overturned previous scaling laws and other scaling laws might overturn them. Furthermore, it's even possible that the transformer model won't even be used in later models. I remember Ilya making the point that just because the transformer model was the first one that looks like it can scale intelligence just by lighting up billions of dollars of GPUs, it doesn't mean it's the last one. Maybe it will even be like, the vacuum tube of AI models, and other ones are being made in secret. A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer.
- guyomes 4y ago
  > A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer This reminds me a TV interview of the author Patrick Modiano, just after he won the literature Nobel price. The presenter asked him if the money would help. The author answered essentially that the next time he would be in front of a white page, the money surely wouldn't help. In the case of surpassing transformers, money could help to give access to more compute power. It could also help to prevent the research from being public.
  - 0xfffafaCrash 4y ago
    If someone is already working on a problem full-time, money only helps to the extent that resources they can be buy with money are the limiting constraint. However, beyond deep work needed for a single individual, when you need to explore potential opportunities in a broad space of possibilities, money can hugely effect the search of that space because work needed for major breakthroughs remains parallelizable. You can delegate subtasks to people if you can afford those people. You can hire more of the few specialized people who know about a niche to work on your problem instead of other problems. You can exploit synergies from crosspollination of ideas from bringing together brilliant minds into the same conversations. The influx of money is very very likely to increase the pace of innovation in AI. The breadth of possible avenues for breakthroughs is largely yet-to-be-explored.
realitysballs 4y ago
I would suspect they probably conditioning data for gpt 5. Im guessing ‘training’ presupposes they have the training data primed & getting data into shape seems to be one of main cruxes
Terretta 4y ago
> Also, it's possible that OpenAI is still training GPT-4, perhaps with additional modalities, and will make future snapshots available as public releases. Read OpenAI API docs on GPT model versions carefully, and look at them again from time to time. https://platform.openai.com/docs/models https://platform.openai.com/docs/models
lumost 4y ago
We are also starting to run out of high quality corpus to train on at such model scales. While Video offers another large set of data, we'll have to look at further RL approaches in the next few years to continue scaling datasets.
- furyofantares 4y ago
  Is there any source for this, aside from it being oft repeated by internet speculators? Ilya has said the textual data situation is still quite good
  - swyx 4y ago
    yeah i need a source on this. GPT3 corpus is, what, a few hundred TB? absoultely nowhere near the total amount of tokens we could collect eg from youtube/podcasts
  - vidarh 4y ago
    If they're running into any limits in that respect, my bet would be that the limit would only on what is easily accessible to them without negotiating access, and that they can easily go another magnitude or two just with more incremental effort to strike deals. E.g. newspaper archives, national libraries and the like (I haven't looked at other languages, but GPT3's - since I don't know of any numbers for GPT4 - Norwegian corpus could easily be scaled at least two orders of magnitude with access to the Norwegian national library collection alone)
  - lumost 4y ago
    Depends on the quality. A ten trillion parameter model should require roughly 10 trillion tokens to train. Put another way, this would be roughly 10k Wikipedia’s or 67 Million books, Roughly 3-4 GitHub’s. It’s been established that LLMs are sensitive to corpus selection which is part of why we see anecdotal variance in quality across different LLM releases. While we could increase the corpus of text by loading social media comments, self published books, and other similar text - this may negatively impact final model quality/utility.
- paddim8 4y ago
fortuna86 4y ago
GPT-4 +
Buttons840 4y ago
In my machine learning experience, if it only takes 10x the parameters brings a significant improvement I feel lucky.
qaq 4y ago
Vicuna offers considerable improvement over LLaMA and it's just 13B delta to 65B model.
rafaelero 4y ago
How noticeable changes will be have little connection with loss reduction during training. Holding very complex thought processes may actually not diminish the loss function all that much. But they are very noticeable when we are interacting with these systems.
fileyfood500 4y ago
I often see mistakes when chatgot is faced with more spatial reasoning, and I wonder if changes as simple as deep convolutional subnetworks in intermediaries layers would help the language model fit better in these situations. In short, I’m excited to see where things go, and can definitely see room for great improvement through improvements to the architecture!

by cs702 4y ago

version_five 4y ago

They could clean up the training data I bet. That would be where I'd focus next.

mathematicaster 4y ago
Is there any indication from OpenAI people that there are low hanging fruits to be picked in this direction?
- summarity 4y ago
  It’s the indication data of current research: train more and better, current models are oversized and undertrained. A good foundation model can exhibit massive quality differences with just a tiny bit of quality fine tuning (e.g. Alpaca vs Koala) Personal opinion, not OAI/GH/MSFT’s

iliane5 4y ago

I bet they’re not saying how big of a model GPT-4 is because it’s actually much smaller we would expect. ChatGPT is IMO a heavily fine-tuned Curie sized model (same price via API + less cognitive capacity than even text davinci-003) so it would make sense that a heavily fine-tuned Davinci sized model would yield similar results to GPT-4.

kyle_grove 4y ago
Yannic Kilcher makes a similar supposition based on results from the tech report https://www.youtube.com/watch?v=2zW33LfffPc&pp=ygUOeWFubmljIGtpbGNoZXI%3D https://www.youtube.com/watch?v=2zW33LfffPc&pp=ygUOeWFubmljI... . It’s about 3/4 of the way through the video if memory serves.
qumpis 4y ago
I wonder why it's slower at inference time then (for members using their web UI), or rather, if it's similar in size to gpt3, how gpt3 is optimized in a way that gpt4 isn't or can't be? I'd expect that by now we would enjoy similar speeds but this hasn't yet happened.
- MacsHeadroom 4y ago
  GPT-4 is the same speed as legacy GPT-3 ChatGPT for me. It's only occasionally slower, which I expect is due to load and not it being larger.
  - qumpis 4y ago
    Interesting. I remember when the speedup of chat-gpt happened, the API prices dropped by around 10x, so I'd imagine there were some tricks of making them run faster. If they still haven't implemented these, it would be positively surprising (to me) to see the model run at similar speeds as chatgpt now. It'd be a great achievement if they really packed such performance on similar architecture (say by just training longer)
    - MacsHeadroom 4y ago
      The speed-up of the free and default "chatGPT" happened because they switched it from the full size GPT-3.5 to "GPT-3.5-Turbo", which is likely a finetune of the 10x smaller GPT-3 Curie. If you have chatGPT Plus you can choose "Legacy" from the drop-down to get the smarter (and slower) 175B Parameter version of GPT-3.5. That version is the same speed as GPT-4 when load is low (early morning EST), which lends credence to the theory that GPT-4 is the same size as overparametrized GPT-3.
      - qumpis 4y ago
        Oh this explains a lot, thank you for the information!
KeplerBoy 4y ago
I wouldn't bet on their pricing being indicative of their costs. If MSFT wants the ChatGPT-API to be a success and is willing to subsidize it, that's just how it is.
- iliane5 4y ago
  It’s not only 10x cheaper, it’s also way faster at inference and not as smart as Davinci. IMO the only logical answer is that the model is just smaller.

ilaksh 4y ago

Actually what he has said is that the biggest performance gains were from the human feedback reinforcement learning. There are also all of the quantization and other tricks out there. Also they have demonstrated that the model already understands images but just haven't completed the API for this. So they use quantization to increase the speed by a factor of 3 while slightly increasing the parameter count. Maybe find a way to make the network more sparse and efficient so in the end with the quantization the model actually uses significantly less memory. and continue with the RHLF focusing on even more difficult tasks and those that incorporate visual data. Then instead of calling it GPT-5 they just call it GPT-4.5. Twice as fast as GPT-4, IQ goes from 130 to 155. And the API now allows images to be passed in and analyzed.

brianjking 4y ago
There is an API for multimodal computer vision and visual reasoning/VQA, and it's available, just not for normies. It's exclusively for their test group and then the Be My Eyes project at https://www.bemyeyes.com/ https://www.bemyeyes.com/.
- ilaksh 4y ago
  I assume they will release this API publicly at some point? It's amazing the extreme levels of advantage that groups have depending on funding and connections.
  - hn_throwaway_99 4y ago
    > It's amazing the extreme levels of advantage that groups have depending on funding and connections. It's actually not.
  - brianjking 4y ago
    The multi-modal vision support? Yes. It's just temporarily available only to BeMyEyes. For now I'm using models like Salesforce/blip2 and OVF and Meta's Segment Anything for visual questioning.
- 4y ago

ftxbro 4y ago

> Left unsaid in this piece is that OpenAI likely would have to increase parameters Maybe true, but he also said "We are not here to jerk ourselves off about parameter count" https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wont-matter-as-much-moving-forward/ https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo... Also, who says that the "transformer scaling laws" are the ultimate arbiter of LLM scaling? They overturned previous scaling laws and other scaling laws might overturn them. Furthermore, it's even possible that the transformer model won't even be used in later models. I remember Ilya making the point that just because the transformer model was the first one that looks like it can scale intelligence just by lighting up billions of dollars of GPUs, it doesn't mean it's the last one. Maybe it will even be like, the vacuum tube of AI models, and other ones are being made in secret. A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer.

guyomes 4y ago
> A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer This reminds me a TV interview of the author Patrick Modiano, just after he won the literature Nobel price. The presenter asked him if the money would help. The author answered essentially that the next time he would be in front of a white page, the money surely wouldn't help. In the case of surpassing transformers, money could help to give access to more compute power. It could also help to prevent the research from being public.
- 0xfffafaCrash 4y ago
  If someone is already working on a problem full-time, money only helps to the extent that resources they can be buy with money are the limiting constraint. However, beyond deep work needed for a single individual, when you need to explore potential opportunities in a broad space of possibilities, money can hugely effect the search of that space because work needed for major breakthroughs remains parallelizable. You can delegate subtasks to people if you can afford those people. You can hire more of the few specialized people who know about a niche to work on your problem instead of other problems. You can exploit synergies from crosspollination of ideas from bringing together brilliant minds into the same conversations. The influx of money is very very likely to increase the pace of innovation in AI. The breadth of possible avenues for breakthroughs is largely yet-to-be-explored.

realitysballs 4y ago

I would suspect they probably conditioning data for gpt 5. Im guessing ‘training’ presupposes they have the training data primed & getting data into shape seems to be one of main cruxes

Terretta 4y ago

> Also, it's possible that OpenAI is still training GPT-4, perhaps with additional modalities, and will make future snapshots available as public releases. Read OpenAI API docs on GPT model versions carefully, and look at them again from time to time. https://platform.openai.com/docs/models https://platform.openai.com/docs/models

lumost 4y ago

We are also starting to run out of high quality corpus to train on at such model scales. While Video offers another large set of data, we'll have to look at further RL approaches in the next few years to continue scaling datasets.

furyofantares 4y ago
Is there any source for this, aside from it being oft repeated by internet speculators? Ilya has said the textual data situation is still quite good
- swyx 4y ago
  yeah i need a source on this. GPT3 corpus is, what, a few hundred TB? absoultely nowhere near the total amount of tokens we could collect eg from youtube/podcasts
- vidarh 4y ago
  If they're running into any limits in that respect, my bet would be that the limit would only on what is easily accessible to them without negotiating access, and that they can easily go another magnitude or two just with more incremental effort to strike deals. E.g. newspaper archives, national libraries and the like (I haven't looked at other languages, but GPT3's - since I don't know of any numbers for GPT4 - Norwegian corpus could easily be scaled at least two orders of magnitude with access to the Norwegian national library collection alone)
- lumost 4y ago
  Depends on the quality. A ten trillion parameter model should require roughly 10 trillion tokens to train. Put another way, this would be roughly 10k Wikipedia’s or 67 Million books, Roughly 3-4 GitHub’s. It’s been established that LLMs are sensitive to corpus selection which is part of why we see anecdotal variance in quality across different LLM releases. While we could increase the corpus of text by loading social media comments, self published books, and other similar text - this may negatively impact final model quality/utility.
paddim8 4y ago

fortuna86 4y ago

GPT-4 +

Buttons840 4y ago

In my machine learning experience, if it only takes 10x the parameters brings a significant improvement I feel lucky.

qaq 4y ago

Vicuna offers considerable improvement over LLaMA and it's just 13B delta to 65B model.

rafaelero 4y ago

How noticeable changes will be have little connection with loss reduction during training. Holding very complex thought processes may actually not diminish the loss function all that much. But they are very noticeable when we are interacting with these systems.

fileyfood500 4y ago

I often see mistakes when chatgot is faced with more spatial reasoning, and I wonder if changes as simple as deep convolutional subnetworks in intermediaries layers would help the language model fit better in these situations. In short, I’m excited to see where things go, and can definitely see room for great improvement through improvements to the architecture!