4 ms·

> if GPT5 is nearly as advertised, watch the govt swoop in under national security guise Re: GPT5, are there any .. reasonable/credible sources of information

> if GPT5 is nearly as advertised, watch the govt swoop in under national security guise Re: GPT5, are there any .. reasonable/credible sources of information on the subject? I've become deaf from all the speculation and while i am very curious, i'm unsure if anything substantiated has actually come out. Especially when considering speculation from Sam himself.

blihp 4y ago
There's barely any credible information about GPT4 (i.e. OpenAI hasn't said very much about what's going on behind the curtain) and there's absolutely none re: any releases beyond that.
zarzavat 4y ago
It’s unlikely to be any time soon. Despite productization by OpenAI, LLMs are still an active area of research. Research is unpredictable. It may take years to gather enough fundamental results to make a GPT-5 core model that is substantially better than GPT-4. Or a key idea could be discovered tomorrow. Moreover, previous advances in GPTs have come from data scaling: throwing more data at the training process. But data corpus sizes have started to peak - there is only so much high quality data in the world, and model sizes have reached the limits of what is sensible at inference time. What OpenAI can do while they are waiting is more of the easy stuff, for example more multimodality: integrating DALL-e with GPT-4, adding audio support, etc. They can also optimize the model to make it run faster.
- SamPatt 4y ago
  There are diminishing returns from compute time but it looks like even though they are diminishing there's still a fair bit on the table. Though my guess is that GPT-5 will be the last model which gains significantly from just adding more compute to the current transformer architecture. Who the hell knows what comes next though?
- dougmwne 4y ago
  I keep keep hearing people claim we are at the end of corpus scaling, but that seems totally unfounded. Where has it been proven you can’t running the training set through in multiple epochs in randomized order? Who’s to say you can’t collect all the non-English corpus and have the performance transfer to English? Who’s to say you can’t run the whole damn thing backwards and still have it learn something?
  - zarzavat 4y ago
    When you train a NN, you train until the loss converges (or until you run out of money). If it happens that you exhaust your training data and the loss is still decreasing, then yes you keep going with a second epoch. On the other hand, if your loss has already converged then starting a new epoch on the same data is just burning money. In any case this wouldn’t be a data size scale but a model size scale. Model size scaling has its own set of limits, such as the aforementioned inference times. It seems safe to assume that OpenAI chose data and model size values for GPT-4 that were basically as big as they could go. We don’t know anything about GPT-4 or how it is trained or what the training data is, so from that perspective anything we can say about it is unfounded speculation. This is an important asterisk to any discussion of it.
- whiplash451 4y ago
  You are missing the scaling on context size.
- bcherny 4y ago
  I know ChatGPT is trained on Wikipedia and other open datasets. I wonder what would happen if you gave it the entire Bing index?

blihp 4y ago

There's barely any credible information about GPT4 (i.e. OpenAI hasn't said very much about what's going on behind the curtain) and there's absolutely none re: any releases beyond that.

zarzavat 4y ago

It’s unlikely to be any time soon. Despite productization by OpenAI, LLMs are still an active area of research. Research is unpredictable. It may take years to gather enough fundamental results to make a GPT-5 core model that is substantially better than GPT-4. Or a key idea could be discovered tomorrow. Moreover, previous advances in GPTs have come from data scaling: throwing more data at the training process. But data corpus sizes have started to peak - there is only so much high quality data in the world, and model sizes have reached the limits of what is sensible at inference time. What OpenAI can do while they are waiting is more of the easy stuff, for example more multimodality: integrating DALL-e with GPT-4, adding audio support, etc. They can also optimize the model to make it run faster.

SamPatt 4y ago
There are diminishing returns from compute time but it looks like even though they are diminishing there's still a fair bit on the table. Though my guess is that GPT-5 will be the last model which gains significantly from just adding more compute to the current transformer architecture. Who the hell knows what comes next though?
dougmwne 4y ago
I keep keep hearing people claim we are at the end of corpus scaling, but that seems totally unfounded. Where has it been proven you can’t running the training set through in multiple epochs in randomized order? Who’s to say you can’t collect all the non-English corpus and have the performance transfer to English? Who’s to say you can’t run the whole damn thing backwards and still have it learn something?
- zarzavat 4y ago
  When you train a NN, you train until the loss converges (or until you run out of money). If it happens that you exhaust your training data and the loss is still decreasing, then yes you keep going with a second epoch. On the other hand, if your loss has already converged then starting a new epoch on the same data is just burning money. In any case this wouldn’t be a data size scale but a model size scale. Model size scaling has its own set of limits, such as the aforementioned inference times. It seems safe to assume that OpenAI chose data and model size values for GPT-4 that were basically as big as they could go. We don’t know anything about GPT-4 or how it is trained or what the training data is, so from that perspective anything we can say about it is unfounded speculation. This is an important asterisk to any discussion of it.
whiplash451 4y ago
You are missing the scaling on context size.
bcherny 4y ago
I know ChatGPT is trained on Wikipedia and other open datasets. I wonder what would happen if you gave it the entire Bing index?