4 ms·

It could be that they are not training GPT-5 for a simple reason: Microsoft ran out of GPU compute [1] and they focus on meeting inference demand for now. Also

It could be that they are not training GPT-5 for a simple reason: Microsoft ran out of GPU compute [1] and they focus on meeting inference demand for now. Also, the GPT-4 message cap at chat.openai.com was shown as something along the lines of "we expect lower caps next week", then changed to "expect lower caps as we adjust for demand" to "GPT-4 currently has a cap of …". This sounds to me like they changed from having lots of compute to being limited by it. Also note how everything at OpenAI is now behind a sign up and their marketing has slowed down. Similarly, Midjourney has stopped offering their free plan due to lack of compute. Seems like we didn’t need a 6 months pause letter. Hardware constraints limit the progress for now. [1]: https://www.deeplearning.ai/the-batch/issue-192/ https://www.deeplearning.ai/the-batch/issue-192/

gremlinsinc 4y ago
That or they're working on something like a 10-30B input model, dubbed GPT-NextGen, that essentially has the same results as gpt4, but with a lot more performance gains, and speed, and improvements. GPT-5 will suck, if it's a similar ratio slower to gpt-4, than gpt-4 is to gpt-3.5. So, I think there's a lot of improvements where maybe gpt-4, is as far you go in terms of inputting data, and maybe better use cases are more customization of data trained on, or finding ways of going smaller, or even some model that just trains itself on the data requirements, similar to how we jump on google when we're stuck, it'd do the same and build up its knowledge that way. I also think we need improvements in vector stores that maybe add weights to "memories" based on time/frequency/recency/popularity.
- qumpis 4y ago
  That sounds like having a mixture of experts model (at high scale popularly developed by Google): train multiple specialised models (say embedders from text to a representation) that could be fed into a single model at the end. Each expert would be an adapter of sorts, activating depending on the type of input
furyofantares 4y ago
> the GPT-4 message cap at chat.openai.com was shown as something along the lines of "we expect lower caps next week" At the time I noticed that the wording they gave technically implied they expected the cap to get more limiting and then that's exactly what happened, and I haven't been able to work out if that is indeed what was the intended message or not.
- qumpis 4y ago
  (Why) Is that technically correct? I'm really curious since I too thought they meant that capping effect would increase (fewer messages allowed), and not decrease (more messages allowed), as was my intuitive understanding.
- deleted 4y ago
  [deleted]
WhackyIdeas 4y ago
You also notice the trickery going on?… I asked it to help me code something. Then it stopped midway through, so I asked it to continue from the last line. …It started from the beginning. Now at the same point, I asked it not to stop. To keep going. It started again from the beginning. It went like this for about another 10 or so prompts. Hell, I even asked it to help me write a better prompt to ask it to continue from the line it cut off and I then used that. It didn’t work at all. Then I ran out of prompts. Three hours later, it did the same crap to me and I lost around 14 prompts to it being ‘stuck’ in an eternal loop. Basically, OpenAI are sneaky devils. ‘Stuck’ my ass - that was intentional to free up resources.
- nulld3v 4y ago
  You can just send a single space character to get the AI to continue its previous output.
  - WhackyIdeas 4y ago
    I’ll try it. I have tried absolutely everything else, so may as well. Where did you read that? You’d think it would be there for people to read right next to the prompt.
    - nulld3v 4y ago
      It's emergent behavior, not something hardcoded into the model. Unfortunately, I don't remember where I read it originally...
  - chrisallenlane 4y ago
    Oh, neat. Thanks for the tip. I usually say "Please continue," but sometimes it reiterates too much of what it stated previously. (I've tried "Please continue where you left off," and so far that has worked 1/1 times that I've tried it.)
- FrenchDevRemote 4y ago
  or maybe you need to stop thinking everything is a conspiracy and realize bugs happen I've been using GPT everyday for the last 3 years, it never happened to me
huijzer 4y ago
Oh. And also they are probably making ChatGPT Plugins ready for public release. Maybe the competition can catch up on the language model, but they will not likely catch up soon to the best language model with the most plugin integrations.

gremlinsinc 4y ago

That or they're working on something like a 10-30B input model, dubbed GPT-NextGen, that essentially has the same results as gpt4, but with a lot more performance gains, and speed, and improvements. GPT-5 will suck, if it's a similar ratio slower to gpt-4, than gpt-4 is to gpt-3.5. So, I think there's a lot of improvements where maybe gpt-4, is as far you go in terms of inputting data, and maybe better use cases are more customization of data trained on, or finding ways of going smaller, or even some model that just trains itself on the data requirements, similar to how we jump on google when we're stuck, it'd do the same and build up its knowledge that way. I also think we need improvements in vector stores that maybe add weights to "memories" based on time/frequency/recency/popularity.

qumpis 4y ago
That sounds like having a mixture of experts model (at high scale popularly developed by Google): train multiple specialised models (say embedders from text to a representation) that could be fed into a single model at the end. Each expert would be an adapter of sorts, activating depending on the type of input

furyofantares 4y ago

> the GPT-4 message cap at chat.openai.com was shown as something along the lines of "we expect lower caps next week" At the time I noticed that the wording they gave technically implied they expected the cap to get more limiting and then that's exactly what happened, and I haven't been able to work out if that is indeed what was the intended message or not.

qumpis 4y ago
(Why) Is that technically correct? I'm really curious since I too thought they meant that capping effect would increase (fewer messages allowed), and not decrease (more messages allowed), as was my intuitive understanding.
deleted 4y ago
[deleted]

WhackyIdeas 4y ago

You also notice the trickery going on?… I asked it to help me code something. Then it stopped midway through, so I asked it to continue from the last line. …It started from the beginning. Now at the same point, I asked it not to stop. To keep going. It started again from the beginning. It went like this for about another 10 or so prompts. Hell, I even asked it to help me write a better prompt to ask it to continue from the line it cut off and I then used that. It didn’t work at all. Then I ran out of prompts. Three hours later, it did the same crap to me and I lost around 14 prompts to it being ‘stuck’ in an eternal loop. Basically, OpenAI are sneaky devils. ‘Stuck’ my ass - that was intentional to free up resources.

nulld3v 4y ago
You can just send a single space character to get the AI to continue its previous output.
- WhackyIdeas 4y ago
  I’ll try it. I have tried absolutely everything else, so may as well. Where did you read that? You’d think it would be there for people to read right next to the prompt.
  - nulld3v 4y ago
    It's emergent behavior, not something hardcoded into the model. Unfortunately, I don't remember where I read it originally...
- chrisallenlane 4y ago
  Oh, neat. Thanks for the tip. I usually say "Please continue," but sometimes it reiterates too much of what it stated previously. (I've tried "Please continue where you left off," and so far that has worked 1/1 times that I've tried it.)
FrenchDevRemote 4y ago
or maybe you need to stop thinking everything is a conspiracy and realize bugs happen I've been using GPT everyday for the last 3 years, it never happened to me

huijzer 4y ago

Oh. And also they are probably making ChatGPT Plugins ready for public release. Maybe the competition can catch up on the language model, but they will not likely catch up soon to the best language model with the most plugin integrations.