4 ms·

In it's current form, no chance. But if we continue on this trajectory (if that's possible of course), then pretty soon. I personally think there will be a barr

In it's current form, no chance. But if we continue on this trajectory (if that's possible of course), then pretty soon. I personally think there will be a barrier popping up soon for further advancement. However, today I am picking olives (not a joke) for my new career if it should happen (a joke; I would just retire).

flir 3y ago
You don't think it's already plateaued? In terms of reasoning ability, anyway.
- anonzzzies 3y ago
  Well yes, that's the feeling I have, but there are now so many very smart people working on smarter ways and billions are poured in (not only on the 'AI for car searches' type of sites...); I'm willing to give it some benefit of the doubt.
- kaibee 3y ago
  Not even close. You can look at the Llama v2 training curves and its clear that even 7B models are still under-trained and they themselves have only been trained on 2T tokens. A 70B or whatever size/model amalgamation GPT-4 is also likely under-trained. Apart from just plain under-training, there's that whole 'grokking' phenomenon that's been observed in smaller models, where it looks like they're not really improving for a long time, but _eventually_ suddenly undergo a massive improvement. I don't know if anyone has been willing to set enough cash on fire to see if grokking can happen in a large LLM and/or how long it would need to be trained for. There's also still a lot of juice left in improving the quality of datasets and in training on larger data sets. OpenAI, having built GPT-4, probably want to work on some applications of it given that they have enough of a lead on everyone else. I think there's definitely a few "burn piles of cash to train a better model" buttons at OpenAI right now, but they have no reason to use them when they're clearly in the lead anyway. There's also the actual limiting factor: there are only so many A100s/H100s in the world, but the amount is growing.
  - Suneel478 3y ago
    Yeah, even RL tuning can be applied to improve reasoning paths for specialized applications, something like RLAIF by anthropic but for specialized agents

flir 3y ago

You don't think it's already plateaued? In terms of reasoning ability, anyway.

anonzzzies 3y ago
Well yes, that's the feeling I have, but there are now so many very smart people working on smarter ways and billions are poured in (not only on the 'AI for car searches' type of sites...); I'm willing to give it some benefit of the doubt.
kaibee 3y ago
Not even close. You can look at the Llama v2 training curves and its clear that even 7B models are still under-trained and they themselves have only been trained on 2T tokens. A 70B or whatever size/model amalgamation GPT-4 is also likely under-trained. Apart from just plain under-training, there's that whole 'grokking' phenomenon that's been observed in smaller models, where it looks like they're not really improving for a long time, but _eventually_ suddenly undergo a massive improvement. I don't know if anyone has been willing to set enough cash on fire to see if grokking can happen in a large LLM and/or how long it would need to be trained for. There's also still a lot of juice left in improving the quality of datasets and in training on larger data sets. OpenAI, having built GPT-4, probably want to work on some applications of it given that they have enough of a lead on everyone else. I think there's definitely a few "burn piles of cash to train a better model" buttons at OpenAI right now, but they have no reason to use them when they're clearly in the lead anyway. There's also the actual limiting factor: there are only so many A100s/H100s in the world, but the amount is growing.
- Suneel478 3y ago
  Yeah, even RL tuning can be applied to improve reasoning paths for specialized applications, something like RLAIF by anthropic but for specialized agents