4 ms·

Here's my semi-educated guess based on observing the rate of progress in this field over the last years: I think there is still improvement to be made in base

by M4v3R 3y ago

Here's my semi-educated guess based on observing the rate of progress in this field over the last years: I think there is still improvement to be made in base models. The amount of progress between GPT-2 and GPT-4 in just 4 years is astonishing so I would be surprised if GPT-4 is the limit of what we can do with this technology. I expect at least one more major version that will significantly outperform GPT-4. After that I think the major focus will be making these models more performant, expanding the context window, making them less resource-hungry, adding multimodality and composing them into larger architectures using techniques similar to Chain of Thought - building fully fledged general-purpose agents. These agents will be able to outperform humans at most of the tasks we throw at them. If I were to guess I'd say this will happen by the end of this decade.

avereveard 3y ago
It seems to me we'll see large glue model, which will be able to read from model last layer and write directly into model first layer and are trained at composing them. Some is already happening, and it cannot be generalized, but the idea of being able to glue already trained medium sized model together and then do a retrain of the whole thing to improve performances seems already establishing itself in products and will allow much efficient expenditure of training resources.