3 ms·
We should not be surprised to see new language models achieving GPT-3 performance with fewer parameters. The purpose (I assume, as an outsider) of the GPT-* pro
by blt 6y ago
We should not be surprised to see new language models achieving GPT-3 performance with fewer parameters. The purpose (I assume, as an outsider) of the GPT-* project is to try to find the upper bound of how good a language model can be, without caring too much about efficiency. GPT-3 essentially scaled up the known good architecture of GPT-2.
If you have a really big compute cluster, it makes sense to do experiments like this. It would be foolish to constantly try new methods without occasionally checking to see how far you can push current methods.
A similar thing happened with VGGnet in image classification. It achieved SoTA with a huge amount of parameters, using the standard techniques of the time, but increasing the network depth. Later, people discovered a lot of tricks to get similar accuracy with fewer parameters.
- teruakohatu 6y ago> A similar thing happened with VGGnet in image classification I know I am at risk of a "Nobody will ever need more than 640KB of memory" comment, but the model size has exploded far in excess of our ability to improve GPU cards. GPT-3 is two orders of magnitude larger than VGGnet-16. Back when the VGG paper was published a Titan Z (2014 gen) had 12gb of RAM, while a Titan RTX (current/previous gen) has just doubled to 24gb.
- social_quotient 6y agoYour point is still valid but yesterday I learned about the newest workstation cards from nvidia - A6000 series with 48GB https://www.engadget.com/nvidia-rtx-a6000-a40-gpu-professional-144053827.html https://www.engadget.com/nvidia-rtx-a6000-a40-gpu-profession...
- jononor 6y agoWe can now each the performance of original VGG16/VGG19 with models that are much smaller. On ImageNet MobileNetsV2 matches the performance with 1/30 the size of weights, and same order of magnitude reduction in inference time and RAM usage. The same will probably happen with GPT-3. Of course we will probably also get models that surpass GPT-3 at the cost of even larger and more expensive models. The two are not at all exclusive.