3 ms·
> A similar thing happened with VGGnet in image classification I know I am at risk of a "Nobody will ever need more than 640KB of memory" comment, but the mode
by teruakohatu 6y ago
> A similar thing happened with VGGnet in image classification
I know I am at risk of a "Nobody will ever need more than 640KB of memory" comment, but the model size has exploded far in excess of our ability to improve GPU cards.
GPT-3 is two orders of magnitude larger than VGGnet-16. Back when the VGG paper was published a Titan Z (2014 gen) had 12gb of RAM, while a Titan RTX (current/previous gen) has just doubled to 24gb.
- social_quotient 6y agoYour point is still valid but yesterday I learned about the newest workstation cards from nvidia - A6000 series with 48GB https://www.engadget.com/nvidia-rtx-a6000-a40-gpu-professional-144053827.html https://www.engadget.com/nvidia-rtx-a6000-a40-gpu-profession...
- jononor 6y agoWe can now each the performance of original VGG16/VGG19 with models that are much smaller. On ImageNet MobileNetsV2 matches the performance with 1/30 the size of weights, and same order of magnitude reduction in inference time and RAM usage. The same will probably happen with GPT-3. Of course we will probably also get models that surpass GPT-3 at the cost of even larger and more expensive models. The two are not at all exclusive.