It used to be 2x number of parameters, so about 350GB. Facebook came with an optimization a couple months ago so it takes only half the memory, so about 175GB

It used to be 2x number of parameters, so about 350GB. Facebook came with an optimization a couple months ago so it takes only half the memory, so about 175GB for GPT-3, not sure if OpenAI has implemented that yet. BLOOM, which is around the size of GPT-3, though not as good currently, works with these optimizations and should then be possible to run with 8x 24GB consumer GPUs. Huggin face has a nice blog post on this https://huggingface.co/blog/hf-bitsandbytes-integration https://huggingface.co/blog/hf-bitsandbytes-integration