The only source I can find is a reddit post from April fool's day, do you have a better one by chance? There's almost no way they make a 30T model that's usefu

The only source I can find is a reddit post from April fool's day, do you have a better one by chance? There's almost no way they make a 30T model that's useful at least? That's over 60TB of memory at 16bit, 15TB at 4 bit. That's 15TB that have to go through either VRAM or TPU Memory (TPUv4 is 32GB I think?). Yes you could distribute it and chunk it, but that would be insanely slow/expensive to run for anything more then a research demo. GPT-4 is most likely 1~2T from what I've heard, and they've been struggling to run that profitably/quickly judging by the crazy pricing and 25messages/3hours if you pay for premium chatgpt. EDIT: The only way this is possible is if they've pulled off some crazy optimization, which is definitely possible, but a lot of the research from them such as their own Chinchilla scaling suggest smaller models that are more fully trained are better. According to their chinchilla paper they would need over 330TB of training data for that, and an absurd amount of training time.