If you read through the paper where they present switch transformers, they train all models to the same performance, so for example t5-large, at 770M parameters

If you read through the paper where they present switch transformers, they train all models to the same performance, so for example t5-large, at 770M parameters[1] is just as good as this model. That’s only 1.54GB [1] https://huggingface.co/t5-large https://huggingface.co/t5-large