The optimal way right now seems to be to train a gigantic model to get strong capabilities and use distillation to make a much smaller model stronger. But you s

The optimal way right now seems to be to train a gigantic model to get strong capabilities and use distillation to make a much smaller model stronger. But you seem to still need that gigantic model and we still don't know if we're running out of space to improve capability just by scaling up the model (for reference GPT-4 is likely 1.76T parameters and we may be able to get much better just by scaling up)