Extremely expensive, though as I understand it, the goal is to get maximum performance out of a given model size so that you can actually inference at product s

Extremely expensive, though as I understand it, the goal is to get maximum performance out of a given model size so that you can actually inference at product scale. A few extra million for training is expensive, but then consider what it costs to run inference for something like Bing for 100 million daily active users. If they can develop new methods to “overtrain” these models they will get more bang out of the smaller parameter model buck.