Training data is the main thing. We have lots and lots of text and text has the special property that a sequence of text contains a lot of information about wha

Training data is the main thing. We have lots and lots of text and text has the special property that a sequence of text contains a lot of information about what is going to come next and is easy for a user to create. This is a rather particular circumstance, the combination of so much freely available data and their being a lot of utility in a purely auto-regressive model. It is difficult to think about what other modalities are in a similar position.