In information theory, information is a property of the distribution generating your data. Informally, for a given distribution, information is defined in terms

In information theory, information is a property of the distribution generating your data. Informally, for a given distribution, information is defined in terms of how much you learn from observing a sample from that distribution on average. If your distribution just puts all the probability mass on the number 3, you learn nothing new by gaining a sample. If your probability mass is really spread out, you gain a lot of information from observing a sample. I haven’t been keeping up with the LLM papers because it’s not really my academic interest but I do find them impressive, so maybe this has been figured out but there are two reasons they could accommodate new data modalities really easily: either the sequential data we generate in the real world is more similar than we would have guessed or the hard part isn’t in handling the domain-specific data, but learning to process and predict future signals really well. The former case would be more surprising to me but it is certainly a possibility — most domains have a “language” of sorts, e.g., visual motifs or licks that get passed between musicians. The network could be picking up on those “linguistic” features born out in data it is fed and just needs to alter its vocabulary from words to pixels or whatever. The second case would be my guess. If you have an algorithm that is good at predicting the future based on the recent past, the hard part is done. The rest is just optimizing it for the task (language, sound, video) at hand.