>> We can break codes by cryptography, a network pre trained on all human languages should be able to derive insights unsupervised. Well, we have a few models

>> We can break codes by cryptography, a network pre trained on all human languages should be able to derive insights unsupervised. Well, we have a few models of English (large language models like BERT and friends, or the GPT's etc). What insights have they derived about English? What insights have we derived about English from such models? Edit: nice paper to read before forming expectations about insights deriveable from language modelling. Climbing towards NLU: On Meaning, Form and Understanding in the Age of Data. https://aclanthology.org/2020.acl-main.463.pdf https://aclanthology.org/2020.acl-main.463.pdf