Actually the architecture of many of these models is profoundly unsurprising. The real difficulty actually is in training them. Just preprocessing the data for

Actually the architecture of many of these models is profoundly unsurprising. The real difficulty actually is in training them. Just preprocessing the data for a language model can take several hundred days of CPU time. Training takes months on thousands of GPUs. We