>I trained a small (~10 million parameter) transformer following Andrej Karpathy’s excellent tutorial, Let’s build GPT: from scratch, in code, spelled out As s

>I trained a small (~10 million parameter) transformer following Andrej Karpathy’s excellent tutorial, Let’s build GPT: from scratch, in code, spelled out As soon as I learned about Andrej Karpathy's NanoGPT, I trained it on War and Peace (in Russian), and what I found interesting is that it almost grokked Russian grammar despite being just a 3 MB model. Russian language has a complex synthetic-inflectional structure. For example, preposition "na" ("upon") requires the following noun to be in accusative case, which is manifested as ending -a for animate masculine nouns, but as null ending for inanimate nouns, or as -ia for nouns which end in a "soft consonant", -u for feminine nouns, etc. etc. Or the verb "to use" requires the following noun to be in instrumental case if it's used as a tool. Although it's not perfect and had mistakes, I found it interesting that NanoGPT was able to infer certain complex rules in just 3 minutes of training - and I searched in the texts for the exact examples it generated and found nothing verbatim. However, despite understanding grammar more-less, semantically, it was complete nonsense.