This definitely. Andrej Karpathy himself mentions tuned weight initialisation in one of his lectures. The TinyGPT code he wrote goes through it. Additionally e

This definitely. Andrej Karpathy himself mentions tuned weight initialisation in one of his lectures. The TinyGPT code he wrote goes through it. Additionally explanations for the raw mathematics of log likelihoods and their loss ballparks. Interesting low-level stuff. These researchers are the best of the best working for the company that can afford them working on the best models available.