3 ms·
It's an order of magnitude comparison. Let's just agree it's 100x-300x more parameters, and let's assume the open ai folks are pretty smart and have a sense fo
by danielmarkbruce 3y ago
It's an order of magnitude comparison.
Let's just agree it's 100x-300x more parameters, and let's assume the open ai folks are pretty smart and have a sense for the optimal number of tokens to train on.
- razodactyl 3y agoThis definitely. Andrej Karpathy himself mentions tuned weight initialisation in one of his lectures. The TinyGPT code he wrote goes through it. Additionally explanations for the raw mathematics of log likelihoods and their loss ballparks. Interesting low-level stuff. These researchers are the best of the best working for the company that can afford them working on the best models available.