It's mainly hardware upgrades (new TSMC nodes), training data quantity/quality improvements, and model architecture (and broader software) changes. Here's a fun

It's mainly hardware upgrades (new TSMC nodes), training data quantity/quality improvements, and model architecture (and broader software) changes. Here's a fun project I've had my eye on recently: https://github.com/KellerJordan/modded-nanogpt/ https://github.com/KellerJordan/modded-nanogpt/