4 ms·

Okay, I just tried this on my pet transformer training benchmark and the results are very disappointing; it converges much more slowly than just using RMSNorm.

Okay, I just tried this on my pet transformer training benchmark and the results are very disappointing; it converges much more slowly than just using RMSNorm. It either needs some significant hyperparameter tuning (besides tweaking alpha, which doesn't seem to do much for me), or some fancier initialization (tried both pytorch default and orthogonal, no difference), or maybe my scalar optimizer doesn't work on it (I have a custom optimizer for scalars which speeds up convergence vs Adam, but for DyT layers it seems to be just as good as Adam), or maybe it only catches up after billions of tokens (which I don't have the budget to test for so long).

kadushka 2y ago
Which model are you training and on what dataset?
- kouteiheika 2y ago
  It's a fully custom architecture, heavily inspired by the modded-nanogpt speedrun (https://github.com/KellerJordan/modded-nanogpt https://github.com/KellerJordan/modded-nanogpt) but written fully from scratch and further tweaked/modified. I use it for experiments and as a testbed when developing my training harness (which I use for training other models too, and which receives all of my non-LLM-specific improvements like e.g. better than Adam optimizers, a custom GPU memory allocator, custom gradient accumulation that accumulates directly into the optimizers' state without using extra VRAM for gradient, etc.). For the dataset I just use FineWeb-Edu.
  - kadushka 2y ago
    Wow, thank you for the link to the code - I haven't seen it before - it contains a ton of useful tricks. Lots to learn from there.
kouteiheika 2y ago
Slight update, more fancy initialization of DyT weights (instead of having them be ones) seems to help a lot in my case (although it's still not as good as just using RMSNorm). Do something like this on the very first training step (`x` is the input to the layer): y = x.to(torch.float32) y = y * torch.rsqrt(y.pow(2).mean(-1, keepdim=True) + 1e-6) z = torch.tanh(self.alpha * x) scale = (y / (z + 1e-6)).mean(dim = -2).flatten() self.weight.detach().copy_(scale) This basically tries to initialize the weights so that the output of DyT is closer to what RMSNorm would have outputted, and it seems to help.

kadushka 2y ago

Which model are you training and on what dataset?

kouteiheika 2y ago
It's a fully custom architecture, heavily inspired by the modded-nanogpt speedrun (https://github.com/KellerJordan/modded-nanogpt https://github.com/KellerJordan/modded-nanogpt) but written fully from scratch and further tweaked/modified. I use it for experiments and as a testbed when developing my training harness (which I use for training other models too, and which receives all of my non-LLM-specific improvements like e.g. better than Adam optimizers, a custom GPU memory allocator, custom gradient accumulation that accumulates directly into the optimizers' state without using extra VRAM for gradient, etc.). For the dataset I just use FineWeb-Edu.
- kadushka 2y ago
  Wow, thank you for the link to the code - I haven't seen it before - it contains a ton of useful tricks. Lots to learn from there.

kouteiheika 2y ago

Slight update, more fancy initialization of DyT weights (instead of having them be ones) seems to help a lot in my case (although it's still not as good as just using RMSNorm). Do something like this on the very first training step (`x` is the input to the layer): y = x.to(torch.float32) y = y * torch.rsqrt(y.pow(2).mean(-1, keepdim=True) + 1e-6) z = torch.tanh(self.alpha * x) scale = (y / (z + 1e-6)).mean(dim = -2).flatten() self.weight.detach().copy_(scale) This basically tries to initialize the weights so that the output of DyT is closer to what RMSNorm would have outputted, and it seems to help.