A weird thing about LLM is that certain internal structures (layers, activation functions) are much more effective than others. No one knows why. There’s a lot

A weird thing about LLM is that certain internal structures (layers, activation functions) are much more effective than others. No one knows why. There’s a lot of work likely being done by it’s internal structure that we don’t really understand, which would explain why it is able to do more than just empirical analysis.