3 ms·
If your ratio was based on 1 fact per parameter, then the ratio doesn't quite work correctly (not that this would be a reasonable premise anyways, but I thought
by godelski 3y ago
If your ratio was based on 1 fact per parameter, then the ratio doesn't quite work correctly (not that this would be a reasonable premise anyways, but I thought of what you said as a speaking tool). The thing is that the interaction of parameters scales super-linearly. Each neuron doesn't encode a single bit or piece of data but rather is part of an overall signal in the network. So we can see this expressed in what's often called "super position"[0] (personally, I don't like this language and think it is misleading), which is probably quite obvious when you think about it and what the (more physics) definition of weak emergence is (phenomena can be described in different ways at different scales. BUT macro can be derived from micro). We can see this with any discrete signal process (and more clearly in analogue). But architecture is going to play a big role here so there isn't really a good way to express the bound for DNNs but I'm sure someone has written down what it is for a n-headed transformer with dim d and a MLP ratio of r. If not, well, fuck it, I will.
[0] https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html