4 ms·
A model with 1 trillion parameters isn't going to perfectly recall 10 trillion random facts. To use round numbers. Contrary to the article, what it does do is
by Nevermark 3y ago
A model with 1 trillion parameters isn't going to perfectly recall 10 trillion random facts. To use round numbers.
Contrary to the article, what it does do is generalize and perform fallible but quick ordinary off the cuff reasoning. And often much better than a human, at the speed of its well worded responses.
(Obviously humans have the option to take longer, and do better. But we are definitely entering the territory of humans and machines differentiating where each is best, across a great deal of what humans do, vs. one being universally overwhelmingly better.)
- godelski 3y agoIf your ratio was based on 1 fact per parameter, then the ratio doesn't quite work correctly (not that this would be a reasonable premise anyways, but I thought of what you said as a speaking tool). The thing is that the interaction of parameters scales super-linearly. Each neuron doesn't encode a single bit or piece of data but rather is part of an overall signal in the network. So we can see this expressed in what's often called "super position"[0] (personally, I don't like this language and think it is misleading), which is probably quite obvious when you think about it and what the (more physics) definition of weak emergence is (phenomena can be described in different ways at different scales. BUT macro can be derived from micro). We can see this with any discrete signal process (and more clearly in analogue). But architecture is going to play a big role here so there isn't really a good way to express the bound for DNNs but I'm sure someone has written down what it is for a n-headed transformer with dim d and a MLP ratio of r. If not, well, fuck it, I will. [0] https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html