3 ms·
Once your model and map get larger than the thing it is modeling/mapping, then what? Let us hope the Pigeonhole principle isn't flawed, else we can find oursel
by Jerrrrrrry 2y ago
Once your model and map get larger than the thing it is modeling/mapping, then what?
Let us hope the Pigeonhole principle isn't flawed, else we can find ourselves batteries in the Matrix.
- anon291 2y agoIn the paper 'Hopfield networks are all you need', they calculate the total number of things able to be 'stored' in the attention layers, and it's exponential in the number of parameters. So essentially, you can store more 'ideas' in an LLM than there are particles in the universe. I think we'll be good. From a technical perspective, this is due to the softmax activation function that causes high degrees of separation between memory points.
- Jerrrrrrry 2y ago> So essentially, you can store more 'ideas' in an LLM than there are particles in the universe. I think we'll be good. If it can compress humanities knowledge corpus to <80gb unquanti-optimized, I think between my ironically typo'd double negative, and your seemingly genuine confirmation, to be absolute confirmation: we are fukt