3 ms·
> They're going to diverge if the underlying manner in which data gets memorized and encoded, such as with RNNs, like RWKV. In the original paper (https://arxi
by in-silico 9mo ago
> They're going to diverge if the underlying manner in which data gets memorized and encoded, such as with RNNs, like RWKV.
In the original paper (https://arxiv.org/abs/2405.07987 https://arxiv.org/abs/2405.07987) the authors also compared the representations of transformer-based LLMs to convolution-based image models. They found just as much alignment between them as when both models were transformers.
- observationist 9mo agoVery interesting - the human bias implicit to the structure of the data we collect might be critical, but I suspect there's probably a great number theory paper somewhere in there that validates the Platonic Representation idea. How would you correct for something like "the subset of information humans perceive and find interesting" versus "the set of all information available about a thing that isn't noise" and determine what impact the selection of the subset has on the structure of things learned by AI architectures? You'd need to account for optimizers, architecture, training data, and so on, but the results from those papers are pretty compelling.