3 ms·
Isn't this also(just?) a description of how high-dimensional embedding spaces work? Putting every kind of concept all in the same space is going to lead to some
by DonsDiscountGas 3y ago
Isn't this also(just?) a description of how high-dimensional embedding spaces work? Putting every kind of concept all in the same space is going to lead to some weird stuff. Different regions of the latent space will cover different concepts, with very uneven volumes, and local distances will generally be meaningful (red vs green) but long-distances won't (red vs. ennui).
I guess we could also look at it the other way; embedding spaces work this way because the underlying neurons work this way.
- bjackman 3y agoI had the same feeling, I have always seen it like: 1. Neurons encode a concept and activate when it shows up. 2. No, it's way more complicated and mysterious than that. And now this seems to add: 3. Actually, it's only more complicated than that in a fairly straightforward mathematical sense. It's not that mysterious at all. I suspect this means that either I'm not picking up on subtleties in the article, or Scott is representing it in a way that slightly oversimplified the situation! On the other hand, the last quote in the article from the researchers does seem to be hitting the "it's not that mysterious" note. A simple matter of very hard engineering. So, I dunno. Cool!