4 ms·

The fundamental idea is "operating in high dimensions", and this does have some solid footing, e.g. see Cover's Theorem (https://en.wikipedia.org/wiki/Cover's_t

by armcat 5y ago

The fundamental idea is "operating in high dimensions", and this does have some solid footing, e.g. see Cover's Theorem (https://en.wikipedia.org/wiki/Cover's_theorem) https://en.wikipedia.org/wiki/Cover's_theorem). In fact I recently did a presentation of another paper (from FB AI Research and Carnegie Mellon), exploring the concept of projections to higher dimensions for sentence encoding tasks, see here: https://github.com/acatovic/paper-lunches/blob/main/fb-randsent/randsent.ipynb https://github.com/acatovic/paper-lunches/blob/main/fb-rands.... There is fair amount of research in the area of high dimensional computing, as well as with sparse representations, which seem to be grounded in neuroscience. As others have pointed out, a number of commercial research labs exists. There is Numenta with their Hierarchical Temporal Memory, and Vicarious (whose founder was one of Numenta's co-founders), as well as Cortical.io (who are borrowing sparse binary coding concepts from Numenta, combining it with self-organizing maps, and applying it to document understanding tasks).

sayonaraman 5y ago
this is awesome, thanks for the link! There seems to be mutual propagation from NLP to CV and from CV to NLP, i'm wondering if there is a visual counterpart for these "Random Encoders". The SOTA for image/text embeddings currently seems to be CLIP [1]. [1] https://openai.com/blog/clip/ https://openai.com/blog/clip/
- armcat 5y ago
  You are welcome :-). With text we have fairly discrete and granular units, i.e. "words", and we see sentences as compositions of words. For images however, it's not so simple. We present the entire image (array of pixels times the number of channels), and all the various layers within the CNN automatically extract "discrete" units like edges, bends etc, but what we get at the end is a semantic encoding of this entire image. But, you could in principle take the encoding of an image using e.g. VGG16 or ResNet or whatever, and then project that encoding onto randomly initialized higher-dimensional space (such as a very high dimensional Echo State Network), and then use that output in your downstream task, like classifying obstacles on the road. I suspect the same principles as what I wrote in the Jupyter Notebook would also apply here, and you will get a performance boost. Not sure if this was done already (computer vision is not really my focus) ;-)

by armcat 5y ago

sayonaraman 5y ago

this is awesome, thanks for the link! There seems to be mutual propagation from NLP to CV and from CV to NLP, i'm wondering if there is a visual counterpart for these "Random Encoders". The SOTA for image/text embeddings currently seems to be CLIP [1]. [1] https://openai.com/blog/clip/ https://openai.com/blog/clip/

armcat 5y ago
You are welcome :-). With text we have fairly discrete and granular units, i.e. "words", and we see sentences as compositions of words. For images however, it's not so simple. We present the entire image (array of pixels times the number of channels), and all the various layers within the CNN automatically extract "discrete" units like edges, bends etc, but what we get at the end is a semantic encoding of this entire image. But, you could in principle take the encoding of an image using e.g. VGG16 or ResNet or whatever, and then project that encoding onto randomly initialized higher-dimensional space (such as a very high dimensional Echo State Network), and then use that output in your downstream task, like classifying obstacles on the road. I suspect the same principles as what I wrote in the Jupyter Notebook would also apply here, and you will get a performance boost. Not sure if this was done already (computer vision is not really my focus) ;-)