4 ms·

This is really cool and the interactive article they created does a great job at diving deeper: https://attentionneuron.github.io/ https://attentionneuron.githu

• mosicr 5y ago
  Although the paper deals with visual applications, in principle it should be possible to extend to text and other data sources (input data is projected into numeric form ). "A similar setup to the CartPole experiment with extra noisy channels could enable a system that receives thousands of noisy input channels to identify the small subset of channels with relevant information. " This could be interesting - real time monitoring of a number of number of securities ticker symbols to hunt for a signal. Hedge funds and investment banks are buyers of data sets that often do not have any useful trading signals. A system like this might indicate point of focus for data set prep.
• gwern 5y ago
  I think if you wanted to do this at scale, you would prefer Perceiver-IO ( https://arxiv.org/abs/2103.03206#deepmind https://arxiv.org/abs/2103.03206#deepmind https://arxiv.org/abs/2107.14795#deepmind https://arxiv.org/abs/2107.14795#deepmind ).