5 ms·
As I understand it the approach here is to use an approximate nearest neighbor database to retrieve highly relevant tokens from across large documents using the
by space_fountain 3y ago
As I understand it the approach here is to use an approximate nearest neighbor database to retrieve highly relevant tokens from across large documents using the existing attention heads. So each attention head retrieves context from entire document. They say this can work without fine tuning, but performance improves with it. This is apparently extending this piece of prior work, but they've managed to re-range the linear algebra of attention so they only need one database for all attention heads across all layers of the model. I'm a bit confused how attention would here for layers below the top and a bit confused about how position is encoded for tokens across a long document like this.
- im3w1l 3y agoI don't understand how this could work. Like if you select a small fixed number of tokens from a large document won't you necessarily lose a lot of important data?
- MagicMoonlight 3y agoWhen you read a book do you remember every word? Even every chapter? You only need the important concepts, not individual words
- Xelynega 3y agoI feel that's a bit different. When skimming or reading a book, even though we don't remember all the information typically a lot of the information is dependent on prior information, so we form a sparse knowledge graph from the entire text. If I'm understanding the article, this approach would not use the skipped words to influence the output, so I thinks ita necessarily different.