3 ms·
Does this semantic search work for longer documents? I know BERT&Co models are limited to around 500-1000 tokens.
by rolisz 5y ago
Does this semantic search work for longer documents? I know BERT&Co models are limited to around 500-1000 tokens.
- crucialfelix 5y agoYou can split the document by paragraph or sentence. Search results will then match a sentence in a document. You display with a link to the document with a reference to the span.
- jamesbriggs 5y agoI don't remember the paper, but the authors working on this same problem suggested to do this + also include an overlap between spans.
- crucialfelix 5y agoWould you include the overlap in the text for training, or just for the displayed result? I assume the later.