4 ms·
FWIW sentence-transformers truncates the input to at most 256 tokens by default; you might just be embedding the first paragraph or so.
by pizza 3y ago
FWIW sentence-transformers truncates the input to at most 256 tokens by default; you might just be embedding the first paragraph or so.
- lsb 3y agoI average the embeddings of every 512 bytes of the page text
- pizza 3y agoAh ok, that makes sense
- theolivenbaum 3y agoThat might actually be making things worse for longer articles. It probably would be better would be to index them separately and aggregate back to article level post query