3 ms·
Yeah it’s an off the shelf sentence-transformer model from over a year ago. The demo was more to show off the embedding database, but the embeddings themselves
by lsb 3y ago
Yeah it’s an off the shelf sentence-transformer model from over a year ago. The demo was more to show off the embedding database, but the embeddings themselves are slightly useful too.
I don’t keep any analytics on the page about what people find and don’t find, so I haven’t set myself up to improve the search results :/
- pizza 3y agoFWIW sentence-transformers truncates the input to at most 256 tokens by default; you might just be embedding the first paragraph or so.
- lsb 3y agoI average the embeddings of every 512 bytes of the page text
- pizza 3y agoAh ok, that makes sense
- theolivenbaum 3y agoThat might actually be making things worse for longer articles. It probably would be better would be to index them separately and aggregate back to article level post query