> Free bonus I haven't seen mentioned elsewhere yet: you can take the average of the N embeddings forming a document to figure out if you should look at the N e

> Free bonus I haven't seen mentioned elsewhere yet: you can take the average of the N embeddings forming a document to figure out if you should look at the N embeddings individually. This does over smooth too, ex. my original test document is the GPT-4 sparks paper, and the variety of subjects mentioned and length (100+ pages) meant it was over smooth when searching for the particular example I wanted it to retrieve (the unicorn SVG) This is something I've been pondering for doing lookups across essays! That is, I saw some people doing per-paragraph embeddings, so you could do something like mouse-select a paragraph in an essay, and then get a list of all other paragraphs site-wide which are 'similar'. However, it seems to me like this would run into trouble with small or generic paragraphs (because their embeddings will be noisy & extreme & there's a lot of them and you may wind up pulling up only sibling paragraphs, defeating the point)... But what if you did it as a tree? Embed each paragraph, then average a couple paragraphs, then average the paragraphs embeddings, and so on to the whole-essay level. Then do the lookups as usual. Any given paragraph might instead pull up a genuinely relevant section, or a range of paragraphs, or a whole essay, rather than some spurious paragraph. (And you could constrain the lookup to be tree-like too: first find the most similar essay-level embedding, then see if any of that essay's section embeddings are more relevant, then check its paragraphs embeddings etc, and stop when the distance stops decreasing compared to the parent node. This would let you enforce diversity of results: eg take only 1 hit per essay, so you don't wind up showing the reader 10 hits which are all on the same page.) Does that correspond to anything? https://en.wikipedia.org/wiki/Hierarchical_Navigable_Small_World_graphs https://en.wikipedia.org/wiki/Hierarchical_Navigable_Small_W... seems tree-like but the trees used in it do not seem to correspond to the actual document structure like I'm proposing, and solely is an optimization on pre-existing embeddings which mostly ignore any document structure.