3 ms·
I'm glad I could make you curious about this. I will gladly expand on the documentation around the language model and querying as soon as I can. In broad terms
by misterman0 8y ago
I'm glad I could make you curious about this. I will gladly expand on the documentation around the language model and querying as soon as I can.
In broad terms: its a 16-bit vector space in which you can encode anything you like. I have chosen to encode phrases and words as bags-of-characters. This separates terms from each other enough that they can be searched for reliably (in almost all cases).
Terms that share a character have vectors that intersect one another and we can measure the cos angle between them. That's the score.
That is represented as a binary tree.
A scan in the tree gives you the closest match and an address into a file on disk; a list of document IDs.
At query time boolean logic is used on the result (document ID list) from each query clause (AND/OR/NOT key:value).
I'll write something up.
- ryanfox 8y agoCould the bags-of-characters approach cause issues with anagrams having the same vector? It would be surprising (to me) to get the same results for e.g. "strange" and "garnets".
- misterman0 8y agoYes this model could cause issues such as the one you describe. With phrase queries/multi-token queries this becomes less of a problem. Phrases aren't anagrams that often. A secondary index might become needed with the most popular terms, to resolve which anagram is the right one.