7 ms·
> Even if they're rare, you're effectively hamstringing anyone who uses a language that includes four byte characters or emoji. I don't think so. Suppose I am
by tialaramex 5y ago
> Even if they're rare, you're effectively hamstringing anyone who uses a language that includes four byte characters or emoji.
I don't think so. Suppose I am looking for some code that I remember has a comment with the shark emoji U+1F988 in double quotes.
So the trigrams this backend is searching for are 0x22F09F and 0xA68822.
The first trigram is only going to appear in documents quoting an emoji animal. Yes it will match kangaroos or indeed crabs in a few Rust crates, but it will entirely skip every program that doesn't have an emoji animal in quotes. The second trigram only matches documents with certain 3 or 4-byte Unicode scalars (such as an emoji) at the end of a quote. It will not, for example, match most of those Rust documents talking about crabs since crabs do not end with the bytes 0xA6 0x88.
So chances are you went from "Oh no, a 4 byte character, surely the worst case" to only the documents you actually wanted before the main algorithm even began running.
> But moreover, the prefix of many of your trigrams is just going to be surrogate pair bytes.
I assume this is just a brain misfiring, this is obviously not how UTF-8 works.
- bastawhiz 5y agoThat works great for emojis and terribly for Japanese, Chinese, Korean, Thai, etc. where each character is a three byte codepoint. Then you're literally just building a reverse index of individual characters. If this was effective, they wouldn't be using trigrams in the first place: it's a much simpler system to implement. Sure, most languages use Latin characters in keywords, but that's unhelpful for comments.