3 ms·
TLDR: they implemented case folding with a lot more SIMD via autovectorization. > almost every fold preserves the UTF-8 length or shrinks it, but two outliers
by inigyou 3mo ago
TLDR: they implemented case folding with a lot more SIMD via autovectorization.
> almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ)
Fix this by reversing it. Fold ⱥ to Ⱥ instead of the other way around. The search index won't only consist of lowercase characters any more, but that never mattered.
- PaulDavisThe1st 3mo ago[dead]
- persedes 3mo agoThis is a nice follow up to the other SIMD article that was posted here a week or so ago hah.
- Georgelemental 3mo ago> The search index won't only consist of lowercase characters any more This isn't the case anyway. Unicode case-folding has a few lowercase-to-uppercase mappings, e.g. Cherokee
- gwking 3mo agoI wonder if making the index uppercase is strictly better in this sense, or if both upper and lowercase have chars that take more bytes.
- goalieca 3mo agoCurious if his isn’t some Over-optimization since how often do those characters ever come up.
- RossBencina 3mo agoIf you can kill-off the extra buffer allocation code path that's got to be a win.
- RossBencina 3mo ago+1 great idea. I get the impression that some of my sibling comments thought that you meant fold all code points in the opposite direction, but it it's clear you mean change the fold direction for only those two code points.
- masklinn 3mo agoIt does not look like the author posted this here since it was posted on the blog last week. So would probably be a better idea to suggest / recommend this at https://github.com/github/rust-gems/tree/main/crates/casefold https://github.com/github/rust-gems/tree/main/crates/casefol...
- inigyou 3mo agoDon't see why I should go out of my way to improve Microsoft shareholder value.