Grapheme segmentation in Unicode is actually kind of interesting and completely data-driven. I wrote my own implementation of the segmentation algorithm in Rust

Grapheme segmentation in Unicode is actually kind of interesting and completely data-driven. I wrote my own implementation of the segmentation algorithm in Rust recently¹ (because the existing segmentation crate didn’t have an interface that let me get the next grapheme cluster from a char iterator). It was fascinating to dig into the specifications and see how everything fits together. ⸻ 1. It will be published to crates.io sometime in the next month or so, there’s possible optimization work to be done, or at least, an alternate interface to the iterator to see whether the speed penalty I’m paying is an algorithmic problem or just an artifact of the trade-offs I have for my use case. I also ended up writing my own character class code as well and in benchmarking against the current popular crate got a 10x speedup,² so even if my segmentation code is useless to anyone but myself, I think that this code at least will be useful. 2. The other crate has a really inefficient implementation of looking up character classes where it ends up having to do a binary search in the data tables for many characters while I use a two-step table approach which lets me get a character’s class in O(1) time (and the tables are designed to be small enough that they’ll stick around in the CPU cache for lookups in a tight loop).