It isn't made clear in the repo but what this is doing [1] is a small amount of code to glue SQLite's tokenizer API to the Rust unicode-segmentation crate [2] w

It isn't made clear in the repo but what this is doing [1] is a small amount of code to glue SQLite's tokenizer API to the Rust unicode-segmentation crate [2] which in turn is using the TR-29 algorithm with generated tables [3]. [1] https://github.com/signalapp/Signal-FTS5-Extension/blob/main/Cargo.toml https://github.com/signalapp/Signal-FTS5-Extension/blob/main... [2] https://unicode-rs.github.io/unicode-segmentation/unicode_segmentation/index.html https://unicode-rs.github.io/unicode-segmentation/unicode_se... [3] https://github.com/unicode-rs/unicode-segmentation/blob/master/src/tables.rs https://github.com/unicode-rs/unicode-segmentation/blob/mast... International Components for Unicode is the reference library for this and other Unicode algorithms. https://en.wikipedia.org/wiki/International_Components_for_Unicode https://en.wikipedia.org/wiki/International_Components_for_U...