3 ms·
If you ever want to do something stuff with text efficiently (full text search and associated processing) I’d argue that it’s quite important.
by FridgeSeal 6y ago
If you ever want to do something stuff with text efficiently (full text search and associated processing) I’d argue that it’s quite important.
- Dylan16807 6y agoIf you want to do search, you need collation, and you can't use any standard encoding for that data.
- rectang 6y agoIt's important to distinguish between sorting in code point order and sorting according what a user would expect for their language. However, sorting in code point order (which is actually equivalent to sorting by memory comparison for UTF-8) is enough to build an inverted index data structure, commonly used for fulltext search. And just like FridgeSeal asserted, the memory footprint of the text representation has performance implications for such an application. (Source: I wrote a search engine library.)
- tsimionescu 6y agoWill that handle things like matching accented and unaccented characters? If I search for 'stefan' in a text that makes frequent references to 'Ștefan', will it correctly find those matches?