3 ms·

For more controlled splitting, I really like Unicode named characters classes[0] for more precise splitting and matching tasks. [0]: https://en.wikipedia.org/

For more controlled splitting, I really like Unicode named characters classes[0] for more precise splitting and matching tasks. [0]: https://en.wikipedia.org/wiki/Unicode_character_property#General_Category https://en.wikipedia.org/wiki/Unicode_character_property#Gen...

Rendello 1y ago
Given that encoded characters must have one and only one General_Category value, it might be too imprecise or arbitrary in some cases. If you ever need more power, it's worth browsing the other character properties Unicode exposes. For example, `Lu` (Uppercase_Letter) only covers some uppercase letters, whereas the `Uppercase` property covers all of them. --- For anyone that wants to learn more about specific Unicode stuff, the three big data sources are The Core Spec, the Unicode Technical Annexes (UAXs), and the Unicode Character Database itself (the database is a bunch of text files. There's an XML version now as well). For further reading on this specifically, it might be worth looking at: [Unicode Core Spec - Chapter 4: Character Properties] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/ https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... ├ [General Category] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124142 https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... └ [Properties for Text Boundaries] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124804 https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... [UAX #44 - Unicode Character Database (Technical Report)] https://www.unicode.org/reports/tr44/ https://www.unicode.org/reports/tr44/ ├ [General Category Values] https://www.unicode.org/reports/tr44/#General_Category_Values https://www.unicode.org/reports/tr44/#General_Category_Value... └ [Property Definitions] https://www.unicode.org/reports/tr44/#Property_Definitions https://www.unicode.org/reports/tr44/#Property_Definitions And, if you're brave and want to see the data itself (skim through UAX #44 first): [Unicode Character Database] https://www.unicode.org/Public/17.0.0/ucd/ https://www.unicode.org/Public/17.0.0/ucd/
- sundarurfriend 1y ago
  Unfortunately, even Properties aren't fully reliable. The Tamil pulli for eg., does not have the Alphabetic property despite being a central part of the Tamil alphabet, pretty much due to historical accident: other Indic languages have a similar looking Virama character that aren't alphabetic, and pulli (despite being a separate, unrelated character) was lumped in with those. When I tried to raise this in the mailing list and get it rectified, the response I got was pretty much that many properties need language-specific processing anyway, and shouldn't be relied upon fully, and so this wasn't worth fixing. [1] https://util.unicode.org/UnicodeJsps/character.jsp?a=0BCD&B1=Show https://util.unicode.org/UnicodeJsps/character.jsp?a=0BCD&B1...

Rendello 1y ago

Given that encoded characters must have one and only one General_Category value, it might be too imprecise or arbitrary in some cases. If you ever need more power, it's worth browsing the other character properties Unicode exposes. For example, `Lu` (Uppercase_Letter) only covers some uppercase letters, whereas the `Uppercase` property covers all of them. --- For anyone that wants to learn more about specific Unicode stuff, the three big data sources are The Core Spec, the Unicode Technical Annexes (UAXs), and the Unicode Character Database itself (the database is a bunch of text files. There's an XML version now as well). For further reading on this specifically, it might be worth looking at: [Unicode Core Spec - Chapter 4: Character Properties] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/ https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... ├ [General Category] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124142 https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... └ [Properties for Text Boundaries] https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124804 https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha... [UAX #44 - Unicode Character Database (Technical Report)] https://www.unicode.org/reports/tr44/ https://www.unicode.org/reports/tr44/ ├ [General Category Values] https://www.unicode.org/reports/tr44/#General_Category_Values https://www.unicode.org/reports/tr44/#General_Category_Value... └ [Property Definitions] https://www.unicode.org/reports/tr44/#Property_Definitions https://www.unicode.org/reports/tr44/#Property_Definitions And, if you're brave and want to see the data itself (skim through UAX #44 first): [Unicode Character Database] https://www.unicode.org/Public/17.0.0/ucd/ https://www.unicode.org/Public/17.0.0/ucd/

sundarurfriend 1y ago
Unfortunately, even Properties aren't fully reliable. The Tamil pulli for eg., does not have the Alphabetic property despite being a central part of the Tamil alphabet, pretty much due to historical accident: other Indic languages have a similar looking Virama character that aren't alphabetic, and pulli (despite being a separate, unrelated character) was lumped in with those. When I tried to raise this in the mailing list and get it rectified, the response I got was pretty much that many properties need language-specific processing anyway, and shouldn't be relied upon fully, and so this wasn't worth fixing. [1] https://util.unicode.org/UnicodeJsps/character.jsp?a=0BCD&B1=Show https://util.unicode.org/UnicodeJsps/character.jsp?a=0BCD&B1...