4 ms·
>Apple's word segmentation Unless they changed it, it's probably similar to CFStringTokenizer which used ICU Boundary Analysis (and maybe mecab for Japanese).
by krackers 5y ago
>Apple's word segmentation
Unless they changed it, it's probably similar to CFStringTokenizer which used ICU Boundary Analysis (and maybe mecab for Japanese).
- peterburkimsher 5y agoThank you! The ICU Boundary Analysis documentation says it uses a dictionary to split Chinese, Japanese, Thai or Khmer. https://unicode-org.github.io/icu/userguide/boundaryanalysis/ https://unicode-org.github.io/icu/userguide/boundaryanalysis... Is that the same as the macOS dictionary being parsed here? It seems like a pretty big file to grep every time!
- krackers 5y agoNo, the ICU dictionaries are seen at: https://github.com/unicode-org/icu/tree/main/icu4c/source/data/brkitr/dictionaries https://github.com/unicode-org/icu/tree/main/icu4c/source/da... I assume at compile time it's converted to a more efficient query format