2 ms·
When I want to segment words in some language, I usually check what Apache Lucene does. In this case, the Thai tokenizer [1] simply uses java.text.BreakIterator
by yorwba 8y ago
When I want to segment words in some language, I usually check what Apache Lucene does. In this case, the Thai tokenizer [1] simply uses java.text.BreakIterator [2] and hopes that Thai is supported.
[1] https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=blob;f=lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java;h=53f71699f2f9041bbdaab2d450e9315a6545cce9;hb=HEAD https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=...
[2] https://docs.oracle.com/javase/10/docs/api/java/text/BreakIterator.html https://docs.oracle.com/javase/10/docs/api/java/text/BreakIt...
- peteretep 8y agoFantastic tip, thanks