3 ms·
That is _awesome_. I've been wanting to do exactly this for Thai Wikipedia for quite a while, but you have some serious issues with identifying words in Thai.
by peteretep 8y ago
That is _awesome_. I've been wanting to do exactly this for Thai Wikipedia for quite a while, but you have some serious issues with identifying words in Thai.
- yorwba 8y agoWhen I want to segment words in some language, I usually check what Apache Lucene does. In this case, the Thai tokenizer [1] simply uses java.text.BreakIterator [2] and hopes that Thai is supported. [1] https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=blob;f=lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java;h=53f71699f2f9041bbdaab2d450e9315a6545cce9;hb=HEAD https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=... [2] https://docs.oracle.com/javase/10/docs/api/java/text/BreakIterator.html https://docs.oracle.com/javase/10/docs/api/java/text/BreakIt...
- peteretep 8y agoFantastic tip, thanks