4 ms·

So what is the most accurate way to tokenize CJK text?

• edflsafoiewq 6y ago
  By hand using a native speaker.
  • tobyhinloopen 6y ago
    Is there an API for that?
    • cferr 6y ago
      Yes, but it hasn't been written yet.
      • edjroot 6y ago
        You joke, but... https://blog.ycombinator.com/scale/ https://blog.ycombinator.com/scale/
    • syrrim 6y ago
      you could try something with mturk, I'm sure.
  • 1024core 6y ago
    Surely someone must've come up with rules for it by now?
    • earthboundkid 6y ago
      Are there rules (as opposed to heuristics) for splitting up English, like “words—separated haphazardly—by em–dashes?”