Be careful what you do with a Japanese tokenizer. Just because you got an ability(or I'd like to call it probability since it's not perfect) to tokenize Japanes

Be careful what you do with a Japanese tokenizer. Just because you got an ability(or I'd like to call it probability since it's not perfect) to tokenize Japanese doesn't mean you can process Japanese like the language you familiar with. Please, don't do clever things. It's better to just leave Japanese string as binary blob and treat it as is, passing left to right without any modification. It is my observation that the programmer of native Indo-European languages tend to have a delusion that their native language is so fundamental that any language processing they deemed necessary must be applied to the any and all languages in the world. Which cause the locale library in most programming languages tend to have a feature that's unfit for Japanese and it break the Japanese. Yes, the presence of the locale library break the Japanese language support. These locale libraries modify the string binary and break the Japanese encoding. You can't easily implement no-op local for Japanese because these modification are often cause by the fundamental common part of the library rather than language specific part. People have a good intention. They want to support multiple languages. but it's a delusion that passing the string binary to locale library magically solve the problem. The locale libraries were often written badly for Japanese so it break the Japanese. I would like to call it locale-tainted. Since The locale library is part of the standard library of the programming languages and many software relies on it. We can't just avoid using locale library without massive refactoring of the source code which is impossible given the available resource for localization. So if we, the unfortunate Japanese programmer, got a job of localizing locale-tainted existing software to Japanese, we have to come up some workarounds. Modifying the source code to remove the locale library usage is the best but often not possible. Hooking locale library to replace it with Japanese-safe version are dirty workaround but it requires to reimplement most of the text manipulation which cost many man-monthes and rigorous testing, Imagine implementing a printf/regex library to replace the standard library that satisfy all the use case in the wild. The worst workaround is let the stupid locale library do whatever the stupid Japanese breaking string binary transformation they damn want, then, reverse-transform after the string binary passing through the locale library to fix the mess. A mess that was unnecessary in the first place. I was involved in C++ Standard Committee but people out there are so delusional they think it's a good idea for a new standard library to depend on broken locale library because it help the localization. Those delusional people are also designing Unicode library now. They won't fart out a good library under such delusion. I lost all hope for this industry. So please. Don't introduce stupid unnecessary language processing to the language you don't understand. Make our job less horrible.