Tokenizing strings is usually not something that happens unnecessarily, but it's required for the implementation of e.g. keyword search. So it's not something p

Tokenizing strings is usually not something that happens unnecessarily, but it's required for the implementation of e.g. keyword search. So it's not something people can just avoid doing. > These locale libraries modify the string binary and break the Japanese encoding. You can't easily implement no-op local for Japanese because these modification are often cause by the fundamental common part of the library rather than language specific part. Can you give an example of a situation where that happens? Is it an issue with e.g Shift-JIS encoding vs. Unicode?