3 ms·
Not exactly double / reverse meanings, but there are several setences that are traditionally used to test Japanese tokenizers. Most of them are tongue twisters,
by polm23 6y ago
Not exactly double / reverse meanings, but there are several setences that are traditionally used to test Japanese tokenizers. Most of them are tongue twisters, like this one:
すももももももももの内
"Japanese plums (sumomo) and peaches are both kinds of peaches"
In speech intonation would make the word boundaries here clear, but in writing it looks odd.
This isn't a tongue twister, but tokenizers often fail on it:
外国人参政権は難しい問題だ
"Voting rights for foreigners is a complex problem."
外国人 (foreigner) / 参政権 (gov participation rights) is the right tokenization, but a common error is to parse it as 外国 (foreign) / 人参 (carrot) / 政権 (political power).