4 ms·
Thanks for the explanation. Very interesting. I notice that that particular tokenization deviates from the morphemic divisions in several cases, including ‘dec
by tkgally 1y ago
Thanks for the explanation. Very interesting.
I notice that that particular tokenization deviates from the morphemic divisions in several cases, including ‘dec-entral-ization’, ‘食べ-させ-られた-くな-かった’, and ‘面白-くな-さ-そうだ.’ ‘dec’ and ‘entral’ are not morphemes, nor is ‘くな.’