3 ms·
Any information on which human languages it works with? Some languages like Thai can be challenging so I am wondering how general this is.
by mark_l_watson 4y ago
Any information on which human languages it works with? Some languages like Thai can be challenging so I am wondering how general this is.
- joelburget 4y agoIt works on all human languages, just inefficiently. I ran it over a sample I found on wikipedia: sample = "ฟองมันฟันหนู, ฟันหนูฟองมัน, ฝนทองฟองมัน" len(sample), len(enc.encode(sample)) This returns `39, 40` so it's just encoding one character at a time. It's probably like this for almost all non-English text.
- theragra 4y agoYeah, at least it does it with Russian