3 ms·
shameless plug for Tatoeba.org I used to code "tatodetect" to do this, https://github.com/allan-simon/Tatodetect https://github.com/allan-simon/Tatodetect it'
by allan_s 3y ago
shameless plug
for Tatoeba.org I used to code "tatodetect" to do this, https://github.com/allan-simon/Tatodetect https://github.com/allan-simon/Tatodetect
it's was my first enlighting that with simple math and stats you can achieve pretty good result without ressorting to "machine learning"
basically the idea was
1. use tatoeba to generate the trigram count
2. remove low frequency trigram
3. apply a bonus score for "uniq" trigram that appears in other languages (it helped to differentiate very similar languages like spanish/french or Chinese dialects)
then you simply sum the trigram with their frequency ponderation and voilà
Tatodetect should differentiate 200+ languages and dialect, regarding of their script (alphabet, sinograms etc. )