7 ms·
One way to improve the rigor of this research would be to embed each word in each language in a common latent space then look for clusters. But creating that l
by cadamsdotcom 1y ago
One way to improve the rigor of this research would be to embed each word in each language in a common latent space then look for clusters.
But creating that latent space and the corresponding embedding algorithm is hard in the first place. Today’s embedding models could be terrible for the fringe languages this research is about, and we wouldn’t know because we don’t know how to evaluate overall semantic accuracy.
Am I off piste here?
- ted_dunning 1y agoNot really off piste. But there is another big issue in that different languages (and especially North American indigenous languages) tend to have radically different notions of "word". As such, deciding what to embed is not easy.