4 ms·

Would appreciate it if someone can ELI5 how they assembled a dataset for this task, without relying on english. I am not able to figure out their LASER 2.0 syst

Would appreciate it if someone can ELI5 how they assembled a dataset for this task, without relying on english. I am not able to figure out their LASER 2.0 system

orange3xchicken 6y ago
They do rely on english & spanish translations to construct the dataset. They have existing tools to jointly embed sentences from multiple languages (laser). These models are trained using parallel corpora involving either english or spanish on a translation task. Using these models and the joint embeddings they produce, fb can mine the web for new pairs by roughly identifying whether two sentences in two different languages correspond to a translation pair.
- whimsicalism 6y ago
  Of course, the problem with these "synthetic" datasets are when the translation models overfit to the problems in these multilingual sentence encoders/embedders. Still, it lets you massively increase the amount of data you're training on, which is generally worth it.
- onurcel 6y ago
  Exactly. I work on Laser2, the approach is the same as Laser [0], Laser2 performs better on some low resource languages. A more ELI5 explanation would be something like this: Laser is an encoder/decoder architecture trained as a translation task from language X to english/spanish, with the particularity of having only one vector between the encoder and decoder. Once this system is trained (with public translation datasets), the vector that the encoder gives you for an input sentence represents the "meaning" of that sentence, since the decoder relies only on that information to generate a translation. And this vector representation is the same for any language the system was trained on. So we use that system to mine data from commoncrawl: giving any language pair (say romanian-nepali), having two vectors close to each other in that latent space means the sentences have the same meaning. We use fastText's language classifier [1] to filter from commoncrawl, compute the vector representations with Laser, and find close vectors thanks to Faiss [2]. [0] https://engineering.fb.com/ai-research/laser-multilingual-sentence-embeddings/ https://engineering.fb.com/ai-research/laser-multilingual-se... [1] https://fasttext.cc/docs/en/language-identification.html https://fasttext.cc/docs/en/language-identification.html [2] https://github.com/facebookresearch/faiss https://github.com/facebookresearch/faiss
  - yorwba 6y ago
    > giving any language pair (say romanian-nepali), having two vectors close to each other in that latent space means the sentences have the same meaning. That's the goal, but have you asked any Romanian-Nepali bilinguals how well it works? (I realize those might be hard to come by.) I had a look at some of the language pairs in CCMatrix, and I noticed that the highest-confidence matches for some of them (English-Chinese, English-Korean) include a lot of quotations from old religious texts. That wouldn't be a problem if they were actually the same quotations, but it looked more like the model managed to identify archaic language and then got overly confident that two archaic sentences must have the same meaning. I wonder whether there's been a human evaluation of the mined training data, or whether you rely on catching any problems downstream when you measure the BLEU of the trained model.
joshvm 6y ago
In theory you could do this with books? Sentences in translations should still be the same, and you could add some heuristics to identify paragraphs, quote marks, etc. You might be wrong occasionally, but doing the extraction per chapter or per paragraph would mitigate it.
- asutekku 6y ago
  Only in theory. Sentences are hardly the same in other languages as the books are not one-to-one translations and a sentence in one language might be three in another. Even the paragraphs might not be the same in asian languages.
  - joshvm 6y ago
    It depends on the language pairings. Take a look at Don Quixote in English and Spanish, they're very comparable at the sentence level (at least this translation). Also if course it depends on the artistic license of the translation. Maybe it's something you could do with a bit of human intervention to sync up the two texts. http://www.cervantesvirtual.com/obra-visor/el-ingenioso-hidalgo-don-quijote-de-la-mancha-6/html/ http://www.cervantesvirtual.com/obra-visor/el-ingenioso-hida... https://www.fulltextarchive.com/page/Don-Quixotex5770/ https://www.fulltextarchive.com/page/Don-Quixotex5770/
ctl0 6y ago
Really disappointed to see technical discussion is buried under personal matters and complains.

Would appreciate it if someone can ELI5 how they assembled a dataset for this task, without relying on english. I am not able to figure out their LASER 2.0 system

orange3xchicken 6y ago

They do rely on english & spanish translations to construct the dataset. They have existing tools to jointly embed sentences from multiple languages (laser). These models are trained using parallel corpora involving either english or spanish on a translation task. Using these models and the joint embeddings they produce, fb can mine the web for new pairs by roughly identifying whether two sentences in two different languages correspond to a translation pair.

whimsicalism 6y ago
Of course, the problem with these "synthetic" datasets are when the translation models overfit to the problems in these multilingual sentence encoders/embedders. Still, it lets you massively increase the amount of data you're training on, which is generally worth it.
onurcel 6y ago
Exactly. I work on Laser2, the approach is the same as Laser [0], Laser2 performs better on some low resource languages. A more ELI5 explanation would be something like this: Laser is an encoder/decoder architecture trained as a translation task from language X to english/spanish, with the particularity of having only one vector between the encoder and decoder. Once this system is trained (with public translation datasets), the vector that the encoder gives you for an input sentence represents the "meaning" of that sentence, since the decoder relies only on that information to generate a translation. And this vector representation is the same for any language the system was trained on. So we use that system to mine data from commoncrawl: giving any language pair (say romanian-nepali), having two vectors close to each other in that latent space means the sentences have the same meaning. We use fastText's language classifier [1] to filter from commoncrawl, compute the vector representations with Laser, and find close vectors thanks to Faiss [2]. [0] https://engineering.fb.com/ai-research/laser-multilingual-sentence-embeddings/ https://engineering.fb.com/ai-research/laser-multilingual-se... [1] https://fasttext.cc/docs/en/language-identification.html https://fasttext.cc/docs/en/language-identification.html [2] https://github.com/facebookresearch/faiss https://github.com/facebookresearch/faiss
- yorwba 6y ago
  > giving any language pair (say romanian-nepali), having two vectors close to each other in that latent space means the sentences have the same meaning. That's the goal, but have you asked any Romanian-Nepali bilinguals how well it works? (I realize those might be hard to come by.) I had a look at some of the language pairs in CCMatrix, and I noticed that the highest-confidence matches for some of them (English-Chinese, English-Korean) include a lot of quotations from old religious texts. That wouldn't be a problem if they were actually the same quotations, but it looked more like the model managed to identify archaic language and then got overly confident that two archaic sentences must have the same meaning. I wonder whether there's been a human evaluation of the mined training data, or whether you rely on catching any problems downstream when you measure the BLEU of the trained model.

joshvm 6y ago

In theory you could do this with books? Sentences in translations should still be the same, and you could add some heuristics to identify paragraphs, quote marks, etc. You might be wrong occasionally, but doing the extraction per chapter or per paragraph would mitigate it.

asutekku 6y ago
Only in theory. Sentences are hardly the same in other languages as the books are not one-to-one translations and a sentence in one language might be three in another. Even the paragraphs might not be the same in asian languages.
- joshvm 6y ago
  It depends on the language pairings. Take a look at Don Quixote in English and Spanish, they're very comparable at the sentence level (at least this translation). Also if course it depends on the artistic license of the translation. Maybe it's something you could do with a bit of human intervention to sync up the two texts. http://www.cervantesvirtual.com/obra-visor/el-ingenioso-hidalgo-don-quijote-de-la-mancha-6/html/ http://www.cervantesvirtual.com/obra-visor/el-ingenioso-hida... https://www.fulltextarchive.com/page/Don-Quixotex5770/ https://www.fulltextarchive.com/page/Don-Quixotex5770/

ctl0 6y ago

Really disappointed to see technical discussion is buried under personal matters and complains.