8 ms·

Amazon releases 51-language dataset for language understanding

166 points by reckel 4y ago

jgmf 4y ago
Hi everyone, Jack FitzGerald here from the Alexa team who created this new dataset, the corresponding code, the leaderboard, etc. I'd be happy to answer any questions you might have, and I'll jump in with the other comments already here. Thank you.
- lvl102 4y ago
  Jack, this project and competition remind me of the Netflix Prize so much! Can’t wait till we see what people can do with this dataset. Congrats on the launch!
  - jgmf 4y ago
    Thank you!
timr 4y ago
This is a dataset of ~19,500 unique phrases in 51 languages. The language coverage is certainly broad, but a 20k phrase set is not huge, in terms of NLU models.
- disgruntledphd2 4y ago
  Yeah, definitely, but the goal (transfer learning across languages) is super good, and this seems like a good dataset for that purpose (though I suspect it's entirely insufficient to learn an NLP model from scratch).
  - cperiz 4y ago
    The paper provides baseline results on how some pre-trained open source models will perform, when fine-tuned on this dataset. https://arxiv.org/abs/2204.08582 https://arxiv.org/abs/2204.08582
mmastrac 4y ago
The paper [1] has an interesting discussion of european famliar/formal pronous: Many languages encode different levels of polite- ness through their use of pronouns. Many European languages distinguish between “familiar” and “for- mal” pronouns, with the “formal” pronouns often morphologically identical to a plural. In French, the second-person singular “tu” is used between friends, while the second-person plural “vous” is used when speaking to a group, or to an individual of higher social rank (such as an employee to a manager). These politeness systems are very heav- ily influenced by social context, and the MASSIVE dataset gives us a chance to see how people adapt their language when speaking to a virtual assistant instead of another human. https://arxiv.org/pdf/2204.08582.pdf https://arxiv.org/pdf/2204.08582.pdf
- nicoburns 4y ago
  Historically those languages have included English too. "you" and "ye" being the formal/plural form (and indeed "you" can still be used to refer to either an individual or a group) with "thou" and "thee" being the now mostly (although not entirely) disused informal version.
- Zababa 4y ago
  > while the second-person plural “vous” is used when speaking to a group, or to an individual of higher social rank (such as an employee to a manager). From my experience "vous" is not always about social rank, but often about "distance". If you go to a supermarket, you'll use "vous" with the employees and they'll use it with you. Most of the time when I use "vous" the other person will also use it. "vous"/"tu" cases are more rare.
rsstack 4y ago
The sentences in Hebrew are hilarious. Either fixated on Omer Adam[1] or very formal language that I don't think people would actually use. Or formal sentences asking about Omer Adam's greatest hits. https://en.wikipedia.org/wiki/Omer_Adam https://en.wikipedia.org/wiki/Omer_Adam
capableweb 4y ago
Are the sentences in the dataset supposed to be correct? (specifically, the `.utt` key) Seeing some strange results in multiple languages, that don't look correct. Even some English ones look incorrect (but not a native English speaker, so maybe I'm wrong). Here are some examples: - "i want to play that music one again" - "what's that the album is current music from" - "which alarms do i have" Maybe I'm misunderstanding the purpose of the dataset...
- bagels 4y ago
  These seem like it probably things people said to Alexa. There is probably a combination of speech recognition errors and actual ungrammatical speech happening. Spoken word is often ungrammatical.
  - 6gvONxR4sf7o 4y ago
    It’s kinda wild to think about how incomplete written language is compared to spoken language. Punctuation does a lot of heavy lifting. If I write “I think well the I mean what album is this the song thats playing from” it takes some work to parse, but that’s what an ASR system has to do. A more human transcription might be “I think-- well the-- I mean, what album is this (the song that’s playing) from?”
  - usr1106 4y ago
    > These seem like it probably things people said to Alexa. That could be an explanation. But some people made fun of Alexa and there was no human to filter out the nonsense: In German I see: "putzen ist gut staub ist so schlecht mach jetzt deinen zauber sauber mein teppich" (cleaning is good dust is so bad make your miracle cleanly my carpet)
  - numpad0 4y ago
    On top of those, these are translated ~nonsense~ text that aren’t suited for translation. Most likely a list of English texts are handed to translators without intended use cases or extra contexts well communicated to them. Some of texts are only correct in literal sense only or acceptably only in translated literature only, and still passing reviews. Edit: “nonsense” was way too harsh of a word
tgv 4y ago
Translation quality is not great. A sentence like "maak de verlichting een beetje meer warm hier" is absolutely not idiomatic Dutch, and "wat is de tijdt in de andere tijdzones" contains an easily caught spelling error. "laat me mijn wekkers zien" is quite ridiculous; it means: "show me my (alarm) clocks". "ontvang me updates van bram´s facebook van het weekend" is ungrammatical. Well, not great.
- 1f60c 4y ago
  "Laat me mijn wekkers zien" is indeed a bit odd, but it makes sense if you think of it in the context of a voice assistant that displays information on a screen.
  - usr1106 4y ago
    In German I see: "putzen ist gut staub ist so schlecht mach jetzt deinen zauber sauber mein teppich" This is complete nonsense. Mark Twain could have written it to make fun of the German language.
    - jgmf 4y ago
      Ah, that's unfortunate to hear. In this case I see grammar_score ranging from 2 (Some errors (the meaning can be understood but it doesn't sound natural in your language)) to 4 (Perfect)--looks like one of the raters was too generous with a rating of 4. I think this feedback is helping me to realize that we should be more explicit about our philosophy of keeping some of the noise in the dataset and allowing people to filter based on judgments. That said, people do say some strange and ungrammatical things to virtual assistants (not saying this example is representative per se), so it's nice to include some of the odd ones. Thanks!
      - usr1106 4y ago
        Thanks for following up! I guess what happened is a user of your system tried to be funny. In this utterance there are 3 short fully grammatical sentences. And a 4th one which is not very grammatical but fully understandable, commanding the device to clean the carpet :) - Cleaning is good. - Dust is so bad. - Make your miracle. - Cleanly my carpet. [1] [1] original is an ungrammatical imperative Some (more) human review would have been needed. The Finnish samples are full of very weird utterances, too. Some like people might have written over 50 years ago, but nobody would speak like this. Maybe they were reviewed on Mechanical turk with moderate requirements / payment? Well, you get what you pay for...
marcodiego 4y ago
Anyone knows a good open source translator that can be ran locally?
- capableweb 4y ago
  Argos Translate fits with those requirements, but its translation quality is not great. I'd love to see something like DeepL (that has really good translation quality) running locally, but I think that's a pipe-dream for now.
- 6gvONxR4sf7o 4y ago
  If you know your way around ML, huggingface has some solid models. I’ve used MarianMT before and it was decent enough.
- unhammer 4y ago
  https://translatelocally.com/ https://translatelocally.com/ is a nice gui around marian/bergamot. So far not very many bundled pairs, though I would guess any of the models from https://github.com/Helsinki-NLP/Opus-MT-train/tree/master/models https://github.com/Helsinki-NLP/Opus-MT-train/tree/master/mo... and https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/results/tatoeba-results-all.md https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/maste... should be usable. There is also Apertium, a rule-based system which is very good for some closely-related pairs that have had a lot of work put into them (especially translation between Romance languages, e.g. Spanish→Catalan, and Norwegian Bokmål→Nynorsk), and the only OK translator for some lesser-resourced languages (e.g. Northern Saami→Norwegian Bokmål), but very underdeveloped for anything to/from English (it feels a bit pointless writing rules for English where there is so much available data; RBMT shines where there's not enough available data, ie. most of the languages of the world). It's packaged for most distros: https://packages.debian.org/search?suite=sid&arch=any&searchon=names&keywords=apertium- https://packages.debian.org/search?suite=sid&arch=any&search...
deleted 4y ago
[deleted]
abadadria 4y ago
lifeisstillgood 4y ago
Just today I was looking at how to use Alexa to help my son with his school spanish. I am not convinced it is perfect but it looks like being part of the puzzle.
deleted 4y ago
[deleted]
numpad0 4y ago
Funny ones I could find, in Japanese: - unanimous grammar score: 4, spelling score: 2: 3046: "今どうしても中華料理が食べたいのでテイクアウトで注文させてください": "I really want to eat Chinese right now, is it okay if I order some delivered(asking the boss nicely)" , (4,2),(4,2),(4,2) 13367: "単語を定義するとき文で明確にします": "[this will be]clarified in sentence when defining the word", (4,2),(4,2),(4,2) 13258: "ビーエムダブリューのマイレージ": "Mileage [reward program] for BMW", (4,2),(4,2),(4,2) 13986: "トップモデルの車": "Top-style car" or "car of top [beauty] model", (4,2),(4,2),(4,2) 14556: "この会社の株価を更新してください": "Please update stock price for this company", (4,2),(4,2),(4,2) 14592: "スマホのサーキットは何なのか教えて": "Tell me what is the smartphone raceway", (4,2),(4,2),(4,2) 15414: "マネージャーが必要だ": "There will have to be a manager", (4,2),(4,2),(4,2) - 2/3 grammar score: 4, spelling score: 2: 6235: "素晴らしい映画を見たのでコピーがリリースされたら予約してください": "Because I watched a magnificent movie, please reserve in case a knockoff is let go", (4,2),(4,2),(3,2) 14401: "あなたが私に探した最後のことを教えてください": "Tell me the last thing you have searched of me", (2,2),(4,2),(4,2) 14407: "感じることができるの": "Are you able to sense that", (4,2),(2,2),(4,2) - less than above 14744: "ブロイラーは何ですか、どのように使えばいいですか": "Broiler [meat] is what, how may I use it(for you/for community)", (4,2),(4,1),(3,2) 15456: "ユナイテッド航空にあなたが私の荷物を無くしたのに怒るとツイートをしたいけど": "Though I want to tweet to [Mr.]United Airlines that you become enraged despite you losing my luggage", (4,2),(4,2),(2,2)
- numpad0 4y ago
  I've since given a bit more thoughts on this - Part B question 3 and 4, from Fig.3 on the paper, asks only whether the sentence sound natural and (technically) correct, without specifying context, that the sentence is supposed to be an utterance spoken to a voice assistant. From my own comment above: 3046: "I really want to eat Chinese right now, is it okay if I order some delivered" 14556: "Please update stock price for this company" 15414: "There will have to be a manager" These are all context errors. What is not widely acknowledged(as I'd like...) is that translations for short sentences like "I need to see the manager" are situational, beyond depending on tones and politeness. Consider following: - I have complaints as a customer, and *I need to see the manager*. - I need to escalate this issue, and *I need to see the manager*. - You asked what I am about to do, and *I need to see the manager*. - "The manager" is a dashboard app, and *I need to see the manager*. I won't list my translations because I've already dumped way too much CJK characters here, but each of above four should turn up differently against each others from organic translators(especially the last one).
euroderf 4y ago
I poked around and didn't see a list of the 51 languages. Maybe I'm stupid.
- jgmf 4y ago
  Sorry, we didn't have enough space to fit it on the blog post. You can find the list of languages on page 4 of our paper: https://arxiv.org/pdf/2204.08582.pdf https://arxiv.org/pdf/2204.08582.pdf I'll add this to our github repo, too. Thanks!
  - euroderf 4y ago
    Cheers! -Curious
amilios 4y ago
Funny ones I found in Greek (my native lang): "και έπεσε σκοτάδι" => "and darkness fell" (???) "είναι βρώμικα εδώ κάνε λίγο φασαρία" => "it's dirty here, make a fuss" But they're actually mostly pretty decent.
- jgmf 4y ago
  "and darkness fell" Yes, this one caused us some consternation :).
Gelob 4y ago
where are all the "shut up, alexa" and "fuck you alexa" phrases?

166 points by reckel 4y ago

jgmf 4y ago

Hi everyone, Jack FitzGerald here from the Alexa team who created this new dataset, the corresponding code, the leaderboard, etc. I'd be happy to answer any questions you might have, and I'll jump in with the other comments already here. Thank you.

lvl102 4y ago
Jack, this project and competition remind me of the Netflix Prize so much! Can’t wait till we see what people can do with this dataset. Congrats on the launch!
- jgmf 4y ago
  Thank you!

timr 4y ago

This is a dataset of ~19,500 unique phrases in 51 languages. The language coverage is certainly broad, but a 20k phrase set is not huge, in terms of NLU models.

disgruntledphd2 4y ago
Yeah, definitely, but the goal (transfer learning across languages) is super good, and this seems like a good dataset for that purpose (though I suspect it's entirely insufficient to learn an NLP model from scratch).
- cperiz 4y ago
  The paper provides baseline results on how some pre-trained open source models will perform, when fine-tuned on this dataset. https://arxiv.org/abs/2204.08582 https://arxiv.org/abs/2204.08582

mmastrac 4y ago

The paper [1] has an interesting discussion of european famliar/formal pronous: Many languages encode different levels of polite- ness through their use of pronouns. Many European languages distinguish between “familiar” and “for- mal” pronouns, with the “formal” pronouns often morphologically identical to a plural. In French, the second-person singular “tu” is used between friends, while the second-person plural “vous” is used when speaking to a group, or to an individual of higher social rank (such as an employee to a manager). These politeness systems are very heav- ily influenced by social context, and the MASSIVE dataset gives us a chance to see how people adapt their language when speaking to a virtual assistant instead of another human. https://arxiv.org/pdf/2204.08582.pdf https://arxiv.org/pdf/2204.08582.pdf

nicoburns 4y ago
Historically those languages have included English too. "you" and "ye" being the formal/plural form (and indeed "you" can still be used to refer to either an individual or a group) with "thou" and "thee" being the now mostly (although not entirely) disused informal version.
Zababa 4y ago
> while the second-person plural “vous” is used when speaking to a group, or to an individual of higher social rank (such as an employee to a manager). From my experience "vous" is not always about social rank, but often about "distance". If you go to a supermarket, you'll use "vous" with the employees and they'll use it with you. Most of the time when I use "vous" the other person will also use it. "vous"/"tu" cases are more rare.

rsstack 4y ago

The sentences in Hebrew are hilarious. Either fixated on Omer Adam[1] or very formal language that I don't think people would actually use. Or formal sentences asking about Omer Adam's greatest hits. https://en.wikipedia.org/wiki/Omer_Adam https://en.wikipedia.org/wiki/Omer_Adam

capableweb 4y ago

Are the sentences in the dataset supposed to be correct? (specifically, the `.utt` key) Seeing some strange results in multiple languages, that don't look correct. Even some English ones look incorrect (but not a native English speaker, so maybe I'm wrong). Here are some examples: - "i want to play that music one again" - "what's that the album is current music from" - "which alarms do i have" Maybe I'm misunderstanding the purpose of the dataset...

bagels 4y ago
These seem like it probably things people said to Alexa. There is probably a combination of speech recognition errors and actual ungrammatical speech happening. Spoken word is often ungrammatical.
- 6gvONxR4sf7o 4y ago
  It’s kinda wild to think about how incomplete written language is compared to spoken language. Punctuation does a lot of heavy lifting. If I write “I think well the I mean what album is this the song thats playing from” it takes some work to parse, but that’s what an ASR system has to do. A more human transcription might be “I think-- well the-- I mean, what album is this (the song that’s playing) from?”
- usr1106 4y ago
  > These seem like it probably things people said to Alexa. That could be an explanation. But some people made fun of Alexa and there was no human to filter out the nonsense: In German I see: "putzen ist gut staub ist so schlecht mach jetzt deinen zauber sauber mein teppich" (cleaning is good dust is so bad make your miracle cleanly my carpet)
- numpad0 4y ago
  On top of those, these are translated ~nonsense~ text that aren’t suited for translation. Most likely a list of English texts are handed to translators without intended use cases or extra contexts well communicated to them. Some of texts are only correct in literal sense only or acceptably only in translated literature only, and still passing reviews. Edit: “nonsense” was way too harsh of a word

tgv 4y ago

Translation quality is not great. A sentence like "maak de verlichting een beetje meer warm hier" is absolutely not idiomatic Dutch, and "wat is de tijdt in de andere tijdzones" contains an easily caught spelling error. "laat me mijn wekkers zien" is quite ridiculous; it means: "show me my (alarm) clocks". "ontvang me updates van bram´s facebook van het weekend" is ungrammatical. Well, not great.

1f60c 4y ago
"Laat me mijn wekkers zien" is indeed a bit odd, but it makes sense if you think of it in the context of a voice assistant that displays information on a screen.
- usr1106 4y ago
  In German I see: "putzen ist gut staub ist so schlecht mach jetzt deinen zauber sauber mein teppich" This is complete nonsense. Mark Twain could have written it to make fun of the German language.
  - jgmf 4y ago
    Ah, that's unfortunate to hear. In this case I see grammar_score ranging from 2 (Some errors (the meaning can be understood but it doesn't sound natural in your language)) to 4 (Perfect)--looks like one of the raters was too generous with a rating of 4. I think this feedback is helping me to realize that we should be more explicit about our philosophy of keeping some of the noise in the dataset and allowing people to filter based on judgments. That said, people do say some strange and ungrammatical things to virtual assistants (not saying this example is representative per se), so it's nice to include some of the odd ones. Thanks!
    - usr1106 4y ago
      Thanks for following up! I guess what happened is a user of your system tried to be funny. In this utterance there are 3 short fully grammatical sentences. And a 4th one which is not very grammatical but fully understandable, commanding the device to clean the carpet :) - Cleaning is good. - Dust is so bad. - Make your miracle. - Cleanly my carpet. [1] [1] original is an ungrammatical imperative Some (more) human review would have been needed. The Finnish samples are full of very weird utterances, too. Some like people might have written over 50 years ago, but nobody would speak like this. Maybe they were reviewed on Mechanical turk with moderate requirements / payment? Well, you get what you pay for...

marcodiego 4y ago

Anyone knows a good open source translator that can be ran locally?

capableweb 4y ago
Argos Translate fits with those requirements, but its translation quality is not great. I'd love to see something like DeepL (that has really good translation quality) running locally, but I think that's a pipe-dream for now.
6gvONxR4sf7o 4y ago
If you know your way around ML, huggingface has some solid models. I’ve used MarianMT before and it was decent enough.
unhammer 4y ago
https://translatelocally.com/ https://translatelocally.com/ is a nice gui around marian/bergamot. So far not very many bundled pairs, though I would guess any of the models from https://github.com/Helsinki-NLP/Opus-MT-train/tree/master/models https://github.com/Helsinki-NLP/Opus-MT-train/tree/master/mo... and https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/results/tatoeba-results-all.md https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/maste... should be usable. There is also Apertium, a rule-based system which is very good for some closely-related pairs that have had a lot of work put into them (especially translation between Romance languages, e.g. Spanish→Catalan, and Norwegian Bokmål→Nynorsk), and the only OK translator for some lesser-resourced languages (e.g. Northern Saami→Norwegian Bokmål), but very underdeveloped for anything to/from English (it feels a bit pointless writing rules for English where there is so much available data; RBMT shines where there's not enough available data, ie. most of the languages of the world). It's packaged for most distros: https://packages.debian.org/search?suite=sid&arch=any&searchon=names&keywords=apertium- https://packages.debian.org/search?suite=sid&arch=any&search...

deleted 4y ago

[deleted]

abadadria 4y ago

lifeisstillgood 4y ago

Just today I was looking at how to use Alexa to help my son with his school spanish. I am not convinced it is perfect but it looks like being part of the puzzle.

deleted 4y ago

[deleted]

numpad0 4y ago

Funny ones I could find, in Japanese: - unanimous grammar score: 4, spelling score: 2: 3046: "今どうしても中華料理が食べたいのでテイクアウトで注文させてください": "I really want to eat Chinese right now, is it okay if I order some delivered(asking the boss nicely)" , (4,2),(4,2),(4,2) 13367: "単語を定義するとき文で明確にします": "[this will be]clarified in sentence when defining the word", (4,2),(4,2),(4,2) 13258: "ビーエムダブリューのマイレージ": "Mileage [reward program] for BMW", (4,2),(4,2),(4,2) 13986: "トップモデルの車": "Top-style car" or "car of top [beauty] model", (4,2),(4,2),(4,2) 14556: "この会社の株価を更新してください": "Please update stock price for this company", (4,2),(4,2),(4,2) 14592: "スマホのサーキットは何なのか教えて": "Tell me what is the smartphone raceway", (4,2),(4,2),(4,2) 15414: "マネージャーが必要だ": "There will have to be a manager", (4,2),(4,2),(4,2) - 2/3 grammar score: 4, spelling score: 2: 6235: "素晴らしい映画を見たのでコピーがリリースされたら予約してください": "Because I watched a magnificent movie, please reserve in case a knockoff is let go", (4,2),(4,2),(3,2) 14401: "あなたが私に探した最後のことを教えてください": "Tell me the last thing you have searched of me", (2,2),(4,2),(4,2) 14407: "感じることができるの": "Are you able to sense that", (4,2),(2,2),(4,2) - less than above 14744: "ブロイラーは何ですか、どのように使えばいいですか": "Broiler [meat] is what, how may I use it(for you/for community)", (4,2),(4,1),(3,2) 15456: "ユナイテッド航空にあなたが私の荷物を無くしたのに怒るとツイートをしたいけど": "Though I want to tweet to [Mr.]United Airlines that you become enraged despite you losing my luggage", (4,2),(4,2),(2,2)

numpad0 4y ago
I've since given a bit more thoughts on this - Part B question 3 and 4, from Fig.3 on the paper, asks only whether the sentence sound natural and (technically) correct, without specifying context, that the sentence is supposed to be an utterance spoken to a voice assistant. From my own comment above: 3046: "I really want to eat Chinese right now, is it okay if I order some delivered" 14556: "Please update stock price for this company" 15414: "There will have to be a manager" These are all context errors. What is not widely acknowledged(as I'd like...) is that translations for short sentences like "I need to see the manager" are situational, beyond depending on tones and politeness. Consider following: - I have complaints as a customer, and *I need to see the manager*. - I need to escalate this issue, and *I need to see the manager*. - You asked what I am about to do, and *I need to see the manager*. - "The manager" is a dashboard app, and *I need to see the manager*. I won't list my translations because I've already dumped way too much CJK characters here, but each of above four should turn up differently against each others from organic translators(especially the last one).

euroderf 4y ago

I poked around and didn't see a list of the 51 languages. Maybe I'm stupid.

jgmf 4y ago
Sorry, we didn't have enough space to fit it on the blog post. You can find the list of languages on page 4 of our paper: https://arxiv.org/pdf/2204.08582.pdf https://arxiv.org/pdf/2204.08582.pdf I'll add this to our github repo, too. Thanks!
- euroderf 4y ago
  Cheers! -Curious

amilios 4y ago

Funny ones I found in Greek (my native lang): "και έπεσε σκοτάδι" => "and darkness fell" (???) "είναι βρώμικα εδώ κάνε λίγο φασαρία" => "it's dirty here, make a fuss" But they're actually mostly pretty decent.

jgmf 4y ago
"and darkness fell" Yes, this one caused us some consternation :).

Gelob 4y ago

where are all the "shut up, alexa" and "fuck you alexa" phrases?