4 ms·

Translation quality is not great. A sentence like "maak de verlichting een beetje meer warm hier" is absolutely not idiomatic Dutch, and "wat is de tijdt in de

by tgv 5y ago

Translation quality is not great. A sentence like "maak de verlichting een beetje meer warm hier" is absolutely not idiomatic Dutch, and "wat is de tijdt in de andere tijdzones" contains an easily caught spelling error. "laat me mijn wekkers zien" is quite ridiculous; it means: "show me my (alarm) clocks". "ontvang me updates van bram´s facebook van het weekend" is ungrammatical. Well, not great.

1f60c 5y ago
"Laat me mijn wekkers zien" is indeed a bit odd, but it makes sense if you think of it in the context of a voice assistant that displays information on a screen.
- usr1106 5y ago
  In German I see: "putzen ist gut staub ist so schlecht mach jetzt deinen zauber sauber mein teppich" This is complete nonsense. Mark Twain could have written it to make fun of the German language.
  - jgmf 5y ago
    Ah, that's unfortunate to hear. In this case I see grammar_score ranging from 2 (Some errors (the meaning can be understood but it doesn't sound natural in your language)) to 4 (Perfect)--looks like one of the raters was too generous with a rating of 4. I think this feedback is helping me to realize that we should be more explicit about our philosophy of keeping some of the noise in the dataset and allowing people to filter based on judgments. That said, people do say some strange and ungrammatical things to virtual assistants (not saying this example is representative per se), so it's nice to include some of the odd ones. Thanks!
    - usr1106 5y ago
      Thanks for following up! I guess what happened is a user of your system tried to be funny. In this utterance there are 3 short fully grammatical sentences. And a 4th one which is not very grammatical but fully understandable, commanding the device to clean the carpet :) - Cleaning is good. - Dust is so bad. - Make your miracle. - Cleanly my carpet. [1] [1] original is an ungrammatical imperative Some (more) human review would have been needed. The Finnish samples are full of very weird utterances, too. Some like people might have written over 50 years ago, but nobody would speak like this. Maybe they were reviewed on Mechanical turk with moderate requirements / payment? Well, you get what you pay for...
    - trhway 5y ago
      >people do say some strange and ungrammatical things to virtual assistants with virtual assistants and the likes appearing more and more around one can wonder whether a new language/dialect can emerge for that setting.
      - usr1106 5y ago
        Absolutely. I thought the same thing. If virtual assistants [1].were widely used and they don't fully understand all natural language the user might want to use, users will adapt and start to use some restricted language. [1] Personally I think it's a useless concept and the examples in the dataset mostly confirmed my view. Come on, a time zone conversion. Do we want to get so stupid that we cannot calculate it ourselves anymore?.Adjusting a light? Most of us sit too much and standing up is good for your body. Such technology is not only useless, but just bad. Not to mention the surveillance capitalism aspect which is proably also there or even prevailing .
  - fhars 5y ago
    Might be some service incantation for flying carpets, "cleaning is good, dust is so bad, now clean your spell, my carpet!" With the right wand movements, it could sort of make sense.
  - deleted 5y ago
    [deleted]
- Insanity 5y ago
  I would probably ask the assistant “Toon mijn wekkers”. Never actually used one in Dutch though and any “command” sounds odd to me in Dutch even though it’s my native language
jgmf 5y ago
Jack here from Alexa. Thanks for the feedback. Quality control was nontrivial, to put it succinctly, but we certainly always want to be better. I'll have to check if the issues you've noticed were detected in the judgement scores. For the first utterance you mentioned, all three raters put a score of 1 for spelling_score, which means "There are 1-2 spelling errors." So that's good. Though we re-collected some utterances with low scores, we didn't have the budget to get perfect scores for all utterances. As such, we decided to include all utterances along with the scores from the 3 raters, such that users can perform filtering as they'd like. Some may want to keep the noise intact to help with training.
- usr1106 5y ago
  This is good info. I just had a quick glance and did not pay attention to the scores.
fxtentacle 5y ago
Luckily, this matters a lot less than one might think. A good AI architecture can tolerate a few % of noise in the data. For example, A LOT of people in the Oscar dataset spelled the German "Metzger" as "Metzker" but AIs trained on it still perform great and strongly prefer the correct spelling.

by tgv 5y ago

1f60c 5y ago

"Laat me mijn wekkers zien" is indeed a bit odd, but it makes sense if you think of it in the context of a voice assistant that displays information on a screen.

usr1106 5y ago
In German I see: "putzen ist gut staub ist so schlecht mach jetzt deinen zauber sauber mein teppich" This is complete nonsense. Mark Twain could have written it to make fun of the German language.
- jgmf 5y ago
  Ah, that's unfortunate to hear. In this case I see grammar_score ranging from 2 (Some errors (the meaning can be understood but it doesn't sound natural in your language)) to 4 (Perfect)--looks like one of the raters was too generous with a rating of 4. I think this feedback is helping me to realize that we should be more explicit about our philosophy of keeping some of the noise in the dataset and allowing people to filter based on judgments. That said, people do say some strange and ungrammatical things to virtual assistants (not saying this example is representative per se), so it's nice to include some of the odd ones. Thanks!
  - usr1106 5y ago
    Thanks for following up! I guess what happened is a user of your system tried to be funny. In this utterance there are 3 short fully grammatical sentences. And a 4th one which is not very grammatical but fully understandable, commanding the device to clean the carpet :) - Cleaning is good. - Dust is so bad. - Make your miracle. - Cleanly my carpet. [1] [1] original is an ungrammatical imperative Some (more) human review would have been needed. The Finnish samples are full of very weird utterances, too. Some like people might have written over 50 years ago, but nobody would speak like this. Maybe they were reviewed on Mechanical turk with moderate requirements / payment? Well, you get what you pay for...
  - trhway 5y ago
    >people do say some strange and ungrammatical things to virtual assistants with virtual assistants and the likes appearing more and more around one can wonder whether a new language/dialect can emerge for that setting.
    - usr1106 5y ago
      Absolutely. I thought the same thing. If virtual assistants [1].were widely used and they don't fully understand all natural language the user might want to use, users will adapt and start to use some restricted language. [1] Personally I think it's a useless concept and the examples in the dataset mostly confirmed my view. Come on, a time zone conversion. Do we want to get so stupid that we cannot calculate it ourselves anymore?.Adjusting a light? Most of us sit too much and standing up is good for your body. Such technology is not only useless, but just bad. Not to mention the surveillance capitalism aspect which is proably also there or even prevailing .
- fhars 5y ago
  Might be some service incantation for flying carpets, "cleaning is good, dust is so bad, now clean your spell, my carpet!" With the right wand movements, it could sort of make sense.
- deleted 5y ago
  [deleted]
Insanity 5y ago
I would probably ask the assistant “Toon mijn wekkers”. Never actually used one in Dutch though and any “command” sounds odd to me in Dutch even though it’s my native language

jgmf 5y ago

Jack here from Alexa. Thanks for the feedback. Quality control was nontrivial, to put it succinctly, but we certainly always want to be better. I'll have to check if the issues you've noticed were detected in the judgement scores. For the first utterance you mentioned, all three raters put a score of 1 for spelling_score, which means "There are 1-2 spelling errors." So that's good. Though we re-collected some utterances with low scores, we didn't have the budget to get perfect scores for all utterances. As such, we decided to include all utterances along with the scores from the 3 raters, such that users can perform filtering as they'd like. Some may want to keep the noise intact to help with training.

usr1106 5y ago
This is good info. I just had a quick glance and did not pay attention to the scores.

fxtentacle 5y ago

Luckily, this matters a lot less than one might think. A good AI architecture can tolerate a few % of noise in the data. For example, A LOT of people in the Oscar dataset spelled the German "Metzger" as "Metzker" but AIs trained on it still perform great and strongly prefer the correct spelling.