6 ms·

Yeah, I know about how current automatic MT benchmarks don't reflect "user satisfaction" very accurately, and that it's an open problem to get one that is servi

by zodiac 6y ago

Yeah, I know about how current automatic MT benchmarks don't reflect "user satisfaction" very accurately, and that it's an open problem to get one that is serviceable. However you make it sound like all deep learning solutions to tasks perform well on the task benchmark but poorly on the real-world task the benchmarks try to approximate, whereas that's not true for MT - they are bad at the benchmarks, but outperform non-deep-learning-based translation approaches at the real-world-task. On the subject of benchmarks, how about speech transcription? I was under the impression that those benchmarks are pretty reliable indicators of "real-world accuracy" (or about as reliable as benchmarks are in general)

YeGoblynQueenne 6y ago
I don't know much about speech transcription, sorry. How do you mean that machine translation outperforms non deep learning based approaches at the real world task? How is this evaluated?
- zodiac 6y ago
  > How do you mean that machine translation outperforms non deep learning based approaches at the real world task? How is this evaluated? There were two ways that performance can be evaluated that I had in mind: 1. Commercial success - what approach does popular sites like Google Translate use? 2. Human evaluation - in a research setting, ask humans to score translations - this is mentioned in your link
  - YeGoblynQueenne 6y ago
    OK, thanks for clarifying. The problem with such metrics is that they're not objective results. It doesn't really help us learn much to say that a system outperforms another based on subjective evaluations like that. You might as well try to figure out which is the best team in football by asking the fans of Arsenal and Manchester United.
    - zodiac 6y ago
      The subjective human evaluations used in research are blinded - the humans rate the accuracy of translation without knowing what produced the translation (whether a NMT system, a non-ML MT system, or a human translator), whereas the football fans in your scenario are most definitely not blinded. There are some criticisms you could make about human evaluation, but as far as how well they correspond to the real-world task, I think they're pretty much the best we can do. I'm very curious to know if you actually think they're a bad target to optimize for. More to the point, you still have yet to show that NMT "serves no other purpose than to show how well modern techniques can model large datasets", given that they do well on human evaluations and they're actually producing value by serving actual production traffic (you know, things humans actually want to translate) in Google Translate. If serving production traffic like this is not "serving a purpose", what is?
      - YeGoblynQueenne 6y ago
        Sorry for the late reply. Regarding whether human evaluations are a good target to optimise for, no, I certainly don't think so. That's not very different than calculating BLEU scores, except that instead of comparing a machine generated translation with one reference text, it's compared with peoples' subjective criteria, which actually serves to muddle the waters even more -because who knows why different people thought the same translation was good or bad? Are they all using the same criteria? Doubtful! But if they're not, then what have we learned? That a bunch of humans agreed that some translation was good, or bad, each for their own reasons. So what? It doesn't make any difference when the human evaluators are blinded, you can make the same experiment with human translations only and you'll still have not learned anything about the quality of the translation- just the subjective opinions of a particular group of humans about it. See, the problem is not just with machine translation. Evaluating human translaion results is also very hard to do because translation itself is a very poorly characterised task. The question "what is a good translation is very difficult to answer. We don't have, say, a science of translation to tell us how a text should be translated between two languages. So in machine translation people try to approximate not only the task of translation, but also its evaluation- without understanding either. That's a very bad spot to be in. In fact, a "science of translation" would be a useful goal for AI research, but it's the kind of thing that I complain is not done anymore, having been replaced with beating meaningless benchmarks. Regarding the fact that neural machine translation "generates value", you mean that it's useful because it's deployed in production and people use it? Well, "even a broken clock is right twice a day" so that's really not a good criterion of quality at all. In fact, as a criterion for an AI approach it's very disappointing. Look at the promise of AI: "machines that think like humans!". Look at the reality: "We're generating value!". OK. Or we could make an app that adds bunny ears and cat noses to peoples' selfies (another application of AI). People buy the app- so it's "generating value". Or we can generate value by selling canned sardines. Or selling footballs. Or selling foot massages. Or in a myriad other ways. So why do we need AI? It's just another useless trinket that is sold and bought while remaining completely useless. And that for me is a big shame.

by zodiac 6y ago

YeGoblynQueenne 6y ago

I don't know much about speech transcription, sorry. How do you mean that machine translation outperforms non deep learning based approaches at the real world task? How is this evaluated?

zodiac 6y ago
> How do you mean that machine translation outperforms non deep learning based approaches at the real world task? How is this evaluated? There were two ways that performance can be evaluated that I had in mind: 1. Commercial success - what approach does popular sites like Google Translate use? 2. Human evaluation - in a research setting, ask humans to score translations - this is mentioned in your link
- YeGoblynQueenne 6y ago
  OK, thanks for clarifying. The problem with such metrics is that they're not objective results. It doesn't really help us learn much to say that a system outperforms another based on subjective evaluations like that. You might as well try to figure out which is the best team in football by asking the fans of Arsenal and Manchester United.
  - zodiac 6y ago
    The subjective human evaluations used in research are blinded - the humans rate the accuracy of translation without knowing what produced the translation (whether a NMT system, a non-ML MT system, or a human translator), whereas the football fans in your scenario are most definitely not blinded. There are some criticisms you could make about human evaluation, but as far as how well they correspond to the real-world task, I think they're pretty much the best we can do. I'm very curious to know if you actually think they're a bad target to optimize for. More to the point, you still have yet to show that NMT "serves no other purpose than to show how well modern techniques can model large datasets", given that they do well on human evaluations and they're actually producing value by serving actual production traffic (you know, things humans actually want to translate) in Google Translate. If serving production traffic like this is not "serving a purpose", what is?
    - YeGoblynQueenne 6y ago
      Sorry for the late reply. Regarding whether human evaluations are a good target to optimise for, no, I certainly don't think so. That's not very different than calculating BLEU scores, except that instead of comparing a machine generated translation with one reference text, it's compared with peoples' subjective criteria, which actually serves to muddle the waters even more -because who knows why different people thought the same translation was good or bad? Are they all using the same criteria? Doubtful! But if they're not, then what have we learned? That a bunch of humans agreed that some translation was good, or bad, each for their own reasons. So what? It doesn't make any difference when the human evaluators are blinded, you can make the same experiment with human translations only and you'll still have not learned anything about the quality of the translation- just the subjective opinions of a particular group of humans about it. See, the problem is not just with machine translation. Evaluating human translaion results is also very hard to do because translation itself is a very poorly characterised task. The question "what is a good translation is very difficult to answer. We don't have, say, a science of translation to tell us how a text should be translated between two languages. So in machine translation people try to approximate not only the task of translation, but also its evaluation- without understanding either. That's a very bad spot to be in. In fact, a "science of translation" would be a useful goal for AI research, but it's the kind of thing that I complain is not done anymore, having been replaced with beating meaningless benchmarks. Regarding the fact that neural machine translation "generates value", you mean that it's useful because it's deployed in production and people use it? Well, "even a broken clock is right twice a day" so that's really not a good criterion of quality at all. In fact, as a criterion for an AI approach it's very disappointing. Look at the promise of AI: "machines that think like humans!". Look at the reality: "We're generating value!". OK. Or we could make an app that adds bunny ears and cat noses to peoples' selfies (another application of AI). People buy the app- so it's "generating value". Or we can generate value by selling canned sardines. Or selling footballs. Or selling foot massages. Or in a myriad other ways. So why do we need AI? It's just another useless trinket that is sold and bought while remaining completely useless. And that for me is a big shame.