4 ms·
No, you're completely on the right track. People usually judge AI systems based on superhuman performance criteria with almost no human baseline. For example,
by AndrewKemendo 9y ago
No, you're completely on the right track.
People usually judge AI systems based on superhuman performance criteria with almost no human baseline.
For example, both Google Translate and Facebook's translation system could reasonably be considered superhuman in performance because the singular systems can translate into dozens of languages immediately more accurately and better than any single human could. Unfortunately people compare these to a collection of the best translators in the world.
So you're exactly on track, that humans are heavily prejudiced against even simple mistakes that computers make, yet let consistent continuous mistakes slide for humans.
- psyc 9y agoI might use this in future critical discussions of AI. “It’s not really intelligent.” Yeah, well, neither am I. On a more serious note, it seems obvious to me that technology is incremental, and we are where we are. Given 20 more years of peacetime, we’ll be further along. When VGA 320x200x256 arrived it was dubbed photorealistic. I wonder what HN would have had to say about that.
- AndrewKemendo 9y agoYes, that's almost exactly how I start most of these conversations: "Do you know how BAD we are at most things?" Venkatesh Rao had a great piece on how mediocre we really are recently: https://www.ribbonfarm.com/2018/04/24/survival-of-the-mediocre-mediocre/ https://www.ribbonfarm.com/2018/04/24/survival-of-the-medioc...
- evrydayhustling 9y agoThis is well said, but one reason this double-standard is rational is that current AI systems are far worse at recovery-from-error than humans are. A great example of this is Alexa: if a workflow requires more than one statement to complete, and Alexa fails to understand what you say, you are at the mercy of brittle conversation logic (not AI) to recover. More often than not you have to start over or worse yet execute an incorrect action, then do something new to cancel. In contrasts, even humans who can barely understand each other can build understanding gradually because of the context and knowledge they share. Our best AIs are superhuman only at tightly scoped tasks, and our prejudice encodes the utility we get from the flexibility and resilience of general intelligence.
- visarga 9y ago> our prejudice encodes the utility we get from the flexibility and resilience of general intelligence I don't think any particular human is so general in intelligence. We can do stuff related to day to day survival (walk, talk, eat, etc) and then we have one or a few skills to earn a living. Someone is good at programming, another at sales, and so on - nobody is best at all tasks. We're lousy at general tasks. General intelligence includes tasks we can't even imagine yet. For thousands of years the whole of humanity survived with a mythical / naive understanding of the world. We can't even understand the world in one lifetime. Similarly, we don't even understand our bodies well enough, even with today's technology. I think human intelligence remained the same, what evolved was the culture, which is a different beast. During the evolution of computers, CPUs remain basically the same (just faster, they are all Turing machines) - what evolved was the software, culminating in current day AI/ML. What you're talking about is better explained by prejudice against computers based on past experience, but we're bad at predicting the evolution of computing and our prejudices are lagging.
- foldr 9y ago>Unfortunately people compare these to a collection of the best translators in the world. I don't think that's really true. They're comparing them to the performance of a person who is a native speaker of both of the languages in question. That seems like a pretty reasonable comparison point, since it's basically the ceiling for performance on a translation task (leaving aside literary aspects of translation). If you know what the sentence in the source language means, and if you know the target language, then you can successfully translate. Translation isn't some kind of super difficult task that only specially trained translators can do well. Any human who speaks two languages fluently (which is not rare, looking at the world as a whole) can translate between them well.
- yongjik 9y ago> Any human who speaks two languages fluently (which is not rare, looking at the world as a whole) can translate between them well. No, sadly, that's not the case. A bilingual person may create a passable translation, in that they can keep the main gist and that all the important words in the original text will appear in the translated text in some form, but that does not automatically make it a "well-translated" text. They are frequently contorted, using syntaxes that hardly appear in any natural sentences, and riddled with useless or wrong pronouns. A good translation requires more skill than just knowing the two languages.
- foldr 9y agoRight, but I don't think many people expect machine translation to do better than that. And I don't think the ability to do good translations is quite as rare among bilinguals as you're making out. Even kids tend to be pretty good at it according to some research: https://web.stanford.edu/~hakuta/www/research/publications/(1991)%20-%20TRANSLATION%20SKILL%20AND%20METALINGUISTIC%20AWARENESS%20IN%20B.pdf https://web.stanford.edu/~hakuta/www/research/publications/(... > They are frequently contorted, using syntaxes that hardly appear in any natural sentences, and riddled with useless or wrong pronouns. I find it hard to believe that this is the case if the translator is a native speaker of the target language. I mean, I might do a bad job of translating a Spanish text to English (because my Spanish sucks), but my translation isn't going to be bad English, it's just going to be inaccurate.
- simen 9y agoBeing able to do many things at a level below what trained humans can isn't what any reasonable person would call superhuman performance. If machine translation could perform at the level of human translators at even one pair of languages (like English-Mandarin), that would be impressive. That would be the standard people apply. But they very cleary can't.
- AndrewKemendo 9y agoIt's a problem with who you're comparing against. Generally people think superhuman = better than the best humans. I understand this and it's an obvious choice, but it assumes that humans are measured on a objective scale of quality for a task, which is rarely the case. Being on the front line of deploying ML systems, I think it's the wrong way to measure it. I think Superhuman should be considered relative to the competence level of average person who has the average amount of training on the task. This is because from the "business decision" level, if I am evaluating between hiring a human with a few months or a year of training and a tensorflow docker container that is reliably good/bad, then I am going to pick the container every time. That's what is relevant today - and the container will get better.
- simen 9y ago> Generally people think superhuman = better than the best humans. Because that's what it means. Also, I shudder to think of the prose coming out of a company that decides to "hire" an AI instead of a translator...
- AndrewKemendo 9y agoWell not explicitly or in any measurable terms [1]. The term 'Superhuman' lacks technical depth in the sense of measurement. So for the purposes of measuring systems we build vs human capability, it's a pretty terrible measure. [1] http://www.dictionary.com/browse/superhuman http://www.dictionary.com/browse/superhuman