4 ms·

Time for the obligatory xkcd again: https://xkcd.com/1958/ https://xkcd.com/1958/ Let's not forget that humans also make a lot of mistakes. The progress in (st

by t_mann 4y ago

Time for the obligatory xkcd again: https://xkcd.com/1958/ https://xkcd.com/1958/ Let's not forget that humans also make a lot of mistakes. The progress in (still mostly specific) AI over the last decade is nothing short of phenomenal. But I think it is good that we hold machines to a higher standard, because it's so much harder to hold anyone accountable.

mdp2021 4y ago
> Time for the obligatory xkcd again Which fails to consider that the ML outcomes fail on unreasonable structural misinterpretations, also explicit in the article (e.g. "with the wrong light the signal becomes a cat"). > Let's not forget that humans also make a lot of mistakes Let's not forget that humans are completely, radically different - in terms of implemented modules. An evaluation over reliability does not just come from test results: it also results from structural assessments. Do modify for the progress of Science and Technology the throat of a dog to make it speak proper cockney: it will remain a dog, just don't be fooled; said «"progress"» is in term of "re-search", not of "re-sults" - you are still searching. Mind the direction, because if you tend to fool yourself that a puppet with a tape player in its mouth does speak, there is a problem.
- t_mann 4y ago
  An evaluation over reliability does not just come from test results: it also results from structural assessments. Yes, but it's actually remarkable how bad humans are at seemingly simple tasks like recalling a short scene that they witnessed in person, or even just estimating how good their recollection is. Humans will routinely get even the most basic facts such as how many people were involved, who did what, in what order events occurred... completely wrong, while often being adamant that their account is correct. There are studies about that in the context of crime witnesses and the type of biases also mentioned in the article. A NN network might give a different answer if we apply some practically invisible Gaussian noise to an image, a human will have a different memory recollection depending on the lighting conditions of the room they're in, whether they're asked before or after lunch break... Again, I'm on board with saying that our standards for machines should be higher than that, but I think we're underestimating how hard those seemingly simple tasks are even for humans.
  - mdp2021 4y ago
    Well, don't call hour human a Witness if you have not implemented the ReferenceArchive module, and don't call your ANN a Judge if you have not implemented the ActiveOnthology (Philosophy) module.
oneoff786 4y ago
One of the examples in the article is gender detection. Humans might get this wrong sometimes. But no human is going to guess that a picture of dog feces is a human male or female. The ai model will. It only predicts male or female. But wait you say, you could teach it to guess male, female, or non human. But then, uh oh, per the article, lots of your Black human mages are no longer classified as human even! AI is hard. Evaluation metrics are hard. Cheering for 20 Int 0 Wis is a bad gamble imo.
- t_mann 4y ago
  But no human is going to guess that a picture of dog feces is a human male or female. Being a bit cheeky now, but I'm not even sure that you'd get zero such answers (or similarly absurd) from humans.
  - ben_w 4y ago
    I vaguely remember a similar thing with (Viking warrior?) skeletons. Everyone(?) assumed they were all male until they did DNA tests and realised they weren’t.
  - ufmace 4y ago
    Might be cheeky, but actually insightful too - all of these experiments share the assumption that the human/intelligence involved has chosen to cooperate with the experiment and do its best at the assigned task as written. So how do you control for things like the person decided that the assigned task was boring and wanted to amuse themselves by intentionally giving an incorrect answer in a way that they found amusing?
    - oneoff786 4y ago
      By acknowledging that these experiments are relatively pointless
      - ufmace 4y ago
        I don't know that I'd go that far. It's something that does need to be accounted for though. On the other hand, if we ever manage to build an AI that's sophisticated enough to intentionally troll us, well then mission accomplished I guess.

by t_mann 4y ago

mdp2021 4y ago

> Time for the obligatory xkcd again Which fails to consider that the ML outcomes fail on unreasonable structural misinterpretations, also explicit in the article (e.g. "with the wrong light the signal becomes a cat"). > Let's not forget that humans also make a lot of mistakes Let's not forget that humans are completely, radically different - in terms of implemented modules. An evaluation over reliability does not just come from test results: it also results from structural assessments. Do modify for the progress of Science and Technology the throat of a dog to make it speak proper cockney: it will remain a dog, just don't be fooled; said «"progress"» is in term of "re-search", not of "re-sults" - you are still searching. Mind the direction, because if you tend to fool yourself that a puppet with a tape player in its mouth does speak, there is a problem.

t_mann 4y ago
An evaluation over reliability does not just come from test results: it also results from structural assessments. Yes, but it's actually remarkable how bad humans are at seemingly simple tasks like recalling a short scene that they witnessed in person, or even just estimating how good their recollection is. Humans will routinely get even the most basic facts such as how many people were involved, who did what, in what order events occurred... completely wrong, while often being adamant that their account is correct. There are studies about that in the context of crime witnesses and the type of biases also mentioned in the article. A NN network might give a different answer if we apply some practically invisible Gaussian noise to an image, a human will have a different memory recollection depending on the lighting conditions of the room they're in, whether they're asked before or after lunch break... Again, I'm on board with saying that our standards for machines should be higher than that, but I think we're underestimating how hard those seemingly simple tasks are even for humans.
- mdp2021 4y ago
  Well, don't call hour human a Witness if you have not implemented the ReferenceArchive module, and don't call your ANN a Judge if you have not implemented the ActiveOnthology (Philosophy) module.

oneoff786 4y ago

One of the examples in the article is gender detection. Humans might get this wrong sometimes. But no human is going to guess that a picture of dog feces is a human male or female. The ai model will. It only predicts male or female. But wait you say, you could teach it to guess male, female, or non human. But then, uh oh, per the article, lots of your Black human mages are no longer classified as human even! AI is hard. Evaluation metrics are hard. Cheering for 20 Int 0 Wis is a bad gamble imo.

t_mann 4y ago
But no human is going to guess that a picture of dog feces is a human male or female. Being a bit cheeky now, but I'm not even sure that you'd get zero such answers (or similarly absurd) from humans.
- ben_w 4y ago
  I vaguely remember a similar thing with (Viking warrior?) skeletons. Everyone(?) assumed they were all male until they did DNA tests and realised they weren’t.
- ufmace 4y ago
  Might be cheeky, but actually insightful too - all of these experiments share the assumption that the human/intelligence involved has chosen to cooperate with the experiment and do its best at the assigned task as written. So how do you control for things like the person decided that the assigned task was boring and wanted to amuse themselves by intentionally giving an incorrect answer in a way that they found amusing?
  - oneoff786 4y ago
    By acknowledging that these experiments are relatively pointless
    - ufmace 4y ago
      I don't know that I'd go that far. It's something that does need to be accounted for though. On the other hand, if we ever manage to build an AI that's sophisticated enough to intentionally troll us, well then mission accomplished I guess.