61 ms·

so ... nobody wonders about the obvious rammification: then any ML scoring systems ... must detect child abuse signals!

mlyle 7y ago
Hey uh, that actually seems valuable. I'd believe that ML could spot abuse that humans miss pretty well from signals like non-overt references in homework and school records, if one could come up with an adequate training set. Much more likely than teaching ML to score reasoned and creative activity in any reasonable way.
- kwhitefoot 7y ago
  What do you think the false positive rate is likely to be?
  - mlyle 7y ago
    Depends upon what false negative rate you're willing to tolerate. ;) And I don't know how good of a signal there is. This is pure handwaving. But this type of thing seems like the exact kind of spooky correlation that ML is good at spotting compared to humans.
    - YeGoblynQueenne 7y ago
      Machine learning techniques are going to be absolutely awful in detecting something like this, the reason being it's exceedingly rare (at least I'm guessing it is; if we're talking about child sexual abuse by one's own parents, it sure sounds extremely unlikely- but even child abuse in general is probably rare [1]). Machine learning systems are awful at identifying rare events. Like the OP seems to suggest, the false positive rate would most likely be very high. "Spooky" machine learning results happen when a correlation is abundant in a dataset [2]. Otherwise, machine learning techniques will probably miss it altogether. ______________ [1] Quick online search: https://www.inquirer.com/philly/blogs/healthy_kids/What-is-the-rate-of-child-abuse-in-schools--rec-groups.html https://www.inquirer.com/philly/blogs/healthy_kids/What-is-t... [2] The archetypal spooky machine learning story is surely the one about Target sending baby item coupos to a girl in high school before her father knowing she was pregnant: https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/#7e249cc66686 https://www.forbes.com/sites/kashmirhill/2012/02/16/how-targ...
      - mlyle 7y ago
        Humans are awful at rare events and vigilance tasks, too. That's part of why we're seeing machine vision and machine learning starting to outperform humans in e.g. grading radiology screening scans. The total incidence of child abuse of all types from infancy to adulthood is on the order of 1 in 3. This is not terrifically rare-- it's of higher prevalence than pregnancy and of positive screening events. A much bigger concern is non-causative correlations. It'd be pretty easy to train ML to be racist or look for e.g. indicators of class, which are correlates of abuse. As to false positive rates-- you can pick your false positive rate to be whatever you want it to be, by twiddling the threshold for a positive result. I'm not sure false positives are of that great of a concern, if the output from a system is a notification to school administrators that they may want to keep an eye out for this student.
      - dragonwriter 7y ago
        > if we're talking about child sexual abuse by one's own parents, it sure sounds extremely unlikely- Child sexual abuse isn't extremely rare and familial abuse is a very large minority of child sex abuse.
    - dragonwriter 7y ago
      > But this type of thing seems like the exact kind of spooky correlation that ML is good at spotting compared to humans. How? Particularly, where do you get training data at the required scale?
      - mlyle 7y ago
        You take samples of hundreds or thousands of past students' schoolwork, e.g. submissions of essays for standardized tests. You survey those kids in adulthood about whether and how they were subject to abuse and other types of relevant adversity. You attempt to control the data so that you don't just latch onto other correlates of abuse (e.g. social class).
  - mschuster91 7y ago
    Who cares? The positives must be evaluated by a human anyway.
    - tonyedgecombe 7y ago
      Who cares? The people whose lives are ruined by being mis-identified by the system. The positives must be evaluated by a human anyway. Those same people whose lack of competence people are bemoaning throughout these comments.
      - mschuster91 7y ago
        > The people whose lives are ruined by being mis-identified by the system. When a child writes "daddy touches me between the legs" in an essay, it doesn't matter if a human spots it or an AI that forwards it to a human, this needs to be investigated either way. > Those same people whose lack of competence people are bemoaning throughout these comments. It's not a lack of competence that's bemoaned, it's a massive amount of understaffing (and resulting overwork) in teachers and other school resources, as well as a drastic lack of financing because it's easy to cut budgets for schools for politicians as the effects only show up two decades afterwards.
        tonyedgecombe 7y ago
        Sprinkling some AI over it won't fix those issues, I'd argue it will make them worse as people blindly accept the results. There were some cases in the UK about a decade ago where bugs in software the Royal Mail was using led to incorrect accusations of fraud. People actually went to jail over this, it took years to resolve.
        mlyle 7y ago
        > When a child writes "daddy touches me between the legs" in an essay, it doesn't matter if a human spots it or an AI that forwards it to a human, this needs to be investigated either way. When a child writes a set of things that individually are not very concerning, they may have cues that could say "hey, this kid, you should maybe keep an eye out for evidence of abuse." Particularly attuned, experienced individuals might spot these cumulative cues, but we all know that this is not all people dealing with children. It's an interesting problem.
- chessturk 7y ago
  This is so dangerous. Society's bigotry is going to flood that bad boy so quick you might as well name it Gobbels. I love ML. I want children to be safe. This is not the place for ML or AI or Quantum or any tech. What needs to exist is better resources for those children, that mother grading the tests, the teachers of those children, and social services that are meant to support them. If you want to make a difference about this, look there. Don't go building a automaton King Solomon who decides why this kid should be taken from these parents because speaking Spanish was worth -0.1 on some goddamn weight trained on data generated from a racist society. This isn't a "spooky" correlation a cool algorithm can detect, it's a serious, layered social problem.
  - mlyle 7y ago
    > Don't go building a automaton King Solomon who decides why this kid should be taken from these parents because speaking Spanish was worth -0.1 on some goddamn weight trained on data generated from a racist society. Totally what I advocated for and not a strawman attack /s. Indeed, the chance that such an algorithm could be racist or classist and there being needs to avoid bad correlations and have appropriate controls is important. I think there are opportunities here. Ideally ed-tech doesn't take humans out of the loop, but asks schoolteachers and administrators questions like, "Hey, are you sure students A, B, and C are being supported correctly for subject Z? Are you sure students D, E doesn't have some kind of abuse or other significant home problem? It sure looks like student F is in this subpopulation that research shows benefits from educational intervention Y. You might want to keep your eye out for that." And then the teacher goes "Oh, crap. Now that I think about D, there were always these little things 1, 2, and 3 that seemed off... maybe this is worth a referral to social services to check on what's up." Or "Oh, ... maybe F's struggles in reading really are a speech problem and we should handle that"
belorn 7y ago
That is not how the law work. The law states that if people at a school is made aware or suspect abuse then they must act on that knowledge. A ML scoring system is obviously unable to be made aware or having suspicions, but the administrators could be help responsible if they happen to see something and chooses to not act. It would be interesting to know if a child psychiatrist could be held liable if incompetence prevented them from seeing obvious sign of abuse, but I doubt that is covered under the cited law above.

Hey uh, that actually seems valuable. I'd believe that ML could spot abuse that humans miss pretty well from signals like non-overt references in homework and school records, if one could come up with an adequate training set. Much more likely than teaching ML to score reasoned and creative activity in any reasonable way.

kwhitefoot 7y ago
What do you think the false positive rate is likely to be?
- mlyle 7y ago
  Depends upon what false negative rate you're willing to tolerate. ;) And I don't know how good of a signal there is. This is pure handwaving. But this type of thing seems like the exact kind of spooky correlation that ML is good at spotting compared to humans.
  - YeGoblynQueenne 7y ago
    Machine learning techniques are going to be absolutely awful in detecting something like this, the reason being it's exceedingly rare (at least I'm guessing it is; if we're talking about child sexual abuse by one's own parents, it sure sounds extremely unlikely- but even child abuse in general is probably rare [1]). Machine learning systems are awful at identifying rare events. Like the OP seems to suggest, the false positive rate would most likely be very high. "Spooky" machine learning results happen when a correlation is abundant in a dataset [2]. Otherwise, machine learning techniques will probably miss it altogether. ______________ [1] Quick online search: https://www.inquirer.com/philly/blogs/healthy_kids/What-is-the-rate-of-child-abuse-in-schools--rec-groups.html https://www.inquirer.com/philly/blogs/healthy_kids/What-is-t... [2] The archetypal spooky machine learning story is surely the one about Target sending baby item coupos to a girl in high school before her father knowing she was pregnant: https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/#7e249cc66686 https://www.forbes.com/sites/kashmirhill/2012/02/16/how-targ...
    - mlyle 7y ago
      Humans are awful at rare events and vigilance tasks, too. That's part of why we're seeing machine vision and machine learning starting to outperform humans in e.g. grading radiology screening scans. The total incidence of child abuse of all types from infancy to adulthood is on the order of 1 in 3. This is not terrifically rare-- it's of higher prevalence than pregnancy and of positive screening events. A much bigger concern is non-causative correlations. It'd be pretty easy to train ML to be racist or look for e.g. indicators of class, which are correlates of abuse. As to false positive rates-- you can pick your false positive rate to be whatever you want it to be, by twiddling the threshold for a positive result. I'm not sure false positives are of that great of a concern, if the output from a system is a notification to school administrators that they may want to keep an eye out for this student.
    - dragonwriter 7y ago
      > if we're talking about child sexual abuse by one's own parents, it sure sounds extremely unlikely- Child sexual abuse isn't extremely rare and familial abuse is a very large minority of child sex abuse.
  - dragonwriter 7y ago
    > But this type of thing seems like the exact kind of spooky correlation that ML is good at spotting compared to humans. How? Particularly, where do you get training data at the required scale?
    - mlyle 7y ago
      You take samples of hundreds or thousands of past students' schoolwork, e.g. submissions of essays for standardized tests. You survey those kids in adulthood about whether and how they were subject to abuse and other types of relevant adversity. You attempt to control the data so that you don't just latch onto other correlates of abuse (e.g. social class).
- mschuster91 7y ago
  Who cares? The positives must be evaluated by a human anyway.
  - tonyedgecombe 7y ago
    Who cares? The people whose lives are ruined by being mis-identified by the system. The positives must be evaluated by a human anyway. Those same people whose lack of competence people are bemoaning throughout these comments.
    - mschuster91 7y ago
      > The people whose lives are ruined by being mis-identified by the system. When a child writes "daddy touches me between the legs" in an essay, it doesn't matter if a human spots it or an AI that forwards it to a human, this needs to be investigated either way. > Those same people whose lack of competence people are bemoaning throughout these comments. It's not a lack of competence that's bemoaned, it's a massive amount of understaffing (and resulting overwork) in teachers and other school resources, as well as a drastic lack of financing because it's easy to cut budgets for schools for politicians as the effects only show up two decades afterwards.
      - tonyedgecombe 7y ago
        Sprinkling some AI over it won't fix those issues, I'd argue it will make them worse as people blindly accept the results. There were some cases in the UK about a decade ago where bugs in software the Royal Mail was using led to incorrect accusations of fraud. People actually went to jail over this, it took years to resolve.
      - mlyle 7y ago
        > When a child writes "daddy touches me between the legs" in an essay, it doesn't matter if a human spots it or an AI that forwards it to a human, this needs to be investigated either way. When a child writes a set of things that individually are not very concerning, they may have cues that could say "hey, this kid, you should maybe keep an eye out for evidence of abuse." Particularly attuned, experienced individuals might spot these cumulative cues, but we all know that this is not all people dealing with children. It's an interesting problem.
chessturk 7y ago
This is so dangerous. Society's bigotry is going to flood that bad boy so quick you might as well name it Gobbels. I love ML. I want children to be safe. This is not the place for ML or AI or Quantum or any tech. What needs to exist is better resources for those children, that mother grading the tests, the teachers of those children, and social services that are meant to support them. If you want to make a difference about this, look there. Don't go building a automaton King Solomon who decides why this kid should be taken from these parents because speaking Spanish was worth -0.1 on some goddamn weight trained on data generated from a racist society. This isn't a "spooky" correlation a cool algorithm can detect, it's a serious, layered social problem.
- mlyle 7y ago
  > Don't go building a automaton King Solomon who decides why this kid should be taken from these parents because speaking Spanish was worth -0.1 on some goddamn weight trained on data generated from a racist society. Totally what I advocated for and not a strawman attack /s. Indeed, the chance that such an algorithm could be racist or classist and there being needs to avoid bad correlations and have appropriate controls is important. I think there are opportunities here. Ideally ed-tech doesn't take humans out of the loop, but asks schoolteachers and administrators questions like, "Hey, are you sure students A, B, and C are being supported correctly for subject Z? Are you sure students D, E doesn't have some kind of abuse or other significant home problem? It sure looks like student F is in this subpopulation that research shows benefits from educational intervention Y. You might want to keep your eye out for that." And then the teacher goes "Oh, crap. Now that I think about D, there were always these little things 1, 2, and 3 that seemed off... maybe this is worth a referral to social services to check on what's up." Or "Oh, ... maybe F's struggles in reading really are a speech problem and we should handle that"

belorn 7y ago

That is not how the law work. The law states that if people at a school is made aware or suspect abuse then they must act on that knowledge. A ML scoring system is obviously unable to be made aware or having suspicions, but the administrators could be help responsible if they happen to see something and chooses to not act. It would be interesting to know if a child psychiatrist could be held liable if incompetence prevented them from seeing obvious sign of abuse, but I doubt that is covered under the cited law above.