5 ms·

The claim of catching 99% of activity is really hard to justify. In many detection systems you cannot know your true number of false negatives. It's inherently

The claim of catching 99% of activity is really hard to justify. In many detection systems you cannot know your true number of false negatives. It's inherently unknowable in this case. A better metric would be mean time to detection for a given CEP sharer or perhaps a metric around how much CEP is shared before detection, or perhaps a metric for network size of CEP and how well FB can clean up an entire network at once. This NYTimes article https://www.nytimes.com/interactive/2019/09/28/us/child-sex-abuse.html https://www.nytimes.com/interactive/2019/09/28/us/child-sex-... is quite good. I don't want to punish FB for doing a good job of detection by turning around clutching my pearls at them. Stamos tweeted that every hosting platform has these challenges and I believe him. There has to be some nuance from the absolute privacy folks on this one. How do we balance the need to fight child abuse with privacy?

6gvONxR4sf7o 7y ago
I mean, you could always do an audit and see what percent of a thing your automated system catches. That definitely doesn't seem inherently unknowable.
- salawat 7y ago
  You're not quite getting it. You measure accuracy of detection via a training set of definitely known examples, and comparing it to the detections reported by your system. You can say with certainty that their system catches 99% of your training data set, but that doesn't mean it Will generalize to "all permutations of child abuse ever". That's two totally different things. In the end their use of that statistic illustrates an attempt to lie with statistics more than anything else.
  - edmundsauto 7y ago
    By this standard, no system could ever say that it generalizes to all permutations. Especially when the definition of the measurement outcome (child abuse) varies.
    - salawat 7y ago
      Now you get it. You are now the thorn in the side of every AI salesman trying to pitch you his Neural Net based wonder machine, or politician trying to sell you on giving up rights because they need to be able to violate everyone's privacy because statistics. If you understand the actual nature of these measurements, it becomes quite a bit more difficult to let the "zomg neato" factor run away with you.
    - cwkoss 7y ago
      This is true. Until we implement omnipotent god-AIs, generalizable perfection is an unattainable standard.
  - lozaning 7y ago
    Alex Pinto gave a phenomenal DefCon talk several years ago that really highlights this issue, "Secure Because Math". It gets into how modern ML/AI/Voodoo IPS systems essentially can't work because the training data or baseline is so impossible to accurately come up with. https://www.youtube.com/watch?v=TYVCVzEJhhQ https://www.youtube.com/watch?v=TYVCVzEJhhQ
  - 6gvONxR4sf7o 7y ago
    If I say 15% of cars in San Francisco are Subarus, are you going to say that's impossible to know? Sure it may be true in my dataset, "but that doesn't mean it will generalize to all possible permutations of [traffic] ever." Well, no shit. But that's not the statement being made. If I wanted to know the accuracy of an important system. I'd do continual audits. Take a (stratified) sample of all things, get ground truth labels for whether they should have been detected, and see what percent were in fact detected. Then I could estimate how much my systems did and didn't catch at any point in time.
    - salawat 7y ago
      >If I say 15% of cars in San Francisco are Subarus, are you going to say that's impossible to know? No, I'd ask you what your methodology was when you measured. For instance, going to the DMV and getting a list of all registered vehicles, and using that as a representative dataset to reason from is fine. (Depending on your definition of cars in San Francisco, and whether or not that is meant to include cars transiently moving through or not.) Now if you told me you were using a Neural Network to recognize all Subaru models that drove by a particular camera, I'd start asking you questions, and want to see your training data, output, etc. >Sure it may be true in my dataset, but that doesn't mean it will generalize to all possible permutations of traffic ever. Well, no shit. But that's not the statement being made. That generally is the statement being made when people make claims about the accuracy of neural networks. Particularly the marketing people. You can only measure their accuracy within a constrained dataset, and overfitting is a thing. >If I wanted to know the accuracy of an important system. I'd do continual audits. Take a (stratified) sample of all things, get ground truth labels for whether they should have been detected, and see what percent were in fact detected. Then I could estimate how much my systems did and didn't catch at any point in time. Which, again means you're operating off things you know a priori to be the case. You don't know the full extent of everything to begin with, which phrasing in this article implies. I'm not saying audits can't be useful; it'll get you a number, and sometimes that's all you need. You just need to be clear about what the numbers actually mean. There's lies, damn lies, and statistics after all.
zzzcpan 7y ago
> There has to be some nuance from the absolute privacy folks on this one. How do we balance the need to fight child abuse with privacy? There isn't. This is a logical fallacy. Violating everyone's privacy is not the way to fight child abuse, not even one of the ways that can do anything to prevent it.
- brunoTbear 7y ago
  I'm not sure what the logical fallacy is you're referring to. I'm not advocating that we backdoor encryption. I'm asking what's the nuance in privacy vs crime prevention. I guess your position is one of an absolute, and therefore, to me, not especially interesting. I am curious what you think are the ways we can fight child abuse. It seems to me we have some obligation as technologists to see that our works aren't used to harm the most vulnerable.
  - ilovetux 7y ago
    I believe the trade-off for privacy has traditionally been done on a case-by-case basis through the use of search warrants. I don't see anything wrong with the use of warrants. If there is reasonable suspicion the police can investigate further, it's sensible. Now we are seeing multiple instances where scapegoats and boogeymen are being used to systematically strip our rights away. The worst part of the whole thing is that when I attempt to fight for my privacy, I would be seen as supporting child pornography. It is an insidious tactic that is unreasonably effective.
  - danShumway 7y ago
    > I'm asking what's the nuance in privacy vs crime prevention. This is the whole problem. There _is_ nuance. Users can still report content, even if it's private. Automated detection systems can still run client-side. Public forums like Facebook itself are still targetable, and we can increase funding and counseling for moderators in those public spaces to reduce turnover. Barr is the one here saying that, "mere numbers" aren't enough to talk about the problem. How exactly are we supposed to compromise with that? What exactly is a 'nuanced' response to the argument that any system that allows even one child to be abused is unacceptable? Barr isn't going to be happy unless law enforcement can read every single message. And he's going to trot out the same arguments every single time. And every time we respond, somebody is going to be jumping in to say, "but why can't we just have a little nuance? Why are all of you so dogmatic about this?"
claytoneast 7y ago
The sources in that article who lead these anti-child-trafficking/image organizations complain about cripplingly inadequate budgets and lack of human resources (employees). They do NOT complain about end-to-end encryption. They ask for more people, and more money, which the government is apparently consistently unwilling to grant them. Instead, the govt foists this absolute horseshit about limiting/backdooring e2e on us, while screaming THINK OF THE CHILDREN.
- brunoTbear 7y ago
  WFIW I didn't advocate backdooring either, I'm just asking how we, as technologists, balance what I think are reasonable and compelling goals: protect user data, and protect children. One reasonable answer is that FB/Google/etc. should donate or donate more (Google is called out as donating already) to these organizations. I think another reasonable answer is that technologists donate our time, perhaps as 20% projects or as sabbaticals to these orgs and help them modernize. It does seem Congress (per the NYTimes article I linked above) isn't doling out as much funding as we would like to see. And, in fairness, children are being harmed here, and at an incredibly alarming rate. Please do read the NYTimes article. The Times is, IMO, a fairly responsible actor in truth telling.
  - claytoneast 7y ago
    I did read the article, though I think I may now see your point: this technology has greatly enabled abusers, perhaps more than it has helped finders-and-punishers-of-abusers. Your question is a good one: "how we, as technologists, balance what I think are reasonable and compelling goals: protect user data, and protect children."
    - 0_gravitas 7y ago
      "How do we, as postal workers, balance the reasonable and compelling goals: get people their mail, and protect children." One is our job that we are directly and morally responsible for (like a civil engineer being responsible for having a bridge not collapse at a load of >2 cars), the other is not. Call me callous, call me cold, but that is not explicitly up to us, we can't be under the delusion that we are the one key to everything and anything. It wouldn't make sense to have a "quick-release" button forcibly installed on all bridges that would cause an instant collapse if triggered "because countless children that are victims of human trafficking are taken over bridges everyday."
psychometry 7y ago
You can never know the exact number, but you can accurately estimate it. This is the entire point of statistical inference.
larkeith 7y ago
I read that as 99% of activity that is caught on Facebook is caught by Facebook's systems, with law enforcement only catching 1%.
- brunoTbear 7y ago
  That's still not an especially resounding success, or an especially relevant metric for us to understand the efficacy of FB's detection systems, and doesn't help us understand whether encryption of content is going to hurt hunting CEP. For example, they might be spotting a large fraction via network analysis, or other metadata approaches. Alternatively, they might just hash every image sent over messenger and match those to known CEP. Encryption screws up one avenue, but not the other.
  - jandrese 7y ago
    All it says is that the cops are terrible at finding this kind of content on FB, which is to be expected since they have only a very limited view into the platform. The only stuff the cops are catching that FB isn't is when they knock down someone's door and grab their FB password to look for stuff the filters missed.
  - akira2501 7y ago
    > encryption of content is going to hurt hunting CEP. Aren't we putting the cart before the horse? The problem is the exploitation of children, the secondary problem is that this exploitation is sometimes photographed and shared. I don't believe that a strong solution the latter problem will have any impact on the former.
4ntonius8lock 7y ago
Well, why bring privacy into the fray at all? I'm not saying online systems for catching child abuse can't be implemented. But is it really where our efforts would be most effective? The question should be: how do you combat child abuse? For that, we need: 1- Strong reporting systems that connect the people who in contact with children (doctors, teachers, social workers) and centralize and distribute that information. Right now, a social worker doesn't know about the doctor or teacher reports very easily. 2- Then, we need strong systems for handling the situation: i.e. social workers with effective systems for intervention. 3- Then, we need strong systems to take the children that are removed from at risk situations and place them in safe places where they can recover and grow in a healthy environment. This is the foster care system and the national health care system (or lack thereof) that will provide mental health counseling for the affected children. Given that 1, 2 and 3 of these systems are terribly dysfunctional if not mostly broken, I'm always skeptical when people talk about social media and child protection. The situation gives me a feeling of leaders justifying the erosion of privacy and those who are authoritarian inclined supporting them, rather than people really caring about protecting children.
- scott_s 7y ago
  The tension between privacy and child abuse is laid out in the linked NYT piece (which I also recommend reading): many of the images and videos of child sex abuse are created because they can be shared. They can be shared because the creators have some confidence they can do so without getting caught. Once the content is discovered, the authorities also have a difficult time finding that child, who may still be in danger. This situation is orthogonal to the (very necessary) steps you outlined.
  - 4ntonius8lock 7y ago
    Is it? I just read the NYT article. Maybe I'm missing something, but it clearly states that the main issues saving children are a lack of enforcement availability (i.e. the 3 points mentioned in my comment) and the lack of timely cooperation of existing orders. Then they talk about how 'some criminals' hide behind encryption. So... we have a situation where many cases are reported but aren't followed up on. Yet we focus on getting more reporting by undermining the right of non criminals? Doesn't seem very logical to me. The whole piece even undermines their own arguments. They mention the 'case of the Love Zone'... which was cracked by investigators finding clues the old fashioned way. Not drag net surveillance. The article also places many issues that aren't tech related as tech problems. It says "Bing was said to regularly submit reports that lacked essential information, making investigations difficult" Let's compare that line to offline: If I own a pizza parlor where pedo's hang out sometimes, and I find a picture of CP in a restroom and hand it in to the authorities or call them up, would you blame the Pizza Parlor for the lack of fingerprints on the pictures? Would you think it's ok then to make wearing gloves illegal? I mean, if people use gloves, pedos can use gloves and hide their fingerprints. Why not target gloves? There is a real issue: Child Abuse. There are real solutions that are complex and extremely resource intensive. A report doesn't create a case. Focusing on tech and the lack of having MORE drag net surveillance seems absurd. This is coming from someone who grew up with abuse. When I go to forums to discuss abuse (it helps me deal with what happened) almost everyone tells of times that the abuse was reported but not well investigated. In fact, I've known people who work in CPS and they almost universally say the system is broken. A child in my city was recently beat to death by his step dad... an CPS had been called on them multiple times. We could be focusing on fixing that (you know, the things the people on the front line say are wrong). But no. Drag net surveillance is the thing we should focus on apparently (even though in the very article you mention it states we have more current reporting than we have resources to deal with it)
awaythrowacct 7y ago
It's easy to detect a vast amount of child porn, simply because the vast majority of child porn sharing is just a few thousand different files, shared over and over again. A few hundred thousand files if you count the non-nude and nude "child modelling" stuff, too. You can catch a majority of such content just by running a dumb hash over the bits, even though the detection rate suffered a little thanks to dumb smartphone users who will screenshot everything instead of sharing the files, thus creating "new" content. If you add some simple "content hashing", like https://pypi.org/project/dhash/ https://pypi.org/project/dhash/ or Microsoft PhotoDNA https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna you can catch an astounding number of content shares, something which really looks great in press releases. My guess would be that the vast, vast, vast majority of those 18M reports that letter mentions came from automated engines detecting the "common" content with dumb bit or content hashes. But you really didn't do much except annoy and scare some online pedophiles* who shared the same 10, 20, 40 year old content* and maybe even put a few of them away for good or caused them to kill themselves. And yet, you almost never actually prevented ongoing child rape and other abuses. The problem however is that you absolutely cannot catch new content that way. Content that isn't widely shared but shared between two people or in rather small groups. These small groups have elaborate vouching and proving systems in place, and are hard to infiltrate, going as far as having to prove yourself by sharing a picture of you victim holding a sign with a time stamp and some passphrase. facebook aimed their AI at porn so they might catch some of this new content by accident; it's porn after all. Then again, my hopes aren't that high, seeing Microsoft's bing fucked up even removing the known child abuse content from search results even as the company runs the PhotoDNA database I mentioned before. And google's AI is "clever" enough to think black teens are gorillas, but they are supposed to catch child porn? Also, those dedicated child abuse content producers do read the news and know not to use facebook or twitter. Most of them do, anyway. There is essentially three types of pedocriminals I have observed: the aforementioned part time online pedos who share the same old child abuse content but do not actually produce it or abuse children in the real world, a group of active pedophiles who do not care about getting caught because they live in places where they do not actually have to fear the police investigating them, and a cautious group who might use common public services to make initial contact and talk a little in code but will immediately exchange tox ids etc or at least link to "how to setup qtox over tor"* guides (I saw a few "groomers" do just that), and never share anything incriminating over a public service. My guess is that they are be behind tor or a VPN even when using those public services. And there is a small number of dumb fucks too who will get caught easily and who end up in the press, and, of course, a probably quite large group of pedophiles who just abuse children but do not tell anybody about it, especially not on the internet. The majority of people you'd actually want to neutralize because they engage in ongoing child abuse you will not catch on facebook or twitter or whatever. I sometimes liken it to catching a lot of drug users, but not the dealers let alone the drug producers. Not that my experience dealing with the police is much better. The average time for them to get back to you is a few months if they are from the West, and usually never in other parts of the world, even if they have dedicated groups and/or tip lines. Maybe sometimes they do investigate without acknowledging the reports you sent, I don't know, but it is my feeling that probably not. I don't want to blame the individual police officers and detectives who have to deal with this shit, it's probably all due to lack of resources and institutional support. But to shit a bit more on the police: I saw some chats with essentially reverse-grooming, where a "girl" started off telling how "wet" she is and how much she likes dicks and only then that she is only "13". After learning about "her" age the dude then usually quickly leaves, but often only after having posted some personal information already. I later learned that such "girls" are either the police "child grooming" units, or blackmailers. I don't even want to know how many of the grooming cases that actually go to court are a result of this tactic, catching stupid horny wannabe pervs, while the actual groomers go free. Burning resources like this instead of using them to catch the truly evil and nasty people again seems to be a political decision. Source: personal experience from having to deal with this shit occasionally, and having caused some pedophiles to go to prison. * There have been a few studies that found that the group of people consuming child abuse media and the group that actually molests and rapes children doesn't have too much overlap, contrary to what one might expect. A lot of the "common" content is rather old, like digitized VHS tapes and magazine scans, or stuff like "1st st" (the producer of which went to prison like 10 years ago) or the East European nude "modelling" stuff from the early 2000s, or webcams from a time when webcams had half a megapixel. There is some newer stuff, like "Rbn" and "Rgirls", but that's the exception. And if you try to fill in the blanks and google any of this, you're stupid and/or evil; don't. * I have had a few tell that to my face, taunting me to report them to their local police. I am singling out tox/qtox because at least at in the recent past it seems to have been the goto tool for security conscious pedophiles, based on my personal observations.
lacker 7y ago
There has to be some nuance from the absolute privacy folks on this one. How do we balance the need to fight child abuse with privacy? This doesn't seem like a valid line of reasoning to me. Just because there are pros and cons to encryption does not mean that the right answer must be a nuanced balance where individuals cannot have completely private communication. The nature of encrypted messaging is that you don't really have a middle ground. Either there is a way for two entities to privately communicate, or all communications are inspected for content by a central party. I believe that allowing private communications is the option that is more suited to the American tradition of free speech.
kerkeslager 7y ago
> There has to be some nuance from the absolute privacy folks on this one. How do we balance the need to fight child abuse with privacy? I'll try my hand here: You're talking about this as "preventing child abuse" versus "privacy". The former is pretty concrete, while the latter is pretty abstract. But allowing government to invade everyone's privacy has concrete effects. Let's be clear here: in 2013, 2.2 million people were incarcerated in the US.[1] 1 in 5 is locked up for a nonviolent drug offense[2]--that's 440,000 people. And 540,000 people are incarcerated because they can't afford bail. It's unclear from the graphs on that page how many are locked up for child abuse, but the total incarcerated for rape and sexual assault convictions is 163,000 people in state jails, which by all accounts is where most rape and sexual assault convicts end up. It's unclear how many of those rape and sexual assault cases were convicted due to evidence from surveillance, but I think we can assume that it wasn't all of them. And in case it's not clear, quite a few of those unethically incarcerated will be raped[3]. In short, unethical incarceration is a much larger threat, by the numbers, than child abuse. We've seen time and time again that violations of privacy are publicized as being used to prevent terrorism and child abuse, but immediately are then used to prosecute nonviolent drug offenders. What it comes down to for me is that I trust average citizens to do the right thing more than I trust the law. The vanguard of law enforcement is average citizens picking up the phone and calling the police when they see something wrong, and I want to keep it that way. I trust the average person to call the police when they witness a crime like murder, rape, or child abuse, and not call the police when they witness drug possession. I do not trust law enforcement to enforce the law as ethically. Pervasive surveillance takes that power out of the hands of citizens and puts it in the hands of law enforcement who has time and time again shown themselves to be untrustworthy with that power. It's dangerous to be right when the government is wrong. And that's in the current US. In the past, US law has been even more wrong. For example, US law enforcement was used to suppress, for example, civil rights activists. That time may come again. It has always been beneficial to let illegal behavior slide under the gaze of law enforcement when that illegal behavior wasn't unethical behavior. [1] https://en.wikipedia.org/wiki/Incarceration_in_the_United_States https://en.wikipedia.org/wiki/Incarceration_in_the_United_St... [2] https://www.prisonpolicy.org/reports/pie2019.html https://www.prisonpolicy.org/reports/pie2019.html [3] https://en.wikipedia.org/wiki/Prison_rape_in_the_United_States https://en.wikipedia.org/wiki/Prison_rape_in_the_United_Stat...
rhizome 7y ago
>How do we balance the need to fight child abuse with privacy? Well, one place to start is with the realization that 99% of molestation comes from people the child knows. Encryption doesn't affect that pipeline at all, so the gov't shouldn't be worried. I'd also remind that probably effectively 100% of physical child abuse and violence also comes from people the child knows. Surveilling strangers can't have anything to do with dealing with any form of child abuse, except the extremely rare case where strangers have been able to find each other and create a subculture with which to provide evidence that exists on Facebook's servers. That's a pretty specific and narrow set of conditions. I'm not saying Facebook allowing government surveillance on their wires can't produce some hits, but I'd be extremely curious to learn of wider implications and costs that are proportional.

6gvONxR4sf7o 7y ago

I mean, you could always do an audit and see what percent of a thing your automated system catches. That definitely doesn't seem inherently unknowable.

salawat 7y ago
You're not quite getting it. You measure accuracy of detection via a training set of definitely known examples, and comparing it to the detections reported by your system. You can say with certainty that their system catches 99% of your training data set, but that doesn't mean it Will generalize to "all permutations of child abuse ever". That's two totally different things. In the end their use of that statistic illustrates an attempt to lie with statistics more than anything else.
- edmundsauto 7y ago
  By this standard, no system could ever say that it generalizes to all permutations. Especially when the definition of the measurement outcome (child abuse) varies.
  - salawat 7y ago
    Now you get it. You are now the thorn in the side of every AI salesman trying to pitch you his Neural Net based wonder machine, or politician trying to sell you on giving up rights because they need to be able to violate everyone's privacy because statistics. If you understand the actual nature of these measurements, it becomes quite a bit more difficult to let the "zomg neato" factor run away with you.
  - cwkoss 7y ago
    This is true. Until we implement omnipotent god-AIs, generalizable perfection is an unattainable standard.
- lozaning 7y ago
  Alex Pinto gave a phenomenal DefCon talk several years ago that really highlights this issue, "Secure Because Math". It gets into how modern ML/AI/Voodoo IPS systems essentially can't work because the training data or baseline is so impossible to accurately come up with. https://www.youtube.com/watch?v=TYVCVzEJhhQ https://www.youtube.com/watch?v=TYVCVzEJhhQ
- 6gvONxR4sf7o 7y ago
  If I say 15% of cars in San Francisco are Subarus, are you going to say that's impossible to know? Sure it may be true in my dataset, "but that doesn't mean it will generalize to all possible permutations of [traffic] ever." Well, no shit. But that's not the statement being made. If I wanted to know the accuracy of an important system. I'd do continual audits. Take a (stratified) sample of all things, get ground truth labels for whether they should have been detected, and see what percent were in fact detected. Then I could estimate how much my systems did and didn't catch at any point in time.
  - salawat 7y ago
    >If I say 15% of cars in San Francisco are Subarus, are you going to say that's impossible to know? No, I'd ask you what your methodology was when you measured. For instance, going to the DMV and getting a list of all registered vehicles, and using that as a representative dataset to reason from is fine. (Depending on your definition of cars in San Francisco, and whether or not that is meant to include cars transiently moving through or not.) Now if you told me you were using a Neural Network to recognize all Subaru models that drove by a particular camera, I'd start asking you questions, and want to see your training data, output, etc. >Sure it may be true in my dataset, but that doesn't mean it will generalize to all possible permutations of traffic ever. Well, no shit. But that's not the statement being made. That generally is the statement being made when people make claims about the accuracy of neural networks. Particularly the marketing people. You can only measure their accuracy within a constrained dataset, and overfitting is a thing. >If I wanted to know the accuracy of an important system. I'd do continual audits. Take a (stratified) sample of all things, get ground truth labels for whether they should have been detected, and see what percent were in fact detected. Then I could estimate how much my systems did and didn't catch at any point in time. Which, again means you're operating off things you know a priori to be the case. You don't know the full extent of everything to begin with, which phrasing in this article implies. I'm not saying audits can't be useful; it'll get you a number, and sometimes that's all you need. You just need to be clear about what the numbers actually mean. There's lies, damn lies, and statistics after all.

zzzcpan 7y ago

> There has to be some nuance from the absolute privacy folks on this one. How do we balance the need to fight child abuse with privacy? There isn't. This is a logical fallacy. Violating everyone's privacy is not the way to fight child abuse, not even one of the ways that can do anything to prevent it.

brunoTbear 7y ago
I'm not sure what the logical fallacy is you're referring to. I'm not advocating that we backdoor encryption. I'm asking what's the nuance in privacy vs crime prevention. I guess your position is one of an absolute, and therefore, to me, not especially interesting. I am curious what you think are the ways we can fight child abuse. It seems to me we have some obligation as technologists to see that our works aren't used to harm the most vulnerable.
- ilovetux 7y ago
  I believe the trade-off for privacy has traditionally been done on a case-by-case basis through the use of search warrants. I don't see anything wrong with the use of warrants. If there is reasonable suspicion the police can investigate further, it's sensible. Now we are seeing multiple instances where scapegoats and boogeymen are being used to systematically strip our rights away. The worst part of the whole thing is that when I attempt to fight for my privacy, I would be seen as supporting child pornography. It is an insidious tactic that is unreasonably effective.
- danShumway 7y ago
  > I'm asking what's the nuance in privacy vs crime prevention. This is the whole problem. There _is_ nuance. Users can still report content, even if it's private. Automated detection systems can still run client-side. Public forums like Facebook itself are still targetable, and we can increase funding and counseling for moderators in those public spaces to reduce turnover. Barr is the one here saying that, "mere numbers" aren't enough to talk about the problem. How exactly are we supposed to compromise with that? What exactly is a 'nuanced' response to the argument that any system that allows even one child to be abused is unacceptable? Barr isn't going to be happy unless law enforcement can read every single message. And he's going to trot out the same arguments every single time. And every time we respond, somebody is going to be jumping in to say, "but why can't we just have a little nuance? Why are all of you so dogmatic about this?"

claytoneast 7y ago

The sources in that article who lead these anti-child-trafficking/image organizations complain about cripplingly inadequate budgets and lack of human resources (employees). They do NOT complain about end-to-end encryption. They ask for more people, and more money, which the government is apparently consistently unwilling to grant them. Instead, the govt foists this absolute horseshit about limiting/backdooring e2e on us, while screaming THINK OF THE CHILDREN.

brunoTbear 7y ago
WFIW I didn't advocate backdooring either, I'm just asking how we, as technologists, balance what I think are reasonable and compelling goals: protect user data, and protect children. One reasonable answer is that FB/Google/etc. should donate or donate more (Google is called out as donating already) to these organizations. I think another reasonable answer is that technologists donate our time, perhaps as 20% projects or as sabbaticals to these orgs and help them modernize. It does seem Congress (per the NYTimes article I linked above) isn't doling out as much funding as we would like to see. And, in fairness, children are being harmed here, and at an incredibly alarming rate. Please do read the NYTimes article. The Times is, IMO, a fairly responsible actor in truth telling.
- claytoneast 7y ago
  I did read the article, though I think I may now see your point: this technology has greatly enabled abusers, perhaps more than it has helped finders-and-punishers-of-abusers. Your question is a good one: "how we, as technologists, balance what I think are reasonable and compelling goals: protect user data, and protect children."
  - 0_gravitas 7y ago
    "How do we, as postal workers, balance the reasonable and compelling goals: get people their mail, and protect children." One is our job that we are directly and morally responsible for (like a civil engineer being responsible for having a bridge not collapse at a load of >2 cars), the other is not. Call me callous, call me cold, but that is not explicitly up to us, we can't be under the delusion that we are the one key to everything and anything. It wouldn't make sense to have a "quick-release" button forcibly installed on all bridges that would cause an instant collapse if triggered "because countless children that are victims of human trafficking are taken over bridges everyday."

psychometry 7y ago

You can never know the exact number, but you can accurately estimate it. This is the entire point of statistical inference.

larkeith 7y ago

I read that as 99% of activity that is caught on Facebook is caught by Facebook's systems, with law enforcement only catching 1%.

brunoTbear 7y ago
That's still not an especially resounding success, or an especially relevant metric for us to understand the efficacy of FB's detection systems, and doesn't help us understand whether encryption of content is going to hurt hunting CEP. For example, they might be spotting a large fraction via network analysis, or other metadata approaches. Alternatively, they might just hash every image sent over messenger and match those to known CEP. Encryption screws up one avenue, but not the other.
- jandrese 7y ago
  All it says is that the cops are terrible at finding this kind of content on FB, which is to be expected since they have only a very limited view into the platform. The only stuff the cops are catching that FB isn't is when they knock down someone's door and grab their FB password to look for stuff the filters missed.
- akira2501 7y ago
  > encryption of content is going to hurt hunting CEP. Aren't we putting the cart before the horse? The problem is the exploitation of children, the secondary problem is that this exploitation is sometimes photographed and shared. I don't believe that a strong solution the latter problem will have any impact on the former.

4ntonius8lock 7y ago

Well, why bring privacy into the fray at all? I'm not saying online systems for catching child abuse can't be implemented. But is it really where our efforts would be most effective? The question should be: how do you combat child abuse? For that, we need: 1- Strong reporting systems that connect the people who in contact with children (doctors, teachers, social workers) and centralize and distribute that information. Right now, a social worker doesn't know about the doctor or teacher reports very easily. 2- Then, we need strong systems for handling the situation: i.e. social workers with effective systems for intervention. 3- Then, we need strong systems to take the children that are removed from at risk situations and place them in safe places where they can recover and grow in a healthy environment. This is the foster care system and the national health care system (or lack thereof) that will provide mental health counseling for the affected children. Given that 1, 2 and 3 of these systems are terribly dysfunctional if not mostly broken, I'm always skeptical when people talk about social media and child protection. The situation gives me a feeling of leaders justifying the erosion of privacy and those who are authoritarian inclined supporting them, rather than people really caring about protecting children.

scott_s 7y ago
The tension between privacy and child abuse is laid out in the linked NYT piece (which I also recommend reading): many of the images and videos of child sex abuse are created because they can be shared. They can be shared because the creators have some confidence they can do so without getting caught. Once the content is discovered, the authorities also have a difficult time finding that child, who may still be in danger. This situation is orthogonal to the (very necessary) steps you outlined.
- 4ntonius8lock 7y ago
  Is it? I just read the NYT article. Maybe I'm missing something, but it clearly states that the main issues saving children are a lack of enforcement availability (i.e. the 3 points mentioned in my comment) and the lack of timely cooperation of existing orders. Then they talk about how 'some criminals' hide behind encryption. So... we have a situation where many cases are reported but aren't followed up on. Yet we focus on getting more reporting by undermining the right of non criminals? Doesn't seem very logical to me. The whole piece even undermines their own arguments. They mention the 'case of the Love Zone'... which was cracked by investigators finding clues the old fashioned way. Not drag net surveillance. The article also places many issues that aren't tech related as tech problems. It says "Bing was said to regularly submit reports that lacked essential information, making investigations difficult" Let's compare that line to offline: If I own a pizza parlor where pedo's hang out sometimes, and I find a picture of CP in a restroom and hand it in to the authorities or call them up, would you blame the Pizza Parlor for the lack of fingerprints on the pictures? Would you think it's ok then to make wearing gloves illegal? I mean, if people use gloves, pedos can use gloves and hide their fingerprints. Why not target gloves? There is a real issue: Child Abuse. There are real solutions that are complex and extremely resource intensive. A report doesn't create a case. Focusing on tech and the lack of having MORE drag net surveillance seems absurd. This is coming from someone who grew up with abuse. When I go to forums to discuss abuse (it helps me deal with what happened) almost everyone tells of times that the abuse was reported but not well investigated. In fact, I've known people who work in CPS and they almost universally say the system is broken. A child in my city was recently beat to death by his step dad... an CPS had been called on them multiple times. We could be focusing on fixing that (you know, the things the people on the front line say are wrong). But no. Drag net surveillance is the thing we should focus on apparently (even though in the very article you mention it states we have more current reporting than we have resources to deal with it)

awaythrowacct 7y ago

It's easy to detect a vast amount of child porn, simply because the vast majority of child porn sharing is just a few thousand different files, shared over and over again. A few hundred thousand files if you count the non-nude and nude "child modelling" stuff, too. You can catch a majority of such content just by running a dumb hash over the bits, even though the detection rate suffered a little thanks to dumb smartphone users who will screenshot everything instead of sharing the files, thus creating "new" content. If you add some simple "content hashing", like https://pypi.org/project/dhash/ https://pypi.org/project/dhash/ or Microsoft PhotoDNA https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna you can catch an astounding number of content shares, something which really looks great in press releases. My guess would be that the vast, vast, vast majority of those 18M reports that letter mentions came from automated engines detecting the "common" content with dumb bit or content hashes. But you really didn't do much except annoy and scare some online pedophiles* who shared the same 10, 20, 40 year old content* and maybe even put a few of them away for good or caused them to kill themselves. And yet, you almost never actually prevented ongoing child rape and other abuses. The problem however is that you absolutely cannot catch new content that way. Content that isn't widely shared but shared between two people or in rather small groups. These small groups have elaborate vouching and proving systems in place, and are hard to infiltrate, going as far as having to prove yourself by sharing a picture of you victim holding a sign with a time stamp and some passphrase. facebook aimed their AI at porn so they might catch some of this new content by accident; it's porn after all. Then again, my hopes aren't that high, seeing Microsoft's bing fucked up even removing the known child abuse content from search results even as the company runs the PhotoDNA database I mentioned before. And google's AI is "clever" enough to think black teens are gorillas, but they are supposed to catch child porn? Also, those dedicated child abuse content producers do read the news and know not to use facebook or twitter. Most of them do, anyway. There is essentially three types of pedocriminals I have observed: the aforementioned part time online pedos who share the same old child abuse content but do not actually produce it or abuse children in the real world, a group of active pedophiles who do not care about getting caught because they live in places where they do not actually have to fear the police investigating them, and a cautious group who might use common public services to make initial contact and talk a little in code but will immediately exchange tox ids etc or at least link to "how to setup qtox over tor"* guides (I saw a few "groomers" do just that), and never share anything incriminating over a public service. My guess is that they are be behind tor or a VPN even when using those public services. And there is a small number of dumb fucks too who will get caught easily and who end up in the press, and, of course, a probably quite large group of pedophiles who just abuse children but do not tell anybody about it, especially not on the internet. The majority of people you'd actually want to neutralize because they engage in ongoing child abuse you will not catch on facebook or twitter or whatever. I sometimes liken it to catching a lot of drug users, but not the dealers let alone the drug producers. Not that my experience dealing with the police is much better. The average time for them to get back to you is a few months if they are from the West, and usually never in other parts of the world, even if they have dedicated groups and/or tip lines. Maybe sometimes they do investigate without acknowledging the reports you sent, I don't know, but it is my feeling that probably not. I don't want to blame the individual police officers and detectives who have to deal with this shit, it's probably all due to lack of resources and institutional support. But to shit a bit more on the police: I saw some chats with essentially reverse-grooming, where a "girl" started off telling how "wet" she is and how much she likes dicks and only then that she is only "13". After learning about "her" age the dude then usually quickly leaves, but often only after having posted some personal information already. I later learned that such "girls" are either the police "child grooming" units, or blackmailers. I don't even want to know how many of the grooming cases that actually go to court are a result of this tactic, catching stupid horny wannabe pervs, while the actual groomers go free. Burning resources like this instead of using them to catch the truly evil and nasty people again seems to be a political decision. Source: personal experience from having to deal with this shit occasionally, and having caused some pedophiles to go to prison. * There have been a few studies that found that the group of people consuming child abuse media and the group that actually molests and rapes children doesn't have too much overlap, contrary to what one might expect. A lot of the "common" content is rather old, like digitized VHS tapes and magazine scans, or stuff like "1st st" (the producer of which went to prison like 10 years ago) or the East European nude "modelling" stuff from the early 2000s, or webcams from a time when webcams had half a megapixel. There is some newer stuff, like "Rbn" and "Rgirls", but that's the exception. And if you try to fill in the blanks and google any of this, you're stupid and/or evil; don't. * I have had a few tell that to my face, taunting me to report them to their local police. I am singling out tox/qtox because at least at in the recent past it seems to have been the goto tool for security conscious pedophiles, based on my personal observations.

lacker 7y ago

There has to be some nuance from the absolute privacy folks on this one. How do we balance the need to fight child abuse with privacy? This doesn't seem like a valid line of reasoning to me. Just because there are pros and cons to encryption does not mean that the right answer must be a nuanced balance where individuals cannot have completely private communication. The nature of encrypted messaging is that you don't really have a middle ground. Either there is a way for two entities to privately communicate, or all communications are inspected for content by a central party. I believe that allowing private communications is the option that is more suited to the American tradition of free speech.

kerkeslager 7y ago

> There has to be some nuance from the absolute privacy folks on this one. How do we balance the need to fight child abuse with privacy? I'll try my hand here: You're talking about this as "preventing child abuse" versus "privacy". The former is pretty concrete, while the latter is pretty abstract. But allowing government to invade everyone's privacy has concrete effects. Let's be clear here: in 2013, 2.2 million people were incarcerated in the US.[1] 1 in 5 is locked up for a nonviolent drug offense[2]--that's 440,000 people. And 540,000 people are incarcerated because they can't afford bail. It's unclear from the graphs on that page how many are locked up for child abuse, but the total incarcerated for rape and sexual assault convictions is 163,000 people in state jails, which by all accounts is where most rape and sexual assault convicts end up. It's unclear how many of those rape and sexual assault cases were convicted due to evidence from surveillance, but I think we can assume that it wasn't all of them. And in case it's not clear, quite a few of those unethically incarcerated will be raped[3]. In short, unethical incarceration is a much larger threat, by the numbers, than child abuse. We've seen time and time again that violations of privacy are publicized as being used to prevent terrorism and child abuse, but immediately are then used to prosecute nonviolent drug offenders. What it comes down to for me is that I trust average citizens to do the right thing more than I trust the law. The vanguard of law enforcement is average citizens picking up the phone and calling the police when they see something wrong, and I want to keep it that way. I trust the average person to call the police when they witness a crime like murder, rape, or child abuse, and not call the police when they witness drug possession. I do not trust law enforcement to enforce the law as ethically. Pervasive surveillance takes that power out of the hands of citizens and puts it in the hands of law enforcement who has time and time again shown themselves to be untrustworthy with that power. It's dangerous to be right when the government is wrong. And that's in the current US. In the past, US law has been even more wrong. For example, US law enforcement was used to suppress, for example, civil rights activists. That time may come again. It has always been beneficial to let illegal behavior slide under the gaze of law enforcement when that illegal behavior wasn't unethical behavior. [1] https://en.wikipedia.org/wiki/Incarceration_in_the_United_States https://en.wikipedia.org/wiki/Incarceration_in_the_United_St... [2] https://www.prisonpolicy.org/reports/pie2019.html https://www.prisonpolicy.org/reports/pie2019.html [3] https://en.wikipedia.org/wiki/Prison_rape_in_the_United_States https://en.wikipedia.org/wiki/Prison_rape_in_the_United_Stat...

rhizome 7y ago

>How do we balance the need to fight child abuse with privacy? Well, one place to start is with the realization that 99% of molestation comes from people the child knows. Encryption doesn't affect that pipeline at all, so the gov't shouldn't be worried. I'd also remind that probably effectively 100% of physical child abuse and violence also comes from people the child knows. Surveilling strangers can't have anything to do with dealing with any form of child abuse, except the extremely rare case where strangers have been able to find each other and create a subculture with which to provide evidence that exists on Facebook's servers. That's a pretty specific and narrow set of conditions. I'm not saying Facebook allowing government surveillance on their wires can't produce some hits, but I'd be extremely curious to learn of wider implications and costs that are proportional.