5 ms·

Apple to roll out child safety feature that scans messages for nudity to UK

64 points by Larrikin 5y ago

kleiba 5y ago
To be honest, I find this optional(!) feature much more appealing than the use of the word "safety" in this context.
- fsflover 5y ago
  > optional(!) feature Until it's not. See also: https://news.ycombinator.com/item?id=28309202 https://news.ycombinator.com/item?id=28309202.
  - ribosometronome 5y ago
    Did you read this article? It's re: functionality that would notify a parent account if a child account had received or sent a message potentially containing nudity. It's a difficult thing to force upon folk who don't want it, as you don't have to mark accounts as being for children.
    - zmgsabst 5y ago
      Right — and how long until they (re)deploy CSAM scanning for all accounts, based on the same technology?
      - Someone 5y ago
        If we forbade all technology that could be used to do harm, we wouldn’t have much technology left, if any. Hammers can and have been used to kill people, as have axes, bricks, cars, electricity, fire, gas, ropes, water, etc. In this case, the question is whether you trust Apple to do what it says it does and not more. If you do, you can use the current iPhone. If you don’t, you can’t use any iPhone they ever produced.
      - nomel 5y ago
        > (re)deploy CSAM scanning for all accounts Isn't Apple already scanning, just in iCloud, like all the other cloud providers?
        theshrike79 5y ago
        Yep, it's pretty much the cost of doing business if you're a cloud storage operator. You either automate the search or your servers will get investigated by the FBI.
      - misnome 5y ago
        How does this help them do that, when they could just do it anyway?
        zmgsabst 5y ago
        By normalizing an idea they were previously rebuked for implementing (“we’ll scan all your photos for goodness!”) into a context people accept under “think of the children”. See how readily your sibling comments accept the premise and normalization of photos on a device being scanned as long as it’s appropriately contextualized — and then you wait five years, say you need to do even more to stop CSAM, and roll out the original scanning of all photos, everywhere.
      - ribosometronome 5y ago
        I don't believe it's based on the "same technology" in any meaningful manner. My understanding was that CSAM scanning was based on comparing perceptual hashes to known CSAM images. This doesn't seem like it would be particularly effective at preventing sexual images from being sent to minors because Apple doesn't have a collection of hashes every sexual image including ones yet to be taken. Apple's got a FAQ for this at https://www.apple.com/child-safety/pdf/Expanded_Protections_for_Children_Frequently_Asked_Questions.pdf https://www.apple.com/child-safety/pdf/Expanded_Protections_... that explicitly states: >These two features are not the same and do not use the same technology. but doesn't clarify the differences. Gotta imagine they're referring to the distinction I made because, again, Apple's probably not developed some sort of precognitive nudie hashing. Just spitballing but I'd imagine they're using some ML model trained to detect generic adult images, y'know, the same technology that they're using the make the Photos app more searchable. But detecting generic adult images has been around for ages.
      - browningstreet 5y ago
        Can we please differentiate between "CSAM" and "not CSAM"? They said they were going to deploy CSAM, and then they retracted (albeit temporarily). A lot of what made CSAM a concerning issue for many (including me) isn't in effect for this particular feature. Maybe they'll redeploy CSAM, maybe they won't, but this isn't CSAM.
legrande 5y ago
> Apple does not get access to the messages I routinely look at the network traffic of my iDevice and see a tonne of traffic being sent to Apple. I can't see the contents of the traffic, since 99% of it is encrypted, but I can safely say Apple is very cozy with your device. Just assume Apple can see everything, and change your behavior accordingly. If I want secure comms, I use battle-hardened things like Tails[0]. What is with the very low adoption of comsec tools like Tails? Does everyone think it's crimeware and only used by criminals? Because I do a lot of my surfing with Tor, but that doesn't mean I'm a darknet drugs kingpin does it? [0] https://tails.boum.org/ https://tails.boum.org/
- keddad 5y ago
  I'm a bit skeptical about Tails and other comsec tools which are a full OS instead of small, separate app. I don't have the time or knowledge to inspect full OS image in detail, and it feels like "secure communications, protected from the guys you don't like" is exactly the thing the "guys you don't like", like government, would create, with proper backdoors in place. After all, that had happened numerously before. (Crypto AG, that story about CIA gaining control of some "secure messaging" devices, etc)
  - cryptonector 5y ago
    Apps can be observed by the host OS, so if you're looking for an app that is secure relative to the host OS, then you're looking for unobtanium and instead need a host OS you can trust. OTOH, the big lesson of Clipper is that every OS, every app, every platform, can be hacked by nation state adversaries.
    - keddad 5y ago
      No, I'm looking for an app which will be secure, as in, lack backdoors or obvious security issues. I understand that OS can compromise it, but I more or less trust my normal OS, and it seems weird to use a "secured" OS, because that's where I'd expect all the backdoors to be. Not to mention the fact that if your threat vector is "Apple looking through you photos", you probably don't even need anything special in the first place - popular Linux disto with some Matrix client will be just fine.
niiinnneee9981 5y ago
I really am against these tools -- the cost is extremely high for a relatively small amount of perpetrators. And any pedophile will know about this and simply disconnect from the Cloud or do everything offline. ...meanwhile everyone else's nude/promiscuous/ all of your other personal photos will be scanned. it is a huge breach of privacy and it really makes me consider whether Apple is the company for me. apparently it is optional but that other furthers the idea - why would a pedophile accept this tool on their phone?
- tlhunter 5y ago
  You should read the article before replying
- brimble 5y ago
  This is about kids sending each other nudes, not the broader iCloud scanning thing that was all over the news a while back.
  - FrenchAmerican 5y ago
    Being a binational, I can't help noticing that, once again, as long as "nudity" is censored or "under control" - then the children are supposedly safe. Not that pornography is not a true problem. I really think it damages kids and even teens in their slow awakening to their sexuality. But if they receive pictures of teens or children killed in Ukraine with gory details, that's all fine? Kids and teens will share dirty pictures anyway. If their phones are totally "secure", they'll print the pictures. The true questions are, IMO: 1/ education: all parents must at an early age talk more freely about sexuality. About respect, boundaries, the difference between having sex and making love, etc, and tell the kids & teens the truth: there is plenty of pornography on the internet and this is not good for you. Because they will watch them. Through friends or unwillingly by trying to illegally download movies or cracked games. I'd prefer that psychologists set up officially authorized websites where teens could watch adults making love explicitly but respectfully, with romance and feelings. They want to watch anyway and they will find a way. But they will stumble onto violent scenes with psychopathic behaviors like incest, ten males onto one much younger woman. Of course parents ought to have a way to block those websites. But they could also parameter their teens' computers to watch things that psychologists chose for their age. Love movies to fight pornography. Explicit but insisting on the feelings and not the organs. If parents can't accept that, well... their kids will watch brothers and sisters making anal sex in a way presenting it as "normal". I think parents should enlist to night courses to learn how to deal with pornography. Censorship won't work. It looks that parents want to be able to lie to themselves, eg being able to say "we did everything in our power" while knowing perfectly well that their teens will be confronted anyway to porn. 2/ As societies, we should reflect on pornography itself. Why are we collectively allowing large corporations making loads of money by producing violent, devious, unrespectful, illegal even (under age, incest, violence) porn. I'm sure "free speech" is so large in the US that no law would hold. But that is not so in Europe - for the better IMO. Of course, laws would be transgressed like all laws are, but that would be a minority and that can be punished. I'm not a prohibitionist, but I don't understand why well-known companies are still allowed to shoot scenes were women are objectified to the point of dehumanization, where they are insulted, abused, raped (even if it is "fake"). I don't believe an instant that laws and cops will suffice to stop degradating porn to circulate. But at least, maybe 80 or 90% of explicit scenes will be "normal people making love" and not a bunch of 50-year guys destroying a young tern. So teens will more likely find "explicit decent love making" and not "gang bangs". At least, they would have a realistic idea of what sex is between non-pathologic, respectful, consenting adults. That's the only way IMO to prepare them for the inevitable, e.g. gore porn they will find if they really try to (using tor or whatever forums). At least, they will be educated enough to tell themselves that what they are watching is not "normal". Note that my definition of "normal" is rather broad. Oral sex is normal, in the sense of a common practice. But the ways it is practiced vary a lot. It can be either showing a woman following her own desire or a girl being forced to. Not at all the same thing. The importance is not the nudity or to see organs in action. It's all about the relationship, adults asking to each other if they feel like practicing this or that and being said "no" regularly. That's education. Of course, it's super difficult to judge when a specific teen is mature enough to watch a specific scene. There are gender issues too, which is a super delicate matter. In all cases, we are failing our children by not putting up laws that define boundaries to porn movies. It will be difficult to agree on those in Europe. But how can we make a difference if we don't challenge ourselves? The whole point is to make most of the porn something just respectful to women, with consent explicitly given and sometimes refused (and obeyed!) and with feelings. Nudity is not the issue. And yes, there should be punishment for sending unwanted nudes. In the US, I can't see a debate even possible with the current radicalization of the GOP around their so-called Christianity (which they know very little about or they would find Bernie not radical enough). Let's make sex great again, let's change pornography! And educate our children and teens according to their own maturity but before they watch gore porn. Any adults in this world??? Where are our political "leaders" about this? Yeah, we give the power to start wars and launch nuclear missiles to "adults" who don't dare step up to the reality of teens' sexual curiosity. Unbelievable. Don't look up! ;-)
Someone1234 5y ago
This is also available in the US, I enabled this on my kid's dedicated iPad linked to a family/child Apple account. There's no real downside, and it is entirely optional. I suspect some replies are confusing this with iCloud photo scanning (for e.g. CP), whereas this is a different feature entirely, they just happen to have been announced at the same time which caused a lot of people to treat them as a monolith. See: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ The biggest problem with Apple's family features right now is that it literally costs more to use. If you share a parent account with a child (or multiple), all IAPs are also shared, whereas if you create a child Apple account app purchases are correctly shared (and some Apple subscriptions) but IAPs are not, resulting in re-buys for every connected device. There is, in theory, some IAPs that share through family sharing, but they're very rare and there's no way to determine which are what before you buy.
- buildbot 5y ago
  Honestly as it is on device, it would be quite useful for people who receive unsolicited/unwanted photos of a sexual nature as well. It is a cool feature, where as the reporting hashes remotely is...not IAPs not being shared kinda makes sense from the perspective of the individual devs actually? The app is shared "purchased software", the account within the app probably is not.
  - Someone1234 5y ago
    I can see why devs don't check the IAP-share-for-family box, but ultimately the result is that fewer people are using child-specific accounts and the safety benefits as a direct result. I understand that devs have to make a living, but how many repeat sales do they expect to make for the same family? Regardless, it would be nice if the Apple app store told you if an IAP is shared or not within a family, it is important information for making purchasing decisions.
lnsru 5y ago
I am just curious about Apple watching my iPhone usage. Sometimes I tend to think, that Apple creates a profile as an applicant there just from phone usage. Apple knows about my not healthy relationship with the phone. Apple also knows about my extended lunch breaks walking with colleagues. Apple knows about my interests from photos taken with iPhone. Some days I just want to get some old used Nokia or Siemens C35 and hide from all this surveillance and profiling. I am also curious about iPhone demographics. I seriously doubt, that German teenagers get iPads and iPhones instead of cheap (and easily replaceable) androids.
- Bud 5y ago
  You're probably incorrect about German teenagers, who appear to have basically the same preferences as US teenagers, i.e., overwhelming preference for iPhone or the nicer tiers of Samsung devices. https://www.statista.com/statistics/700712/smartphone-market-share-in-germany-by-model/ https://www.statista.com/statistics/700712/smartphone-market... https://www.statista.com/statistics/828257/smartphone-ownership-teenagers-germany/ https://www.statista.com/statistics/828257/smartphone-owners...
  - lnsru 5y ago
    Thanks! Weirdly I see seldom Apple products in the hands of my German co-workers.
throwawaymanbot 5y ago
some-human 5y ago
This will sure stop the only 2 children in the UK that use iMessages? I'm not sure if this is going to be rolled into other apps or the camera app itself but the main app kids use to communicate in the schools in my local area are Snapchat, TikTok and Insta DMs, and they have Facebook messenger and WhatsApp for messages from their parents. I'm sure some of them aren't even aware there is a stock messaging app.

64 points by Larrikin 5y ago

kleiba 5y ago

To be honest, I find this optional(!) feature much more appealing than the use of the word "safety" in this context.

fsflover 5y ago
> optional(!) feature Until it's not. See also: https://news.ycombinator.com/item?id=28309202 https://news.ycombinator.com/item?id=28309202.
- ribosometronome 5y ago
  Did you read this article? It's re: functionality that would notify a parent account if a child account had received or sent a message potentially containing nudity. It's a difficult thing to force upon folk who don't want it, as you don't have to mark accounts as being for children.
  - zmgsabst 5y ago
    Right — and how long until they (re)deploy CSAM scanning for all accounts, based on the same technology?
    - Someone 5y ago
      If we forbade all technology that could be used to do harm, we wouldn’t have much technology left, if any. Hammers can and have been used to kill people, as have axes, bricks, cars, electricity, fire, gas, ropes, water, etc. In this case, the question is whether you trust Apple to do what it says it does and not more. If you do, you can use the current iPhone. If you don’t, you can’t use any iPhone they ever produced.
    - nomel 5y ago
      > (re)deploy CSAM scanning for all accounts Isn't Apple already scanning, just in iCloud, like all the other cloud providers?
      - theshrike79 5y ago
        Yep, it's pretty much the cost of doing business if you're a cloud storage operator. You either automate the search or your servers will get investigated by the FBI.
    - misnome 5y ago
      How does this help them do that, when they could just do it anyway?
      - zmgsabst 5y ago
        By normalizing an idea they were previously rebuked for implementing (“we’ll scan all your photos for goodness!”) into a context people accept under “think of the children”. See how readily your sibling comments accept the premise and normalization of photos on a device being scanned as long as it’s appropriately contextualized — and then you wait five years, say you need to do even more to stop CSAM, and roll out the original scanning of all photos, everywhere.
    - ribosometronome 5y ago
      I don't believe it's based on the "same technology" in any meaningful manner. My understanding was that CSAM scanning was based on comparing perceptual hashes to known CSAM images. This doesn't seem like it would be particularly effective at preventing sexual images from being sent to minors because Apple doesn't have a collection of hashes every sexual image including ones yet to be taken. Apple's got a FAQ for this at https://www.apple.com/child-safety/pdf/Expanded_Protections_for_Children_Frequently_Asked_Questions.pdf https://www.apple.com/child-safety/pdf/Expanded_Protections_... that explicitly states: >These two features are not the same and do not use the same technology. but doesn't clarify the differences. Gotta imagine they're referring to the distinction I made because, again, Apple's probably not developed some sort of precognitive nudie hashing. Just spitballing but I'd imagine they're using some ML model trained to detect generic adult images, y'know, the same technology that they're using the make the Photos app more searchable. But detecting generic adult images has been around for ages.
    - browningstreet 5y ago
      Can we please differentiate between "CSAM" and "not CSAM"? They said they were going to deploy CSAM, and then they retracted (albeit temporarily). A lot of what made CSAM a concerning issue for many (including me) isn't in effect for this particular feature. Maybe they'll redeploy CSAM, maybe they won't, but this isn't CSAM.

legrande 5y ago

> Apple does not get access to the messages I routinely look at the network traffic of my iDevice and see a tonne of traffic being sent to Apple. I can't see the contents of the traffic, since 99% of it is encrypted, but I can safely say Apple is very cozy with your device. Just assume Apple can see everything, and change your behavior accordingly. If I want secure comms, I use battle-hardened things like Tails[0]. What is with the very low adoption of comsec tools like Tails? Does everyone think it's crimeware and only used by criminals? Because I do a lot of my surfing with Tor, but that doesn't mean I'm a darknet drugs kingpin does it? [0] https://tails.boum.org/ https://tails.boum.org/

keddad 5y ago
I'm a bit skeptical about Tails and other comsec tools which are a full OS instead of small, separate app. I don't have the time or knowledge to inspect full OS image in detail, and it feels like "secure communications, protected from the guys you don't like" is exactly the thing the "guys you don't like", like government, would create, with proper backdoors in place. After all, that had happened numerously before. (Crypto AG, that story about CIA gaining control of some "secure messaging" devices, etc)
- cryptonector 5y ago
  Apps can be observed by the host OS, so if you're looking for an app that is secure relative to the host OS, then you're looking for unobtanium and instead need a host OS you can trust. OTOH, the big lesson of Clipper is that every OS, every app, every platform, can be hacked by nation state adversaries.
  - keddad 5y ago
    No, I'm looking for an app which will be secure, as in, lack backdoors or obvious security issues. I understand that OS can compromise it, but I more or less trust my normal OS, and it seems weird to use a "secured" OS, because that's where I'd expect all the backdoors to be. Not to mention the fact that if your threat vector is "Apple looking through you photos", you probably don't even need anything special in the first place - popular Linux disto with some Matrix client will be just fine.

niiinnneee9981 5y ago

I really am against these tools -- the cost is extremely high for a relatively small amount of perpetrators. And any pedophile will know about this and simply disconnect from the Cloud or do everything offline. ...meanwhile everyone else's nude/promiscuous/ all of your other personal photos will be scanned. it is a huge breach of privacy and it really makes me consider whether Apple is the company for me. apparently it is optional but that other furthers the idea - why would a pedophile accept this tool on their phone?

tlhunter 5y ago
You should read the article before replying
brimble 5y ago
This is about kids sending each other nudes, not the broader iCloud scanning thing that was all over the news a while back.
- FrenchAmerican 5y ago
  Being a binational, I can't help noticing that, once again, as long as "nudity" is censored or "under control" - then the children are supposedly safe. Not that pornography is not a true problem. I really think it damages kids and even teens in their slow awakening to their sexuality. But if they receive pictures of teens or children killed in Ukraine with gory details, that's all fine? Kids and teens will share dirty pictures anyway. If their phones are totally "secure", they'll print the pictures. The true questions are, IMO: 1/ education: all parents must at an early age talk more freely about sexuality. About respect, boundaries, the difference between having sex and making love, etc, and tell the kids & teens the truth: there is plenty of pornography on the internet and this is not good for you. Because they will watch them. Through friends or unwillingly by trying to illegally download movies or cracked games. I'd prefer that psychologists set up officially authorized websites where teens could watch adults making love explicitly but respectfully, with romance and feelings. They want to watch anyway and they will find a way. But they will stumble onto violent scenes with psychopathic behaviors like incest, ten males onto one much younger woman. Of course parents ought to have a way to block those websites. But they could also parameter their teens' computers to watch things that psychologists chose for their age. Love movies to fight pornography. Explicit but insisting on the feelings and not the organs. If parents can't accept that, well... their kids will watch brothers and sisters making anal sex in a way presenting it as "normal". I think parents should enlist to night courses to learn how to deal with pornography. Censorship won't work. It looks that parents want to be able to lie to themselves, eg being able to say "we did everything in our power" while knowing perfectly well that their teens will be confronted anyway to porn. 2/ As societies, we should reflect on pornography itself. Why are we collectively allowing large corporations making loads of money by producing violent, devious, unrespectful, illegal even (under age, incest, violence) porn. I'm sure "free speech" is so large in the US that no law would hold. But that is not so in Europe - for the better IMO. Of course, laws would be transgressed like all laws are, but that would be a minority and that can be punished. I'm not a prohibitionist, but I don't understand why well-known companies are still allowed to shoot scenes were women are objectified to the point of dehumanization, where they are insulted, abused, raped (even if it is "fake"). I don't believe an instant that laws and cops will suffice to stop degradating porn to circulate. But at least, maybe 80 or 90% of explicit scenes will be "normal people making love" and not a bunch of 50-year guys destroying a young tern. So teens will more likely find "explicit decent love making" and not "gang bangs". At least, they would have a realistic idea of what sex is between non-pathologic, respectful, consenting adults. That's the only way IMO to prepare them for the inevitable, e.g. gore porn they will find if they really try to (using tor or whatever forums). At least, they will be educated enough to tell themselves that what they are watching is not "normal". Note that my definition of "normal" is rather broad. Oral sex is normal, in the sense of a common practice. But the ways it is practiced vary a lot. It can be either showing a woman following her own desire or a girl being forced to. Not at all the same thing. The importance is not the nudity or to see organs in action. It's all about the relationship, adults asking to each other if they feel like practicing this or that and being said "no" regularly. That's education. Of course, it's super difficult to judge when a specific teen is mature enough to watch a specific scene. There are gender issues too, which is a super delicate matter. In all cases, we are failing our children by not putting up laws that define boundaries to porn movies. It will be difficult to agree on those in Europe. But how can we make a difference if we don't challenge ourselves? The whole point is to make most of the porn something just respectful to women, with consent explicitly given and sometimes refused (and obeyed!) and with feelings. Nudity is not the issue. And yes, there should be punishment for sending unwanted nudes. In the US, I can't see a debate even possible with the current radicalization of the GOP around their so-called Christianity (which they know very little about or they would find Bernie not radical enough). Let's make sex great again, let's change pornography! And educate our children and teens according to their own maturity but before they watch gore porn. Any adults in this world??? Where are our political "leaders" about this? Yeah, we give the power to start wars and launch nuclear missiles to "adults" who don't dare step up to the reality of teens' sexual curiosity. Unbelievable. Don't look up! ;-)

Someone1234 5y ago

This is also available in the US, I enabled this on my kid's dedicated iPad linked to a family/child Apple account. There's no real downside, and it is entirely optional. I suspect some replies are confusing this with iCloud photo scanning (for e.g. CP), whereas this is a different feature entirely, they just happen to have been announced at the same time which caused a lot of people to treat them as a monolith. See: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ The biggest problem with Apple's family features right now is that it literally costs more to use. If you share a parent account with a child (or multiple), all IAPs are also shared, whereas if you create a child Apple account app purchases are correctly shared (and some Apple subscriptions) but IAPs are not, resulting in re-buys for every connected device. There is, in theory, some IAPs that share through family sharing, but they're very rare and there's no way to determine which are what before you buy.

buildbot 5y ago
Honestly as it is on device, it would be quite useful for people who receive unsolicited/unwanted photos of a sexual nature as well. It is a cool feature, where as the reporting hashes remotely is...not IAPs not being shared kinda makes sense from the perspective of the individual devs actually? The app is shared "purchased software", the account within the app probably is not.
- Someone1234 5y ago
  I can see why devs don't check the IAP-share-for-family box, but ultimately the result is that fewer people are using child-specific accounts and the safety benefits as a direct result. I understand that devs have to make a living, but how many repeat sales do they expect to make for the same family? Regardless, it would be nice if the Apple app store told you if an IAP is shared or not within a family, it is important information for making purchasing decisions.

lnsru 5y ago

I am just curious about Apple watching my iPhone usage. Sometimes I tend to think, that Apple creates a profile as an applicant there just from phone usage. Apple knows about my not healthy relationship with the phone. Apple also knows about my extended lunch breaks walking with colleagues. Apple knows about my interests from photos taken with iPhone. Some days I just want to get some old used Nokia or Siemens C35 and hide from all this surveillance and profiling. I am also curious about iPhone demographics. I seriously doubt, that German teenagers get iPads and iPhones instead of cheap (and easily replaceable) androids.

Bud 5y ago
You're probably incorrect about German teenagers, who appear to have basically the same preferences as US teenagers, i.e., overwhelming preference for iPhone or the nicer tiers of Samsung devices. https://www.statista.com/statistics/700712/smartphone-market-share-in-germany-by-model/ https://www.statista.com/statistics/700712/smartphone-market... https://www.statista.com/statistics/828257/smartphone-ownership-teenagers-germany/ https://www.statista.com/statistics/828257/smartphone-owners...
- lnsru 5y ago
  Thanks! Weirdly I see seldom Apple products in the hands of my German co-workers.

throwawaymanbot 5y ago

some-human 5y ago

This will sure stop the only 2 children in the UK that use iMessages? I'm not sure if this is going to be rolled into other apps or the camera app itself but the main app kids use to communicate in the schools in my local area are Snapchat, TikTok and Insta DMs, and they have Facebook messenger and WhatsApp for messages from their parents. I'm sure some of them aren't even aware there is a stock messaging app.