27 ms·

On Apple’s “Expanded Protections for Children” – A Personal Story

• greatgib 5y ago
  Btw, now there is a very nice vector of attack to "swat" persons that you don't like: You create anonymous account, especially pretending to be underage, you send blacklisted pictures to the target. It will be automatically stored on the device of the victim at reception and maybe automatically uploaded to icloud for backup. Now you just have to wait for an automatic apple shit to trigger and have the company and cops ruin the life of your victim automatically. It is SaaS: Swat as a service :-)
  • samrolken 5y ago
    It seems like it isn’t possible to store images in someone’s iCloud Photo Library without their involvement. This would be a major security incident and immediately patched, if someone were able to find a way to do this.
    • jbarrs 5y ago
      Don't most messaging apps these days automatically save all photos by default? I believe this is the default setting on WhatsApp and even Telegram. Even if the photo won't be saved until the recipient opens the message, all that's needed is for them to forget about this setting. Then the photos are automatically uploaded to iCloud, also by default.
  • kilroy123 5y ago
    This is exactly right and my biggest issue with this. On my iPhone all images I look at on WhatsApp get automatically added to my photos, which in turn automatically get uploaded to the cloud.
• GistNoesis 5y ago
  These sort of databases and automated systems can easily be weaponized by anybody. Suppose your target post a picture from their iPhone to a social network. You just have to add this picture perceptual hash to the offensive database for bad things to happen to them. To do this : you grab their picture, overlay some offensive content on top rendered very transparently so that the new image is like 95% original + 5% offensive content. You also generate a second image with ratio 98% original + 2% offensive content. Then you post the pair of images on a public site that usually get scanned for offensive content. (You may even add the original image for good measure). Given that the offensive and original image can be exactly reconstructed, and that even at low transparency the human eye can still distinguish the offensive image it won't take long before all images get labelled as offensive. Their hashes added to the database, and a collision with the picture of your victim registered. Launch a script and do it for every pic they post. Of course there are plenty of variations of the above technique.
  • samrolken 5y ago
    It remains to be seen if this is possible at all… let alone “easily” done. Have any other big tech companies that scan for illegal images been shown to be vulnerable to this?
    • GistNoesis 5y ago
      For sensitive content you can store like nudity, you can use some neural networks trained on a dataset you can manually check and sanitize, and you won't be susceptible to this attack. But if you can't store the data and you are only allowed to store hashes, you become dependent of the quality of the hash database which is troublesome to verify, and therefore susceptible to poisoning. How hard the game will be for the various players is not really the point, given that it's a game that shouldn't be played : only Apple know the rules and can verify them, and not even Apple should have legal access to offensive raw database to check that it hasn't be poisoned. If they use some neural networks, it can easily turn into a PR disaster, once someone take their weights and use a Generative neural network to hallucinate offensive content based on the weight they release.
      • samrolken 5y ago
        Apple’s documentation discusses how they use a neural network to generate the bits which get turned into the hash.
• mrmikardo 5y ago
  Many thanks for posting this. It's a very well-written and timely piece.
• bengale 5y ago
  I don't understand why people are suddenly losing their minds over this. The technology has been in use by basically every cloud storage platform and social network for ages now. Where are all the stories of people jailed for false positives? Or even arrested and released? There aren't any because they don't arrest people on apple/google/facebooks say, I would assume they just open an investigation. I would also take a guess that people consuming this content tend not to have a single image that might flag this system, it must be extremely unlikely for someone to trigger two or more false positives.
• Lichtso 5y ago
  This is great, I can already see reverse ransomware. It does not remove desired data from your device, it adds undesired data and then deletes itself including any evidence that you are a victim instead of a criminal.
• Aissen 5y ago
  And then the next Pegasus will be used to plant CSAM instead of spying, just to burn someone publicly. Good luck getting to the forensic level of Amnesty International or Citizen Lab to prove anything.
• pharmakom 5y ago
  My next phone will not be an iPhone.
  • lovelyviking 5y ago
    Will it help? Imagine what other companies will do if Apple will get away with that...
    • pharmakom 5y ago
      I plan to get a high end Android phone and load an open OS onto it. TBH, I thought Apple would be a good steward for iOS. Oh well…
• rPlayer6554 5y ago
  I was seriously considering switching to Apple for my next phone because of all their privacy marketing. With this update, I'm not touching Apple with a 10 foot pole.
• faeriechangling 5y ago
  >False positives If this was a problem, would we not be seeing a slew of complaints about innocents being dragged through the mud with OneDrive and PhotoDNA? The only thing unique about Apples implemention is that it's client-side.
  • angulardragon03 5y ago
    Yes. The concern is reasonable, but given that this technology is widely implemented in basically every cloud storage offering, there isn't much to suggest that the described outcome will actually happen.
    • Croftengea 5y ago
      Mega anyone?
    • yreg 5y ago
      Do we know the described outcome is not happening with PhotoDNA, etc? In the story it didn't became a well known issue since it would happen only to few unlucky individuals. The same goes for the unintended asset freezing.
      • angulardragon03 5y ago
        Maybe anecdotal evidence, but I've read multiple posts/news stories about people affected by sanctions because of a shared name. I figure that if there was an issue where PhotoDNA had actually led to someone being arrested, that would have made headlines somewhere, at least here.
  • jx47 5y ago
    But there is a difference. Sure the same technology might be integrated into several cloud storage provider. But i can choose to use a storage provider or not. If Apple is activating this technology on your phone you can not opt out. Your phone will be searched.
    • 5y ago
• akomtu 5y ago
  This sort of automated crime detection systems will be reserved for peons, i.e. 99% of the population.
• napolux 5y ago
  I have a very common name too... I was an inch away to be denied a loan for my house because there was this guy with my same name, surname and date of birth that was in the CRIF (a sort of registry for people with very low credit score here in Italy). I know the feeling. And while we all agree about the fact the child porn should be stopped and people should go to jail for it, apparently "what happens on your iPhone stays on your iPhone" only until Apple says so. https://twitter.com/chrisvelazco/status/1081330848262062080 https://twitter.com/chrisvelazco/status/1081330848262062080
  • GordonS 5y ago
    I don't think my name is particularly common, but I got stopped while going through immigration in Australia a while back, and was quizzed in a private room by 2 men in black suits for quite a while - who I was, why I was there, who I worked for, that kind of thing. In the end, they thanked me for cooperating (they were actually quite friendly and polite in general, not like going I to the US, for example) and explained that they were looking for someone with the same name as me.
  • DoubleGlazing 5y ago
    I have an unusual, hard to spell surname. One time when I lived in the UK I found it hard to open a new bank account. After a bit of investigating I found out that I had been placed on the Credit Industry Fraud Avoidance System because different spelling of my name had been used in previous applications. This was seen as an attempt to get past credit checking, in reality the people entering my name in to their systems mis-spelled/mis-typed my name. As a result of someones carelessness I was flagged as being risky.
  • systemvoltage 5y ago
    Re: Twitter image link - This is why you should never ever trust any corporation and their marketing BS. PASS LAWS INSTEAD.
    • zimpenfish 5y ago
      > PASS LAWS INSTEAD. Do you suppose any government is going to pass laws which prohibits companies from scanning photos for CSAM either on-device or in the cloud? When half the time it's being done at the insistence of those self same governments?
• wayanon 5y ago
  Can’t the blockchain help with misidentifying individuals? As for the photo scanning isn’t it already happening?
  • skinkestek 5y ago
    > Can’t the blockchain help with misidentifying individuals? It can absolutely help with misidentifying individuals forever and ever.
  • fogihujy 5y ago
    Out of curiosity: How, specifically, would you like to solve this with blockchain tech?
  • qecez 5y ago
    You must be into Etherium
• rich_sasha 5y ago
  I wonder how Apple will fish out “child porn” from tons of innocuous pictures of babies, often mostly in the nude (children like to wear no clothes). Is their algo clever enough to understand context? I somehow doubt it.
  • mobiuscog 5y ago
    In theory they are simply matching against a known database of existing images, and as such your baby/family images will not be a match - they are personal images that have not been catalogued elsewhere. There is also meant to be human verification of any flagged images. In practice ? (Edit: This is for the iCloud scanning. The (i)Message warning around sexting is a whole other issue)
    • tjpnz 5y ago
      >There is also meant to be human verification of any flagged images. That shouldn't make anyone sleep soundly at night. The person at the other end has no incentive to not turn your life upside down.
    • lm28469 5y ago
      > There is also meant to be human verification of any flagged images. "Rest assured our system is safe, a random guy working in a digital sweatshop at the end of the world will get to see your naked baby pictures just to make sure you're not a criminal"
      • mobiuscog 5y ago
        This is definitely an 'added' risk they need to qualify - how they will stop any verification from leaking private images. One would hope that such checks should be minimal and therefore performed by suitably qualified (and equally audited) personnel. As you allude, it is more likely that such work is outsourced in the same way that the manufacturing of the phone is.
        • samrolken 5y ago
• gary17the 5y ago
  I don't know about you, but I just decided to switch from an iPhone back to a Nokia feature phone (where is mobile Linux when you need it?) and start backing out of Apple's ecosystem. I'm quite positive I can live without apps, just as I used to before Y2K. And, in case you're wondering, no: I have nothing to do with illegal adult imagery. [Edit #1] This article gets (at least) one thing right: when it comes to certain kinds of crime, an accusation is ALL it takes to destroy your life. [Edit #2] I used to laugh at all the "we have your password, send Bitcoin or..." spam. The first "you have a picture of a gray cat on your phone, send Bitcoin or..." message would give me a friggin' heart attack.
  • catmanjan 5y ago
    Which model did you choose? I'm looking at doing the same and I noticed Nokia has published the source code for the 8110 (I believe kaios runs ontop of Linux)
    • gary17the 5y ago
      I'll get a small folder with decent battery life - anything that's dumb as a fencepost and fits inside my pocket. I was getting tired of carrying a large holster on my belt, anyway.
  • c7DJTLrn 5y ago
    The only thing I'd struggle to lose is a map (Google Maps/Apple Maps). For me, it's essential if I'm going somewhere unfamiliar.
    • catmanjan 5y ago
      The feature phones do have google maps, but from what I've read they all take some time to acquire a lock (could be a software issue?)
      • withinboredom 5y ago
        Probably hardware. GPS either requires a bootstrap starting point or it has to completely solve the equations to calculate the location (slow!). Expensive phones triangulate via towers (fast!) and use that to bootstrap the GPS unit. Some GPS units don’t support that bootstrapping and are, thus, slower.
• visarga 5y ago
  Minority report style police based on AI embedded in iPhones? That's a killer feature! They could call it iSWAT or something. Let's see if fear of AI is greater than brand loyalty and convenience. An unstoppable force meets an immovable object. Apparently Apple has a team working as AI police. These people decide who gets reported. > While noting the 1-in-1 trillion probability of a false positive, Apple said it "manually reviews all reports made to NCMEC to ensure reporting accuracy." (https://arstechnica.com/tech-policy/2021/08/apple-explains-how-iphones-will-scan-photos-for-child-sexual-abuse-images/ https://arstechnica.com/tech-policy/2021/08/apple-explains-h...) Don't even get me started with the 1 in a trillion probability, we know adversarial images could be created that will trick the system even when you don't have access to the model.
  • dsign 5y ago
    One in a trillion? I wonder what assumptions went into computing that number, and I cheer for the "courageous" person who is bluffing it inside the company. Anybody in tech knows that whatever assumptions were valid yesterday are very likely to break today or tomorrow, and with it, whatever piece of software used it. Apple should know better, they have zero-day exploits regularly. And that's before even accounting for the political pressures. I can't precisely buy a new phone in a whim, but I'm not trusting my well-being to any "one in a trillion" in the marketing of a company.
    • withinboredom 5y ago
      One in a trillion. How many photos are taken every day on iPhones? I can assure you that it’s a non-zero number of “one in a trillion” at least once a week.
      • maxnoe 5y ago
        1000 Photos per iPhone per week sounds a bit much. Not arguing with the general sentiment though.
  • npteljes 5y ago
    >Let's see if fear of AI is greater than brand loyalty and convenience I don't think so - I think corporations and governments are getting better and better at PR and so, we'll see, slowly but surely, more and more features like this.
• high_derivative 5y ago
  Im really starting to develop a visceral hatred for Apple. They are talking sanctimoniously about privacy out of one end of their mouth, put on a family friendly face by banning anything offending prude sensibilities, while having no qualms about doing as much business as they possibly can in China. Now this. They are the ultimative dystopian corporation, I find them even worse than Facebook in that regard, at least Facebook has some interest in free speech and does not operate in China. I guess the reason I am so angry is that I like the devices and engineering marvels they can produce. But I cannot be a customer of this.
  • MichaelEstes 5y ago
    It's almost comical when you go back and watch their "1984" ad
  • culturestate 5y ago
    > at least Facebook has some interest in free speech and does not operate in China. That wasn’t really their choice, and they’ve been trying to find a way back into the market since the day they were banned. They still make billions of dollars every year on ads from Chinese buyers that run globally.
  • kmlx 5y ago
    > Most cloud services — Dropbox, Google, and Microsoft to name a few — already scan user files for content that might violate their terms of service or be potentially illegal, like CSAM.
    • marticode 5y ago
      There's an expectation that what you upload on the cloud will get some scrutiny, being on somebody else machine. But here they'll scan the local files on your own device, that you supposedly "own" (but not really, because Apple already locked it up to make sure you can only install stuff they have approved and taken a cut off)
      • kmlx 5y ago
• piokoch 5y ago
  First it will be to catch those pesky pedophiles. And terrorists, they fell out of fashion recently, but let's not forget about them. Then, obviously, we need to cut hate speech, homophobia, transgenderphobia, antifeminism, racism and all those bad, bad things. And fake news and misinformation, all those covid vaccination/lockdown skeptics in the first place. And at the end we will be allowed to talk only about the stuff that three or four corporations and their political associates are fine with.
  • cookieswumchorr 5y ago
    terrorists didn't fall out of fashion, they just need a break to recover. Hence the retreat from Afghanistan. You can't fight against terrorism if you run out if terrorists
  • user-the-name 5y ago
    https://yourlogicalfallacyis.com/slippery-slope https://yourlogicalfallacyis.com/slippery-slope
• zarzavat 5y ago
  Apple has gifted a golden gift to ransomware attackers. What's more scary than losing all your data? Getting you arrested.
• dvt 5y ago
  How can a company that openly deals with the Chinese Communist Party claim any kind of moral high ground? It just boggles the mind.
• bell-cot 5y ago
  Interesting - glancing through the 330-and-counting comments here, I see an enormous amount of harsh language about Apple, feelings of betrayal by Apple, promises to never buy another Apple product, etc. But virtually zero mentions of the U.S. Congress, President, judges, political parties, etc. The folks who are presumably behind this new Apple feature. (Does anyone see a "Add Evil Feature...Profit" story here, that does not require massive government arm-twisting?) IANAH (...Not A Historian), but I'll guess that if the well-educated citizens of a country do not believe that it is a real democracy - or feel that being politically active on issues that they care about is an unacceptable burden - then (at best) it very soon won't be.
  • Clubber 5y ago
    It smells like this is government influenced. They leverage they are using is the tech anti-trust circus going on right now and the "save the children," decoy they used for the drug war.
• viktorcode 5y ago
  Can someone clear this for me, why wouldn't the author sue the bank which locked their cash?
  • goodcanadian 5y ago
    I do not know the details, so I can speculate only. A lawsuit may yet happen if that is the only way to free up the money. However, a lawsuit is expensive and time consuming, so it is probably not the best place to start. Also, I don't believe the bank has any obligation to do business with a customer, so the most you could hope for would be to free up the funds. I believe the author would prefer to continue to be able to bank unimpeded (and switching banks won't help as the problem will follow unless and until it is cleared up). Finally, the bank has a pretty good defense. They are required by law to impose the sanctions, so prudence requires them to act first and ask questions later.
  • rrobukef 5y ago
    The laws enforce this behaviour. The terms of service in the contract you signed allows them to. He has no recourse.
  • astura 5y ago
    Because banks are 100% allowed to freeze transactions and freeze accounts, they have extremely broad leeway in who they do business with. This stuff happens all the time. Banks will even shutdown your account for suspicious activity and refuse to tell you what activity was suspicious.
• vmception 5y ago
  > It’s a common name, like Robert Brown or Mary Johnson. Shouldn’t it be easy to prove that I’m not person X? I bet its probably more like Abdul Mohammad and the 100 variations of spelling that in English edit: as anyone that gets OFAC list updates daily would notice
  • areoform 5y ago
    Even if it is, would that matter? Should we disconnect millions of people from the global economy because of their given name? Also, it’s hard not to read your comment and wonder if you hate a large proportion of the human species, > I bet its probably more like Abdul Mohammad and the 100 variations of spelling that in English
    • dapids 5y ago
      Dude, it's a name, chill out. Names have many variations, some more than others. Don't turn this into something it's not...
    • vmception 5y ago
      at first I was going to ignore this comment and all the leading questions, but then I saw you were OP. it wasn't a xenophobic or islamophobic comment nor was it condoning the reality or your experience with it, it was an observation from how you wrote your post and what I know about the world
      • areoform 5y ago
        Apologies for divining malintent. No, I don't have a name like that. And it shouldn't matter even if I have. If you read my name, you wouldn't think I'd be on a list. But here I am. The law of large numbers.
• Andrew_nenakhov 5y ago
  I wonder if the problem of a post author could be resolved by changing his legal name to some unique name? And, consequently, a sanctioned person could probably do the same to avoid sanctions.
• fredsir 5y ago
  Now we just need to place a photo on our opponents phones - journalists, politicians, someone whos job you want, you name it - and their phone will call the police itself and bring the proof too, and it's their burden to prove their innocence.
  • samrolken 5y ago
    An attacker who has a way to place multiple illegal images onto someone’s phone and into their iCloud Photo Library without their consent… it’s an interesting idea but how is it possible?
    • hoppyhoppy2 5y ago
      Not very long ago we would have said, "an attacker who has a way to remotely control someone's iPhone without their consent simply by sending a text message... It's an interesting idea but how is it possible?" Then we learned about NSO Pegasus and that it was not only possible, but was already being done to the phones of unsuspecting victims across the world.
    • fredsir 5y ago
      There’s the whole NSO Pegasus that answers for how regarding remotely. However social engineering is also possible. Drunk out in town makes an easy target for a skilled social engineer. All in all - it’s very possible.
  • azinman2 5y ago
    Planting something bad on someone was always possible. That hasn’t changed.
    • sschueller 5y ago
      yes, but now you won't have to bribe or pump up probable cause for a search warrant to find such plant. It makes it 100 times easier.
    • tjpnz 5y ago
• lovelyviking 5y ago
  The issue here has nothing to do with children, they would always use 'children' as excuse. The issue here is installation of Spyware Engine. The very fact of existence of such Spyware Engine has effect that each person would behave as 'someone is looking'. This means zero privacy. This means 'no privacy' feeling. Everyone is more 'careful' when people around. And this has prolonged effect on society's freedom, democracy and rights of individual because independent thinking happens when true privacy is there. Even a fear of false triggering the system and exposure of one's private thoughts/pictures/events to other people during 'sorting out false alarm' would make people behave differently. Even if they have nothing to do with any children the fear that their private moments can become public just because some AI would decide so would make each person think twice about each step. This is a huge attack on privacy of individual and should not be taken lightly. The effect will be also huge. Just imagine what they will do next if they get away with this. Just imagine what other companies will do once this is accepted for apple devices. I think this can/should also be considered as a fraud because when one bought apple device one was never told that spyware would be installed some day later. Edit: I think it should be illegal/made illegal to install any spyware on any device for any excuse without warant. Spyware is a Search of 'home' without warrant to some degree, isn't it? Done through the hands of some private company under some excuse... What about Fourth Amendment?
  • nine_k 5y ago
    > each person would behave as 'someone is looking'. They (that is, we) should already. We should assume being looked at, unless we can reasonably prove it's not so. In all public places, for certain. Using all public services, for certain. This decision erodes the trust in personal handheld terminals (quaintly still named "phones"). This is great that the announcement is made publicly. I can easily imagine a similar feature to be deployed tacitly in other countries and other platforms.
  • croes 5y ago
    It's always to protect children or protect from terrorists or organized crime. And it is always pretended that the bad guys can only exist on the client side and not on the supervisor side.
• villgax 5y ago
  I guess Framework laptop now needs to start with slim phones as well.
  • aembleton 5y ago
    That's pretty much the Fairphone https://www.fairphone.com/en/ https://www.fairphone.com/en/
• deleted 5y ago
  [deleted]
• villgax 5y ago
  Custom Android ROMs are bound to make a bigger comeback if Android manufacturers aren't going to take a stand against similar crap
  • Markoff 5y ago
    as someone who use custom ROMs entirely for 10+ years there is hardly any comeback possible, since they would need to be used in first place in past, how many phones had custom ROMs at peak 0.1% ? Plus good luck installing custom ROM without affecting functionality of phone or installing at all, Sony will mess your camera, Huawei/Honor doesn't allow it at all, Oppo/Realme will disable fingerprint sensor, Samsung will trip the knox and some features will never work again, so what's there left? pretty much only Pixel and Xiaomi, unless you wanna tablet only Pixel which is hardly distributed anywhere in world
    • celsoazevedo 5y ago
      Custom ROMs are like Linux. Only advanced users will use them. Regarding the functionality we lose when using a custom ROM, I think the trick is to use a device from a "modding friendly" brand. Samsung, Sony or Huawei are not modding friendly. Get a OnePlus, Asus, Xiaomi/Poco/Redmi or Google Pixel and most if not all hardware will work fine. (My experience is based using the OnePlus 3, OnePlus 8 Pro and Redmi Note 9S with LineageOS and a Asus Zenfone 6 with OmniROM. Fingerprint works fine and even more exotic features like the IR blaster on the 9S or the flip camera on the Zenfone 6 works.) Now, by replacing the original OS you also remove some things that makes the phone better. Eg: the stock camera app, which often gives you better processing (there are modded GCam apps that might help fix this, alternatively you can save DNG/raw files). One useful feature that I had on the Zenfone 6 that custom ROMs can't give me is the ability to stop charging at a certain % (can be done with Magisk, but it's not supported by the ROM itself). The main problem for me are the apps. Many require Google services and Google's safetynet and both are a problem if you decide to use "pure" Android. Even if one decides to install Google Apps to have the Play Store and their other services, there's still the safetynet issue that gets triggered just for unlocking your device.
• villgax 5y ago
  Tomorrow are all Macs(heck PCs) also going to come under the same ambit? How is one supposed to work on classified business or government tasks if a piece of software is actively gong to rummage through it all.
• Grustaf 5y ago
  The poster was mistaken for someone that was correctly sanctioned (at least in the legal sense). The risk she perceives with Apple is that they will falsely accuse people of crime. These are completely different situations. Not to mention how infinitely unlikely it is that _several_ of your photos would be similar enough to known abusive material to be flagged. >There aren’t enough people (nor PR capital) to look through all of your photos. This is a nonsense sentence. They ARE going through all your photos, both in this effort, comparing hashes, and for classifying your photos with ML, a highly publicised feature. Both happen on device.
  • mkotowski 5y ago
    > The risk she perceives with Apple is that they will falsely accuse people of crime. In my country, there was not so long ago a case of a person wrongly accused of murder[0]. He spent 18 years in prison, despite there being no convincing evidence of him committing the crime. The people invested in the case wanted to have a scapegoat ASAP. Cases related to pedophilia are often very emotionally-charged, given the nature of the crimes. It can easily lead to hastily made decisions that can ruin many lives. > Not to mention how infinitely unlikely it is that _several_ of your photos would be similar enough to known abusive material to be flagged. Given enough time and enough attempts, there still is a possibility of something like that occurring. And this is with an assumption that the algorithm used will be correctly implemented with no false positives at all – something that I doubt even a manual analysis by humans would be able to achieve given the volume of data to check. [0]: https://pl.wikipedia.org/wiki/Sprawa_Tomasza_Komendy https://pl.wikipedia.org/wiki/Sprawa_Tomasza_Komendy (in Polish)
    • Grustaf 5y ago
      My point was that the article has nothing to do with the Apple situation. Being confused for someone rightly accused of a crime is very different from being falsely accused. And I'm not sure what your main point is. Yes, sometimes people are wrongly accused, even convicted. Is that argument against having a legal system, or what are you trying to say?
      • 5y ago
• apples_oranges 5y ago
  At least these popups will install the knowledge in children, that technology is not trustworthy and secure by default. Better than making them believe otherwise.
• monkeydust 5y ago
  >> Random noise due to lighting conditions that takes an innocent photo (or, more likely, an intimate photo of you or your partner) and makes the algorithm assume that it is similar to a restricted photo. What algo is Apple using to compare 'similar' hashes from images, does anyone know?
  • Popegaf 5y ago
    Does Apple openly reveal such details about their software? Could revealing it make it possible to drive somebody to find out how to trick it?
  • n3mes1s 5y ago
    You can find technical details here: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
• villgax 5y ago
  With an M1 Mac or heck any soldered storage chip, you can't really be sure if data is swiped correctly. Whether on your own devices or on a second on device, before getting caught by stupid client side application with check for arbitrary documents.
• moogly 5y ago
  Anyone know how well these similarity neural nets deal with something as simple as, say, rotation? It would be trivial to generate 4 hashes for the 90° cardinal rotations (and, I guess 4 more for axis flips), but how about, for example, 11°? It would really suck if everyone starts implementing this and there's such a simple way to circumvent it.
  • samrolken 5y ago
    The neural net is specifically designed to handle such cases. This is covered in Apple’s documentation about this.
• jjgreen 5y ago
  Pre-internet, I used to go drinking with a chap called John, one of the pubs we went to was next to the main police station, so unsurprisingly full of plod. One night we walked into the pub and it fell almost silent; it was only later we found out why -- John was the spitting image for the just-arrested Peter Sutcliffe: the Yorkshire ripper.
• the-dude 5y ago
  In The Netherlands, it is common now for the police to report on a bust "also quite a substantial amount of cash was found". Sometimes, this is the only substantial allegation. Show me the man, I will show you the crime ( Stalin? Beria? )
  • jjgreen 5y ago
    Qu'on me donne six lignes écrites de la main du plus honnête homme, j'y trouverai de quoi le faire pendre. If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him. Cardinal Richelieu
• kmlx 5y ago
  time for some myth busting: 1. It's been turned on for all major cloud providers for some time. Upload something into google cloud/photos/dropbox? Yeah, the system's there as well. It just seems most people have been unaware of this so far. 2. It's only for the US. From my understanding Europe and ROW doesn't get this. At least for now. 3. I don't like this way of doing things, but I do understand that there's been major push from the US Government for this kind of tech. Basically, the tech companies had two choices: either add an actual backdoor, or provide another way of checking the content of all their users. They went with the latter. 4. If anyone has a problem with this solution, they should contact their US Representative, as they've been pushing this like crazy "for the children" :) 5. it's not scanning your local files. only the ones uploaded to apple photos. just like all the other actors.
  • tpush 5y ago
    > 2. It's only for the US. From my understanding Europe and ROW doesn't get this. At least for now. AFAIK the EU is actually pondering legislation to make proactive scanning for CSAM mandatory. Right now, in the US at least, proactive scanning like practically all cloud providers do is not required by law.
    • vaylian 5y ago
      The EU law was passed recently: https://www.patrick-breyer.de/en/chatcontrol-european-parliament-approves-mass-surveillance-of-private-communications/ https://www.patrick-breyer.de/en/chatcontrol-european-parlia...
      • kmlx 5y ago
        happy times :( i fear more governments will take this position now that the big ones managed to push it thru.
  • draw_down 5y ago
    > I don't like this way of doing things, but Yeah yeah. We know. Won’t someone think of the children.
• drvdevd 5y ago
  This article makes a good case for turning off every apple device you own.. and never turning them back on. And it's not because Apple has implemented this algorithm per-se, it's because the way they have implemented it punts the problems directly to Law Enforcement. Maybe they will (or have) proposed at least a manual review process after flagging before involving Law Enforcement. Otherwise it's tuttle/buttle all the way down...
• sschueller 5y ago
  "You have an illegal video called 'CollateralMurder.mp4' [1] and you have been reported to the authorities." How long until this is reality? [1] Collateral Murder is the war crime the US committed and was leaked via Wikileaks and is the reason Assange is still behind bars.
  • cwkoss 5y ago
    How can the public confirm that NCMEC doesn't already contain non-CSAM images of US officials engaging in illegal, compromising or embarrassing situations? Are there nonpublic images from Jeffrey Epstein's blackmail collections already in the database? Is there any system that could be developed to trustworthily prevent this situation without distributing child porn?
• chopin 5y ago
  What I don't understand is that Apple does seem to believe that this this has no possibility to open them up to very visible and expensive lawsuits in case of false positives. I'd assume if they hit the wrong person it would get ugly for them. Especially as civil rights organizations would love to support anyone who gets falsely accused if it was only for the publicity.
• camillomiller 5y ago
  Unpopular opinion here. I see the slippery slope, but I would say that this is the bare minimum price Apple is willing to pay to enforce privacy and encryption all over the ecosystem. They needed to give something to the governments to take the “think of the children” element from the broader privacy discourse. I think for example that the radical stance of Telegram on this is much more strategically detrimental to our broader right to privacy than this compromise Apple has settled with. That said, we need full transparency here. Apple’s statement that the probability of false flagging is one in a trillion is not enough and can’t be taken at face value.
  • cblconfederate 5y ago
    > needed to give something to the governments Why? do car companies feel also compelled to give your car's location to all the governments? There is no need for apple to do anything, it's a choice, they are doing tit-for-tat with governments. That's always nefarious
    • bengale 5y ago
      I thought modern cars did this already?
      • jbarrs 5y ago
        Perhaps not automatically, but many cars do save GPS data in their diagnostic data, which could easily be accessed if a case was brought against you.
      • cblconfederate 5y ago
        they do?
    • camillomiller 5y ago
      Because this is the biggest capitalistic entity in history, and the only risk it faces is governmental actions. When it comes to governmental action against privacy, that means undermining encryption for everyone.
• cblconfederate 5y ago
  > Apple has decided to scan images on your phone. I wonder what would be the defense to apple if they decided to scan the images on your desktop / laptop as well. Is there a limit to how intrusive tech companies can get?
  • layoutIfNeeded 5y ago
    Ummmm… I have bad news: “These features are coming later this year in updates to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey.” https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ :^)
• neximo64 5y ago
  I think I will selfishly say I couldn't care less about child abuse as much as my iPhone being entirely mine with only actors/people with my explicit opt-in consent to be able to look at. I find that perhaps the root of what Apple thinks is not many people think that way but it's not even something I have to think about: I know clearly that is enough to move off the Apple ecosystem. If anyone thinks or says otherwise they ought to think about what else they would agree to where the other person can do what they want to them without their consent.
• test6554 5y ago
  Just swap out the child abuse hashes for Tiananmen square photo hashes and january 6th insurrection photo hashes and you got yourself some pretty dystopian technology. Wow, come to think of it, will the hash database on your device change when traveling between countries? A popular photo of two women kissing on your phone could be perfectly legal in the US, but suddenly get you jailed if you travel through Saudi Arabia and forget to delete it.
  • ju-st 5y ago
    Hash database checks would be a privacy friendly alternative to custom agents physically accessing your phone when you enter a country. This would be more efficient as well, so checking 100% of travellers phones (and connected cloud storages) would be possible. A very effective way of protecting your country from IS terrorists!
    • vaylian 5y ago
      > A very effective way of protecting your country from IS terrorists! I sense irony. But I'll still reply: > Hash database checks would be a privacy friendly alternative to custom agents physically accessing your phone when you enter a country The border agent could always claim that their system detected something and therefore they need to look through all your private files.
  • nmeofthestate 5y ago
    > the hash database on your device Is there one? Reading a BBC News story on this, it sounds like checking is done when you try to upload an image to iCloud, so it's on the server.
    • jhayward 5y ago
      If you use an Apple phone as intended every photo you take is automatically uploaded to iCloud. It takes no usr action and does not imply an intent to share or distribute the photo.
    • fragileone 5y ago
• hungryforcodes 5y ago
  So HN seems to irrationally hate Bitcoin and crypto, but the entire first part of this story could have been solved with that.
  • hungryforcodes 5y ago
    The downvotes only prove my point.
  • mkotowski 5y ago
    On the other hand, using Bitcoin means that there is no realistic protection against theft and unavoidable mistakes. Maintaining a business without a way to correct mistakes and any kind of protection against scams seems to be hard to implement in practice.
    • hungryforcodes 5y ago
      Nice boilerplate. Did you read the article? The author in question got cut off from transferring funds due to whatever reason (a confusion in identity at this point we understand). Funds could have easily be transferred to him via crypto.
      • mkotowski 5y ago
        Yes, I read the article. All I said is that while Bitcoin resolves some issues, its use presents many more.
        • hungryforcodes 5y ago
          I am detailing a solution to the author's problem. And you...?
  • JohnJamesRambo 5y ago
    Ctrl-f crypto, ah there it is. The solution to their banking madness ignored on HN and grayed out, of course.
• bArray 5y ago
  Just a quick reminder: * Microsoft scans and deletes stuff on your drive it believes is harmful [1]. There was also some indication a while back that they would delete 'pirated content' - i.e. anything some algorithm detects you "shouldn't have" [2]. * Google Drive have been deleting 'pirated content' for quite a while based on hashes [3]. I imagine other cloud services do this as well [4] and I recommend you do not sync your important files with such a service. This was always on the cards. Next it will be 'terrorist materials', then it will be 'harmful content' and soon simply 'DMCA content' stored locally will be reported to the police. Freedoms definitely erode if not defended. On a related note, projects like Pine64's Pinephone are early stages (read: difficult to daily drive) but definitely in the right direction [5]. Some day in the near future you will be able to get a decent mobile experience without having to worry about spyware/adware/malware. [1] https://news.ycombinator.com/item?id=27914752 https://news.ycombinator.com/item?id=27914752 [2] https://www.indiatoday.in/technology/news/story/windows-10-will-automatically-delete-pirated-software-288310-2015-08-14 https://www.indiatoday.in/technology/news/story/windows-10-w... [3] https://torrentfreak.com/google-drive-uses-hash-matching-detect-pirated-content/ https://torrentfreak.com/google-drive-uses-hash-matching-det... [4] https://www.extremetech.com/computing/179495-how-dropbox-knows-youre-a-dirty-pirate-and-why-you-shouldnt-use-cloud-storage-to-share-copyrighted-files https://www.extremetech.com/computing/179495-how-dropbox-kno... [5] https://wiki.pine64.org/index.php/PinePhone https://wiki.pine64.org/index.php/PinePhone
  • phendrenad2 5y ago
    > There was also some indication a while back that they would delete 'pirated content' - i.e. anything some algorithm detects you "shouldn't have" [2] You should read the linked article more carefully. It makes an incorrect conclusion. IANAL, but if you read the EULA lines carefully, you'll see thst they're saying, basically, "We might auto-update software, which may incidentally break your pirated version, and we're sorry and don't want to get sued if this happens to you".
    • bArray 5y ago
      Ah, likely the wrong article. I have tried to find one that did state this but it doesn't surface now. I do remember very clearly that Microsoft were deleting pirated version of Office on purpose and this EULA basically said "oops, we told you it might happen".
• MightyOwl13 5y ago
  This is a serious situation that has been unfolding for years now in the world of fraud in general (from both the retailers and the banks' perspective). I used to work in a company doing fraud prevention. The first thing that shocked me when I started in the role, was the sheer indifference people working there had for rejected transactions. The way these were supposed to be handled was like this: 1. Transaction is rejected because of a rule/condition/bank decline 2. Investigate transaction manually and identify root cause 3. Contact customer and provide information on how they can resolve this (ask for more details if you are unsure, mark them as safe to allow transactions as it was a clear false positive or ask them to speak to their bank or payment provider as the system wasn't the one that rejected the transaction) The general view was though that if the customer think it's important, then they should get in touch with the company and there shouldn't be proactive work done on these transactions since chances are "most of them are dodgy". The problem was that there were thousands of these every single day and a very small team working in that department - less than 10 people. I voiced the fact that even if someone is fast at identifying these and going through them (once you developed experience in identifying the cause of the decline and doing the administrative work of allowing the user to complete transactions, emailing them and then keeping an eye on their account for when they try to make a purchase) they couldn't work through more than maybe 2-300 transactions in a day. That was 2-300 of someone working ONLY on this task, almost automatically and having a deep understanding of what happened in each of them. Realistically it was more around 50-100 transactions per day. The rejection list could have up to 3-4000 entries per day. I argued for improvement of the rules that were pushing rejects into that queue, I argued for better analysis to be completed before a rule would be added, I argued for accepting a higher risk but allowing lower value transactions through to reduce friction and make the queue more manageable. I ended up getting some of the things I was asking for, but what happened was that the rejections went down to 1-2000 per day and people just assumed now that those are fraudulent for sure since now we're "better at spotting criminals"... The introduction of another system that employed machine learning made fraud agents even more indifferent towards those queues since now "you can't fool AI". It was a very sad state of affairs and since I left there I seriously doubt that this has improved in any way. I remember the email chains you would get from some users who tried to make important purchases or even just regular purchases and not being able to get in touch with a human. I remember the frustration, the friction and the blanket statements that would emanate in team meetings saying that "it's fine as long as we protect the company from losses, so what if a few people have issues"... Those few people could have been them in other scenarios. I think this will only keep getting worse and worse with time. Each new software that attempts to bring "efficiency" to some of these types of tasks can cause a huge array of problems downstream. You end up with companies that will have 5 people working in a department that should have 500 because machine learning will "take care of the issue". I know I haven't even touched the privacy concerns that this raises, and there are PLENTY and covered in great detail in this post and in the one made by the EFF, but we're forgetting here that those affected by these things are other people. And as long as governments will say "You need to regulate this SOMEHOW" and companies will come back with "We'll use the power of AI and ML to train NN to easily identify any problems and thus bring efficiency and safety to all of our customers". We're people, we're not machines and we have a vast array of personal circumstances and elements that make us unique and saying that a blanket solution will "fix all problems" is absolutely inhuman and shows a crass misunderstanding of how these tools work and what their unintended consequences are. TL;DR - using ML/AI/NN will cause a huge array of problems. We need more people working these queues which increase exponentially so that innocent users do not get caught in your "high-tech solution". For each criminal fraud prevention rules catch, a few hundred innocent people cannot complete their activities or are labelled as criminals until proven innocent.
• stefanve 5y ago
  Seems that most comments are on the privacy side of the argument. So I will try to defend this :) "Apple’s system is less invasive in that the screening is done on the phone, and “only if there is a match is notification sent back to those searching”, said Alan Woodward, a computer security professor at the University of Surrey. “This decentralised approach is about the best approach you could adopt if you do go down this route.” So it will only happen when you upload data to the iCloud. So the question is then: is it okay for apple to make sure no illegal content is stored on its servers? I think you could argue both ways but to me at least it is less privacy invasive than it seems. it is about expectations of privacy, in the same way you can expect less privacy in a public space in the real world. It should be clear by now that you can expect less privacy on the public web. Of course a counter argument is that is is perhaps more like renting a hotel room than it is a total public space. But you can also expect less privacy in a hotel room than you can expect in your own home. For many years privacy and freedom of speech has been a hallmark feature of the internet. I was (kinda still am) a big proponent of this. But we have seen what will happen if you take it to its extreem. From disinformation campaigns to illegal activities. So to me (and incidentally Facebook) we should get better (inter)national rules about how we want to use the public space of the internet. Just in the same way we have rules and laws in public space. Internet is blending more and more in to the real world. So governments should make it clear in what way we can expect privacy and free speech and where we can expect it (self hosting, public platforms etc) and what responsibilities companies have to enforce it. The hardline arguments of privacy advocates are the same that the NRA is using to prevent any form of gun control. It starts with A what is them going to prevent to do B. Well rules, regulations and laws. You should be very aware that privacy isn't watered down and law enforcement should make sure they don't mis use what rules there are. But to me checking hashtags for know child abuse photos is the same as doing a basic background check before giving someone an assault weapon That being said it will stop no one from encrypting images before uploading it to iCloud.
  • rdedev 5y ago
    The problem with client side verification is it dosent take much have the system look at all local files for verification, not just the files being uploaded to iCloud. Now that Apple has some form of client side verification, an authoritarian government with enough economical clout can preassure apple into extending the system to look for things beyond child pornography. The govt could even just provide apple with their own hashes which may or may not be CP and mandate that Apple verify these hashes too. So in the end it all depends on how well apple is ready to push back against such government and if they are willing to take a hit to their profit margins by being effectively banned from a country
• tpush 5y ago
  I recommend people watch [0], a discussion of Apple's CSAM scanning by Alex Stamos, Matthew Green, Riana Pfefferkorn and David Thiel. Gives a well-balanced overview of what's happening and what the trade-offs are, IMO. [0] https://www.youtube.com/watch?v=dbYZVNSOVy4 https://www.youtube.com/watch?v=dbYZVNSOVy4 Edit: tracking -> scanning
• nwellnhof 5y ago
  Reminds me of the German novel QualityLand, where rich people give their children names with deliberate misspellings just to avoid cases like that.
• nikolayasdf123 5y ago
  At this rate Apple is no better than Xiaomi or Huawei: using forced labor to make their phones; produce phones in China; biggest market is China; checking your phone for violated content. Maybe time to switch to Xiaomi, it is 3x cheaper and better tech.
  • layoutIfNeeded 5y ago
    Better tech, not really. Otherwise it’s accurate.
  • plaidfuji 5y ago
    > biggest market is China When I first read this I thought it was impossible. But it seems like, while Apple’s total revenue from the US is double that of China, actual iPhone sales volumes may not be as far apart. The numbers aren’t publicly available, but Bloomberg has an estimate as of 2017 showing US iPhone sales of 70 million vs 50M in China: https://www.businessinsider.com/apple-iphone-sales-region-china-chart-2017-3 https://www.businessinsider.com/apple-iphone-sales-region-ch... It looks like iPhone sales were going down in China as of then, but their GDP has also grown rapidly in recent years and I’ve seen other charts showing Apple’s China revenue basically increasing in concert with Chinese GDP, so it may not be a stretch to say that China is the biggest iPhone market today. Whoa.
• deleted 5y ago
  [deleted]
• flippinburgers 5y ago
  What happens when parents take photos of their children?
  • npteljes 5y ago
    The algorithm is supposed to check for known hashes. So, a new photo shouldn't trigger it.
• tvirosi 5y ago
  Not long until people will use deep learning style techniques to modify images to falsely trigger this alarm on harmless pictures and strew them randomly on the web.
• john_minsk 5y ago
  Hmmm. So after past 5 years posing as best privacy-focused company out there we get this? Major red flag. I want my home server to work as my personal cloud: music, photos, movies, files collections. And everything should be end-to-end encrypted so that no one has any way to look inside it. With 5G and broadband this might even work without much optimization. Someone knows about such solution?
  • fragileone 5y ago
    Nextcloud
• 100011 5y ago
  In about less than 4 years it will be shown this was used for completely other purposes against political enemies, but accepted because they were your political enemies. And then it will be used against you, after which there's an outrage thread and "how could this have have happened???" posts.
• 74d-fe6-2c6 5y ago
  I'm curious about what sort of hash algorithm they intend to use. Just some SHA hashing or is it some spatial hashing where you can calculate a meaningful difference between hashing and maybe even infer content. I'd assume this might be interesting with regard to building a profile for choosing relevant ads.
• marcus_holmes 5y ago
  I don't understand how this is supposed to do anything except generate false positives. If someone takes a new picture of a child being abused, that won't be on the database so won't get flagged. Are they expecting child-abusers to take photos of existing photos of children being abused? Or are they hoping to catch abusers sending each other known photos of child abuse unencrypted? Because that's: a: stupid (the even-remotely-smart abusers will encrypt their abuse) and b: rife for mis-use (I, using a non-Apple device, send your Apple device a photo of abuse. You're now flagged as an abuser and your life shuts down) As others have said, I don't think this has anything to do with child abuse, because it plain doesn't work as a method of preventing child abuse. It totally works for flagging "politically unacceptable" images, though. Maybe the CCP got fed up of all those memes?
  • donkeyd 5y ago
    I'm surprised how shocked people are about this new thing from Apple. Both Google and Microsoft are already doing this stuff and have been since 2014[0][1]. How come Apple is getting such a huge amount of hate for this, but nobody even seems aware of the others already having implemented such technologies. I also hope nobody here ever gets mad about the rampant child pornography going around on platforms like Kik, because that would be incredibly hypocritical. 0: https://www.theverge.com/2014/8/5/5970141/how-google-scans-your-gmail-for-child-porn https://www.theverge.com/2014/8/5/5970141/how-google-scans-y... 1: https://nakedsecurity.sophos.com/2014/08/10/microsoft-scans-email-for-child-porn-images-leads-to-arrest%E2%80%8F/ https://nakedsecurity.sophos.com/2014/08/10/microsoft-scans-...
    • hu3 5y ago
      It has to do with the way Apple advertises being a herald of privacy. Also your links say that Google and Microsoft scan e-mails on the cloud. Not phone storage as in Apple's case.
      • donkeyd 5y ago
        > Also your links say that Google and Microsoft scan e-mails on the cloud. Not phone storage as in Apple's case. Another confirmation that Google and MS have access to the data you store in the cloud and Apple does not. It's very unfortunate that scanning this locally is viewed as a negative in stead of a positive.
• supperburg 5y ago
  This is so awful. I trusted Apple because of their stance on privacy — I felt as though I could trust them so I bought into their ecosystem and now my iPhone is basically the nerve center of my life. I keep everything on my iPhone and access all my primary services through it. All with the confidence that all I have to do is push the lock button five times and all of it is locked away behind a very long diceware pass-phrase and that nobody, not even the government, could pry into it. So reading all the headlines tonight was like a physical punch to the gut. All of a sudden I’m tasked with migrating all my shit somewhere else? I literally don’t even know where to begin… it’s going to take me weeks to research everything and find a new device. I don’t even know if there is an alternative device. And no matter what move I make next, it’s going to be a downgrade in hardware specs. What a bummer. And here’s another thing. In the OP, they make it sound as if the imaginary mark was a female. We all know that females won’t be affected in any way. Most females will probably not even care about this. Because they don’t know what it’s like to be nervous around children or to have to make sure you are never alone with children even by accident or happenstance because everyone assumes that you are a pedophile, rapist kidnapper. Leery looks from adults all the time whenever there are young girls around. It’s a uniquely male experience. It’s also uniquely male to have your guts cut open and ripped out by your cell mate. When people start being brought in for a hash match and then they find incedental images of young girls that look like they might not be 18, innocent MEN are going to die.
• Crontab 5y ago
  If it is really only to be used when iCloud Photos is turned on, why not do the hashing on Apple's servers when the photos are uploaded to the service? It is my understanding that this kind of hash-checking of files is a pretty standard thing on major cloud storage services. I can only think of two reasons for this: - for a future expansion into files that are not uploaded, even with cloud services turned off - Apple doesn't want to foot the energy costs of doing it Finally, it should be noted that this will also apply to macOS Monterey, not just iOS/iPadOS.
  • macintux 5y ago
    Third possibility: Apple is going to E2E encrypt iCloud photos, but the only way the U.S. government will allow it while still complying with CSAM laws is if the photos to be uploaded are scanned on the device.
    • fragileone 5y ago
      There are no CSAM laws that require this sort of privacy violation, and it would be a gross overstep by any government to mandate it. If they enabled E2EE iCloud after this it'd be just for marketing purposes, because client-side encryption is being circumvented it would render the whole promise of privacy as a lie.
      • Terretta 5y ago
        > client-side encryption is being circumvented Client side encryption of what? A plain photo? That you designate to upload to cloud? At which point it's encrypted, or hash-checked then encrypted. Where's the circumvention?
    • rasz 5y ago
      https://blog.elcomsoft.com/2021/01/apple-scraps-end-to-end-encryption-of-icloud-backups/ https://blog.elcomsoft.com/2021/01/apple-scraps-end-to-end-e...
  • starbugs 5y ago
    > why not do the hashing on Apple's servers when the photos are uploaded to the service? Their reasoning seems to be that it would break E2E encryption.
• andi999 5y ago
  So this guy is probably having a life altering problem, he might never have money again, and he worries about Apple? Is this second part just to attract attention to his peril? Or am I reading something wrong here.
• i_have_an_idea 5y ago
  "won't someone please think of the children" ..
• donkeyd 5y ago
  This post starts out alright, but the idea that police will raid your house without even requesting a copy of the picture that triggered the system is outright ridiculous. A better (and more related to his issues) scenario would be Apple automatically kicking you off their platform and you losing access to all your data, including your passwords which you stored in iCloud Keychain. Because that is an actual realistic scenario.
  • bondarchuk 5y ago
    Yeah, they would never do that... https://en.wikipedia.org/wiki/No-knock_warrant#Controversy https://en.wikipedia.org/wiki/No-knock_warrant#Controversy Key quote near the bottom: Due to errors or acting on bad or faulty tips without double-checking information, Chicago Police Department has raided many wrong addresses. So yes, raids without double-checking the information (i.e. looking at the picture that lead to the tip) has been happening quite recently.
• filipeteles 5y ago
  The irony is the Big Tech giants rub shoulders with known pedophiles (Bill Gates + Jeffrey Epstein). George Orwell was right, he was just not enough pessimistic about how bad things would get.
• tamaharbor 5y ago
  I have been telling Alexa all along that “Biden is not my president”. Do you think I am in trouble? (Please don’t downvote for a joke).
• SkipperCat 5y ago
  I ask this question in earnest - what's in this for Apple, why is Apple doing this? I haven't seen anyone who likes this direction taken by Apple. Why would they do something that has the potential to detract from their product demand. Apple always said "We're different - we sell you our products, we don't sell your data." Now, they're giving away the data and for what? Child safety - I don't buy it. I remember when Apple refused to help law enforcement unlock the iPhone of a mass shooter is California. Why the change of tact now?
  • nmeofthestate 5y ago
    > why is Apple doing this? They don't want child pornography stored on their servers?
  • kleene_op 5y ago
    Prepping up to meet China's expectations?
  • Miner49er 5y ago
    Is it maybe due to FOSTA-SESTA? Maybe they're worried about being liable for crimes under that bill?
• nmeofthestate 5y ago
  >The company claimed the system had an "extremely high level of accuracy and ensures less than a one in one trillion chance per year of incorrectly flagging a given account". >Apple says that it will manually review each report to confirm there is a match. It can then take steps to disable a user's account and report to law enforcement. They say all reports will be manually reviewed, so the blog post, while painting a scary picture, doesn't really paint a picture that matches the reality of how this works - ie automated systems flagging up with no human intervention. I know the blogger tries to illustrate how human intervention isn't enough, but in this case we're talking about a human just comparing an image with another image, not a hypothetical lazy legal department "playing it safe" instead of doing due diligence.
• caf 5y ago
  I don't buy the equivalence here. Notice that while the author was blackballed by an arse-covering bank, they weren't actually arrested for sanctions-busting. This is broadly because you need a bit more than "appears to match entry in database" before you get to probable cause for an arrest.
  • nmeofthestate 5y ago
    In the same way neural nets can be fooled into identifying images wrongly, I think nerd brains can be fooled into making false comparisons between non-comparable scenarios, especially as soon as "privacy" is involved.
• bsenftner 5y ago
  It is time to give up mobile phones. The human race is simply too immature to provide such a technology without mass abuse, corruption, and generation of misery at scale. Seriously, how many spam calls a day to you receive? Do you even bother to answer calls from unknown numbers? How's about your text experience, do you receive a lot of unsolicited text spam? They telecommunication companies are flawed to the core and are not addressing these core issues at all. Their product is a crime engine masquerading as a communications device.
• throwzaway20102 5y ago
  My question to Apple … why now? If this heinous problem has existed, why didn’t it get priority over Animoji? Did China tell Apple, hey we want to spy on our citizens and if you don’t help us you need to get the fuck out of China? So they create this “think of the children” solution for the USA but can easily swap it out for another thing a government considers a nuisance?
• disruptalot 5y ago
  Remember when Apple punted privacy as their strength? In a closed proprietary system, it's always just a promise. It's the same story then as it is now. Never trust a company promising you privacy. If you can't verify your privacy, consider it absent.
• throwaway6352 5y ago
  In this day and age I am basically guilty of heresy for speaking out against the insanity of our times. Nobody should be prosecuted for possessing or viewing any text, image or audio recording, unless they created it themselves by abusing someone. That is a fundamental principle of a free society which has been acknowledged for if not decades, hundreds of years. By all means prosecute the distribution of such material - but to prosecute possession is returning us to the Middle Ages hunting witches again. It is only because of radical feminists and other moralists creating moral panics in the 1970s and 1980s are we faced with the present situation in the 21st Century, which is extremely dangerous in a highly interconnected society such as ours, where it is very difficult if not impossible to stop people from unintentionally coming across such material. We might as well have a nuclear reactor in our own homes, yes it's very useful and provides lots of free electricity but one day it can melt down destroying the entire family? Do you see the analogy I'm making? Because the Internet is just as dangerous. The penalties for possession of such material and being put on the sex offenders register are so horrific, it's totally unreal. It's like a real life nightmare. I cannot believe I'm typing this here in 2021. What has happened to our country? We criticize dangerous products that are unsafe and burn your house down or electrocute you, but why can't we criticize the dangers of the Internet and all the ridiculous draconian laws involving it? It is as if the law itself and the crazy ways it's made is beyond discussion? Nobody should have to fear their own computers (unless they are doing major hacking/fraud/sending death threats, etc...).. This simply cannot be happening in a free society. It looks like we are no better than China - it's just over different stuff here in the West.
• srmarm 5y ago
  Out of interest what happens if an image gets flagged client side? If it's a CSAM scanner online that can presumably have a human do a sanity check before informing the authorities. But will Apple have the ability to pull the file from your phone or does get reported directly. Both quite concerning options TBH.
• tyingq 5y ago
  Regarding the OFAC story on the front, Paypal's implementation of this is stunningly brain-dead. They do naive string matching for any field in the transaction, including some rather short OFAC listed aliases. So you can have a situation where a very short string, like "JONAS" in a non-payee-name field, like a "note" or "memo", causes the transaction to be held. Then they won't release the funds until you produce scanned identifications, detailed write-ups, etc.
  • wiredfool 5y ago
    There are a lot of common names in OFAC when you get to the alias list. (because, apparently, a common name works well as an alias) Jesus. Lopez. Mohammed. Mark Taylor. Sally Jones. Park. Wong. Kim. Chuck Taylor used to be on there, but isn't anymore. (not the shoes) Since there are a lot of alias matches, you get a lot of people who wind up matching when there's no reasonable way that there's an actual match. (i.e., they're happily living in the US and have an ordinary bank account there, not the dead dictator of Liberia, or someone in Columbia, or North Korea.)
    • tyingq 5y ago
      Right. The interesting part is that there doesn't seem to be good guidance from OFAC on exactly how tight/loose the matching needs to be, and on what parts/fields of the transaction need to be scanned or not. So, everyone implements it differently.
• dalbasal 5y ago
  So... the obvious Serious Issues here are privacy and child pronography. This blog though, is about an issue I don't know how to define concisely. It's about getting labeled, for whatever reason as "high risk." I flagged some paypal thing years ago, when I moved countries. It was too much trouble to sort out, so I abandoned my $20 and moved on. Paypal has me in a no recourse bucket. Most transactions run through such risk assessments. False positives are tolerated. Child pronography scanning, as this post lays out, will have similar issues. We're already used to credit scores and insurance behaving like this. I'm not diminishing the privacy aspects, but this neokafka world of automated, risk accessing bureaucracy is also a world we're constantly walking towards. It's also bad.
  • tornato7 5y ago
    Basically, these companies know that false positives happen. To them, it's just a statistic - they can afford to wrongly ban 0.1% of customers in the name of preventing abuse. But for the person they're wrongly banning, that's not so fun. I'm worried this will apply to more seriously some day. "This algorithm says you're guilty of a crime, and it's 99.8% accurate, which we've determined is acceptable, so there will be no way to appeal your verdict."
• helgee 5y ago
  A second degree acquaintance of mine in Germany had his house raided by police and all of his electronics (including his work laptop and phone) temporarily confiscated on suspicion of possession of child pornography because... ...an anonymous member of a child pornography forum used the handle "$NAME_$YEAR" and had his location set to $TOWN. The innocent bystander's middle name is $NAME and he was born in $TOWN in $YEAR but had not lived there for decades. He only learned about this absolutely nonsensical justification and got his stuff back by hiring a lawyer who requested access to the DA's investigation file. If you can get a judge to sign a warrant with flimsy non-evidence like this, what will they do when the robot says "95% match detected". This gives me pause...
• err4nt 5y ago
  Considering how frequently my innocent images (petting my cat, taking a photo of a pepper growing in my garden, pointing to a word printed on a page in a book) trip Discord's automatic-bad-image-blocker and are unable to be uploaded to Discord because they think they are bad, I'm a little worried that when they turn this technology on, instantly many 100% innocent photos people have taken will be quietly flagged and put people on lists without them ever knowing it, and as they go through life this non-thing that they were never charged with (to be able to be found innocent) will follow them and haunt them and impact other business relationships, etc. All because of pictures a computer thought was something else.
  • nmeofthestate 5y ago
    Sounds like you're describing a completely different system than the one being discussed. It compares to a corpus of known-bad images.
    • cwkoss 5y ago
      The contents of this corpus are unverifiable. They are claimed to be only known-bad images. However, the public doesn't (and shouldn't) have the ability to inspect these images to verify that non-CSAM politically threatening images are not included. (ex. pictures of politicians doing cocaine with adult prostitutes, or images of war crimes committed by US troops) I think that distinction is important. Because even if the NCMEC database is only child porn right now (which we cannot know), the public will also not know if the scope of the database later changes because now Apple will call the police on whoever possesses an image that matches this database. Adding this capability increases the incentive for misuse of the database.
• flareback 5y ago
  I've recently bought a pinephone, I'll be glad to get out of the apple eco-systemm
• dblock 5y ago
  Time to (re)watch Brazil!
• gvhst 5y ago
  As a survivor of child abuse, one of the major obstacles to recovery is to identify and let go of this generalized fear that goes with you everywhere you go. I find it extremely ironic that, under the pretense of "saving the children," one would implement technologies that subject the general population to that same type of overbearing fear.
• Bellamy 5y ago
  As a stock owner, owning thousands of dollars of Apple equipment, and respect for the company I think this is a huge mistake. This technology benefits practically nobody, consumes resources, and is bad for the reputation of the "privacy" aware company. If something like this is to be implemented the door is opened for scanning everything else. Here is a question: If I know my photos will be scanned for child pornography, would I keep child pornography in my phone?
• gvhst 5y ago
  Given the massive progress seen in image generation and synthesis with NNs, I'd imagine it will only be a matter of time until the production of CP doesn't require a child. Get ready, because the situation is about to get a whole lot more complicated.
• commandlinefan 5y ago
  > Eventually, the forensics will return nothing. Your lawyer will succeed in getting you to walk free. If you're lucky. If you're unlucky, they'll insist that you have it stored "encrypted" in a file whose extension their automated scanners don't recognize and keep you locked up until you produce the password. And insisting that there is no password and that the file isn't an encrypted file (it's some system file and you don't even know what it's for) is exactly what a criminal would say.
• CraftingLinks 5y ago
  Don't reply with your wallet is a small start.
• dariosalvi78 5y ago
  Look at the evolution: from mainframes (expensive machines owned and controlled by others) to Personal Computers (cool! computers for the masses! we can do whatever we want with them!), to the smartphone (we can now bring the PC with us!), to a device you can't control, owned by dystopian corporations that now continuously looks into your data to see if you break the law.
• fastaguy88 5y ago
  This person has a personal story about how bank bureaucracy cannot be bothered to get him off of some watch list. Everything else in this story has not happened, and requires a substantial extrapolation of what could conceivably happen. But all the things that could conceivably happen could always conceivably happen. My sense is that Apple is trying to placate some groups to avoid providing more intrusive changes, such as being able to decode iMessages and removing end-to-end encryption. While there is a lot of concern about a slippery slope, one could imagine this is actually a path that avoids much steeper, harder to avoid slopes. Apple’s announcement of a limited program of photo matching does not increase or decrease their ability to scan files in the future, but it’s not what they are describing now, and it’s unclear why they would provide such capability since their brand has emphasized privacy so strongly.
• literallyaduck 5y ago
  Don't like this story? Get involved with local and national government to remove qualified immunity for false accusers, and put stiff penalties on companies that erroneously implicate wrong doing and put bonus penalties for the deaths of pets by SWAT.
• deleted 5y ago
  [deleted]
• perryizgr8 5y ago
  I don't understand the hullabaloo about this. Apple is a private company. They are selling their product. You are free to not buy it. That's what people have been saying anytime the topic of anti trust is brought up in the context of app store. So same deal here. Lol.
• YeBanKo 5y ago
  Ironically, this happened with Apple recently: https://news.ycombinator.com/item?id=27878333 https://news.ycombinator.com/item?id=27878333 misapplication of face recognition.