12 ms·

Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. The reality is more nuanced: People understand exact

Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. The reality is more nuanced: People understand exactly how it works, and how it works is that it is turn-key onboard spyware, that Apple pinky-swears isn't being used wrong today. For example if the scope/mission expands (e.g. foreign governments), suddenly you've created a drag-net for whatever "badness" is of interest in whatever today's moral panic is (e.g. terrorism after 9/11). Plus perceptual hashing, by its very design, is created to be less precise than traditional cryptographic hashing. A cryptographic hash + file size combo is unlikely to have a false positive within our lifetime (and it has been used successful by multiple companies to combat CP). The interesting thing about a perceptual hash is that the closer the source material is to the banned material in terms of actual content (e.g. nudity), the more likely for a false positive. Therefore, if Apple does mess up via false-positive and manually review your material, it is more likely to be sensitive private materials (involving consenting adult(s), not CP) because that is what the perceptual hashes are looking for similarities to. PS - If you think this concept cannot happen in a Western country, see the UK's internet filters as a textbook example. Originally started to fight CP, and now used to fight a ton of other stuff too with more proposals every year: https://en.wikipedia.org/wiki/Web_blocking_in_the_United_Kingdom#Planned_programmes https://en.wikipedia.org/wiki/Web_blocking_in_the_United_Kin...

sneak 5y ago
> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. The reality is more nuanced: People understand exactly how it works, and how it works is that it is turn-key onboard spyware, that Apple pinky-swears isn't being used wrong today. It's not a mistake. It's gaslighting.
ASalazarMX 5y ago
Sometimes the slippery slope is not a fallacy, but the natural consequence of seeking more control. This feature can be horribly misused in the future, for example, to find people who have sent/received/downloaded Tiannanmen pictures, because instant messaging pictures can be automatically added to your albums. You can't trust a feature whose only impediment from being abused in the future is such pinky promise.
- eurasiantiger 5y ago
  Add hashes of police uniforms and now you cannot take pictures of law enforcement. Add hashes of politicians. Businessmen in suits. Army uniforms. At the end you have a weapon that can only shoot civilians.
  - midev 5y ago
    Do you know how discovery works in court? I'm guessing not. Once a hash matches, law enforcement would need a subpoena to access the raw image. If a person were to be arrested, that evidence would be turned over to the defense. It would be very obvious that a picture of a police officer was not child pornography. Are you under the impression people are jailed for hash collisions? Because that's not the case at all....
    - eurasiantiger 5y ago
      Have you heard of the chilling effect?
      - midev 5y ago
        Yeah, which is irrelevant to the discussion. This thread is about photos of law enforcement and politicians used to convict people on CP charges. Let's stick to the topic of discovery and how courts work.
        eurasiantiger 5y ago
        Let’s conveniently ignore all the facts that speak against the planned dystopia, and focus on the positives of having all facets of your life constrained and controlled by authorities.
        midev 5y ago
        I'm happy to have that discussion, I think it's critical and we need to update laws to protect privacy. You still don't get to be technically incorrect, though. You still need to know how courts work, if you want to participate in that discussion.
mickotron 5y ago
They make the premise so innocuous that it would seem unreasonable to object, eg who would object to keeping kids safe? However it's what comes after this, the slippery slope of what to enforce and not. The genie doesn't go back in the bottle.
- sharken 5y ago
  Exactly. It's the same argument when Corona contact-tracing was discussed and later implemented. First using paper forms and later using apps. Why would you object to limiting the spread of the Coronavirus ? Fast forward to the present where law enforcement in Germany, Australia and Singapore have used contact-tracing for other purposes. So no to this plan from Apple, they must discard all plans of client-side scanning and instead increase their privacy stance.
  - mung 5y ago
    Could you please expand on the other uses of contact tracing in those countries?
    - sharken 5y ago
      This is a compilation of what others have posted: - Germany https://www.golem.de/news/hamburg-polizei-nutzt-corona-kontaktlisten-nach-straftat-2007-149482.html https://www.golem.de/news/hamburg-polizei-nutzt-corona-konta... https://www.ccc.de/de/updates/2021/luca-app-ccc-fordert-bundesnotbremse https://www.ccc.de/de/updates/2021/luca-app-ccc-fordert-bund... - Australia Australia, WA, check-in data: https://www.abc.net.au/news/2021-06-15/safewa-app-sparks-urgent-law-change-after-police-access-data/100201340 https://www.abc.net.au/news/2021-06-15/safewa-app-sparks-urg... Australia, WA, border pass data: https://www.abc.net.au/news/2021-06-17/g2g-app-data-accessed-for-investigations-as-well-wa-police-admit/100222804 https://www.abc.net.au/news/2021-06-17/g2g-app-data-accessed... Australia, VIC, check-in data: https://www.theage.com.au/politics/victoria/police-sought-access-to-qr-check-in-data-intended-for-contact-tracing-20210621-p582x4.html https://www.theage.com.au/politics/victoria/police-sought-ac... Australia, QLD, check-in data: https://www.abc.net.au/news/2021-06-29/queensland-coronavirus-check-in-app-police-data-search/100249624 https://www.abc.net.au/news/2021-06-29/queensland-coronaviru... - Singapore https://fortune.com/2021/02/01/singapore-covid-data-tracetogether-use-law-criminal/ https://fortune.com/2021/02/01/singapore-covid-data-tracetog... https://www.straitstimes.com/singapore/politics/police-can-access-tracetogether-data-only-through-person-involved-in-criminal https://www.straitstimes.com/singapore/politics/police-can-a... https://www.straitstimes.com/singapore/proposed-restrictions-to-safeguard-personal-contact-tracing-data-will-override-all-other https://www.straitstimes.com/singapore/proposed-restrictions... https://www.straitstimes.com/singapore/politics/bill-limiting-use-of-tracetogether-for-serious-crimes-passed-with-govt-assurances https://www.straitstimes.com/singapore/politics/bill-limitin...
userbinator 5y ago
Also, there's a "we can't show you the exact CP image that one of yours matched, because that would also be illegal" catch-22 that basically gives them the power to shroud this whole system in legally-obligated secrecy. Secrecy which will no doubt be abused to hide much more.
- bryanrasmussen 5y ago
  I do worry this is basically automating and scaling up the whole "idiot working at the mall photo development booth turns family in for child porn because they took pictures of 3 year old in bath" story that shows up every now and then.
  - mlindner 5y ago
    It's exactly that though. Except instead of the "idiot" it's "completely dumb neural network".
  - bryanrasmussen 5y ago
    reading through Daring Fireball's technical discussion of it https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope https://daringfireball.net/2021/08/apple_child_safety_initia... I suppose does much to allay my worries though, although my natural distrust of authority and the powerful still makes me not like anything like this - no matter the spin.
  - deleted 5y ago
    [deleted]
  - Ancapistani 5y ago
    I spent about a year working a retail photo processing lab - I can assure you that what you describe is exceedingly rare. I saw “questionable” images on a probably weekly basis. Maybe once a month something would come through that I thought warranted bringing my supervisor over to provide a second set of eyes because I didn’t think it warranted calling the police over but wasn’t entirely sure. An example of that that comes to mind was a series of photos that appeared to show a young woman bound and gagged to a pipe in a basement area - but she was obviously of age, her eyes in the photos didn’t look fearful, and the rest of the roll showed her free and apparently happy. Finally, I was the person who accepted the roll of film and knew that she was the one who dropped it off. We didn’t call anyone on that, but I figured it wouldn’t hurt to get a second opinion. When she came to pick them up, I did ask her to review them and make sure everything was OK with them. Twice in about a year we have photos come through that were obviously what’s now called “CSAM”. In both cases I called the police without consulting management then told them afterward. Also in both cases, the negatives were put into the store’s safe and the owners were arrested on-site when they came to pick them up. All of this is to say - photo lab techs see some shit. If they called the police every time there was a picture of a kid in the bath, police brutality would be at an all time low because they’d not have time to respond to anything else. :)
midev 5y ago
> people misunderstand how it works You can literally read any thread on HN or Reddit to see this is the case. Thousands of comments about facial recognition and AI, completely misunderstanding how it works. The EFF article itself was FUD. The EFF article starts right off claiming a backdoor, which is completely incorrect. Taking hashes of client-side content is not a backdoor, and the EFF should be embarrassed for not understanding that.
- rdedev 5y ago
  This client side scanning of local files can be easily extended to scan the whole device. That's why the eff was calling it a backdoor. All it takes is for the right govt to put the right preassure on apple. In a way it's kinda worse than a backdoor since your phone is actively reporting any suspicious looking files to the authority all the time on its own. The govt dosent need to actively look for it. Just add the appropriate has of what it wants discovered to the db and wait
  - midev 5y ago
    > This client side scanning of local files can be easily extended to scan the whole device Correct, Apple can access unencrypted files stored on your device. > That's why the eff was calling it a backdoor Accessing unencrypted files on a device is not a backdoor > actively reporting any suspicious looking files to the authority all the time on its own Yep, that's a problem. Not a backdoor, though.
    - rblatz 5y ago
      What is your deal. You are out here arguing in bad faith all over this thread. What is your motivation?
      - midev 5y ago
        What's the bad faith? What's single thing have I said that is incorrect? It's not breaking E2E encryption. No one is locking people up for political photos. It's not using facial recongition. I don't like FUD driving discussions. I think we need to have serious discussions about this technology. I think people that don't understand it shouldn't participate....
tasogare 5y ago
> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. They don't believe this. The lack of understanding trick is often used by governments and corporations because it robs the opposite party the choice to oppose, as opposition is framed as lack of understanding instead of a political choice. It's a semi-elaborate way of calling idiots people who disagree. This is why governments "educate" on topics they don't people to oppose.
Lammy 5y ago
> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. I've read too many contentious-change memos to believe it's a mistake. These things are authored by committee with Legal always present, and everything is written expecting it to leak. It's just a passive way to manufacture consent by convincing at least some segment of people to doubt their own thought processes. I'd call it "gaslighting" if that word wasn't already massively misused. See also: iPhone 4 owners who were "holding it wrong".
smnrchrds 5y ago
And HN's mistake is thinking any of this will matter. Apple only cares about money, always has, always will. And I'm willing to bet they are going to continue beating their revenue and profit record quarter after quarter. When their actions do not affect the one thing they care about, why should they care what anyone says?
- jacquesm 5y ago
  Ah ok, so as long as it is profitable ethics don't matter. Got it.
  - kyriakos 5y ago
    Companies don't have to be ethical, they just have to be legal.
    - jacquesm 5y ago
      This is the sort of reasoning that got us into all this crap in the first place, and it certainly won't get us out of it.
      - chronic2703 5y ago
        > This is the sort of reasoning that got us into all this crap in the first place I don’t consider us in “crap”. I welcome Apple’s decision.
      - kyriakos 5y ago
        Not saying its good but these are the facts. Companies try to maximise profit within the boundaries of the law.
    - quonn 5y ago
      So just like everyone else? We hold people to certain standards and can do the same for companies.
pinum 5y ago
About that "manual review"... Refer to pages 10 and 11 of the technical paper. >The server then uses the decryption key to decrypt the inner encryption layer and extract the NeuralHash and visual derivatives for the CSAM matches. This "visual derivative" term shows up repeatedly. To me, the implication seems to be that Apple doesn't look at the actual suspected image before deciding whether to proceed with a report. Instead, I infer that they only verify whether (as the device reports) the image's neuralhash is indeed present in the NCMEC database. If my understanding is correct, their "manual review" process actually provides no protection at all against collisions or erroneous database entries. Further supporting this, on page 4: >Apple reviews each report to confirm there is a match It only refers to a match, not about whether the image appears to be illegal. This makes perfect sense from Apple's perspective- who would want to be in the business of reviewing reports of probably-illegal images?- but it means that the references to a manual review safeguard would seem to be false reassurance. Maybe I'm misunderstanding the paper.
- smsm42 5y ago
  I don't see any reasonable way here. Either somebody looks at the images - i.e. they have some underpaid poorly trained grunt to look at horrible abuse images which is probably illegal to look at whole day, having about 3 seconds to make a decision on each - or what they're doing is just a sham, "Did the computer say 'it matches'? Yes it did! Review complete, match confirmed!" I don't see any non-horrible way of doing non-sham reviews here.
  - peakaboo 5y ago
    People keep trading away privacy for convenience. We won't have any privacy left if we continue to go down that road. People need to start looking at these tech giants with extreme scepticism. Like we do if companies are from China. We need to think about our American companies the exact same way, and stop trusting them to care about our privacy.
    - ptero 5y ago
      > We won't have any privacy left if we continue to go down that road. If people look at today's state (phone movement tracking, voice capture, cameras and face ID, etc.) from the mental model of 30 years ago, they would see neither privacy nor freedom. At all. Sadly, it is not the case of preserving what we still have, if we want freedom and privacy we will have to reacquire them. And a lot of the price will have to be paid in blood.
deleted 5y ago
[deleted]
mctub 5y ago
This still means iOS 15 is adversarial and *Apple lacks credible neutrality*.
xxs 5y ago
>"A cryptographic hash + file size combo" SHA already uses the length in the output hash. Having the explicit length is quite superfluous
- verygoodname 5y ago
  > Having the explicit length is quite superfluous Not entirely. It makes it so that, to achieve a "full" collision, you have to ensure that the sets of data collide both in SHA hash and in length, helping to prevent attacks that rely on appending/prepending/removing data (for example, "length extension attacks" involve manipulation of the hash by appending data). TL;DR: It is harder to find a collision SHA(B) for SHA(A) if you add the additional constraint that the length of B must match the length of A.
  - tialaramex 5y ago
    This is completely wrong. The known collision attacks for the MD-family and SHA-1 all in fact produce collisions with the exact same length. The method used necessarily does this.
    - verygoodname 5y ago
      > This is completely wrong. Which part? The fact that storing "length" along with a hash is not superfluous? You can probably find many things which have a SHA hash of "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb" (infinite things, if we assume arbitrary-sized inputs), but you can only find ONE thing which has that hash and has length 1. I just made it impossible (not just unlikely) for you to find a collision. > The known collision attacks for the MD-family and SHA-1 all in fact produce collisions with the exact same length. Emphasis mine. And note that I did not claim otherwise in my comment.
      - tialaramex 5y ago
        > Which part? The fact that storing "length" along with a hash is not superfluous? The part where you make a false claim out of ignorance. > You can probably find many things which have a SHA hash of "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb" No reason I should go looking for such things. You're the one making the false claims, if you have found "many things" with that hash then list them to prove your point, otherwise go away.
ren_engineer 5y ago
A good way to think about any potential law like this is to imagine your worst enemy or most hated politician was given power to abuse said law, and then think about all the ways they could use it to make your life miserable always think about the worst case scenario when it comes to privacy, because that will be the end result
- telxoss 5y ago
  To me we don't even consider the actual worse case scenario. Is there any doubt 50 years from now all this will be on algorithmic autopilot for digital authoritarianism? All societies will have Chinese style monitoring but to an unimaginable degree even to people in China right now. People outside IT were barely even using the internet 25 years ago. We are at the very start of this and already too far gone. You just have to enjoy these times and what we still have.
jimbob45 5y ago
I disagree. Google announced the same thing for Google Drive and didn’t receive this level of pushback. If you’re Apple, the only way that’s possible is if you explained it badly (i.e. differently than Google did). I’m guessing their real prerogative is to keep unlawful content from ever reaching their servers. You wouldn’t know that from reading their press releases though.
- chaostheory 5y ago
  The difference is that Google doesn’t advertise itself as a bastion of privacy. Google spying on you is expected. More consumers also pay for iCloud vs Google Drive. Expectations change when you’re paying for a service.
  - okamiueru 5y ago
    Isn't the difference that with google drive, they scan the files you upload to their servers? And, this latest apple debacle is on scanning files locally on the device itself?
    - ziml77 5y ago
      If Apple is to be believed (and doesn't change anything later) then they're doing the same thing. They're scanning the things that are being uploaded to iCloud. They're just doing it locally instead of doing it after they're on the iCloud servers.
- john_minsk 5y ago
  Why wouldn't they show a pop-up on the phone highlighting which images may be problematic and that report is being sent to the government? I might change my mind on uploading these things to my iCloud. Can't wait for all examples of false positives that will surface. Why wouldn't they delete them directly, but rather send report to the government? Why wouldn't they go even further and if image detected on the webpage that is problematic => It doesn't allow it to get to my filesystem either, so that I don't get into trouble if for some reason police decides to search my laptop. All relevant questions. Answer is the only one: so that when police knocks on your door you were blissfully unaware. Quite honestly this is all too sad. Again this argument about CP that everyone will buy, but which doesn't work in the wild. There are some bad people out there, but if these people are into buying or creating CP => they will rarely upload it into iCloud. Especially now, when everyone knows Apple is looking for it. So they will go even deeper into hiding, while everyone else will have to live with devices that are spying on them...
stjohnswarts 5y ago
I see whataboutism (google drive scans! ms scans!) and "here are the technical details of why we can't just look..." . The bottom line is that none of that matters. the 99.999% problem is Apple is spying on you on your. own. device. I don't know how many techies I've had to shoosh and point that out. It doesnt matter what kind of hash or algo or "layers of protection" are being used. What they are doing is inherently wrong and totally the antithesis of their privacy mantra. It will also inevitably be used by governments for ant-humanitarian purposes to imprison people for thought crimes against the state. Unless they (Apple) backpedal this abomination soon I'm definitely minimizing my apple purchases in the near future and moving off the platform entirely long term. Even google isn't so bold as to put their spyware on their devices to police you for the government. Suddenly ads don't seem that bad ...
- smsm42 5y ago
  The mistake here is thinking iphone is your own device. If they can view anything stored there, remove anything from there, control anything you install and run, disable it at any moment they wish - do you actually own it, or you just granted the use of it as long as Apple approves of it? Apple certainly doesn't ask your permission to do anything on "your own device" - but you have to ask theirs. Is that how ownership works?
  - phs318u 5y ago
    Having ditched Apple laptops this year (after 18 years of Mac computers) for Linux (on System76 hw), I've been wondering what it would take to pry the iPhone out of my hands. I think I've found it.
BiteCode_dev 5y ago
No, Apple fully understands the situation, and in fact, you don't see many Apple fanboys in the comments defending the firm for once because even they know it's wrong. Apple just communicates this message because that's what they need to say for their business, that's all. And they can do so because they also know this uproar will disapear like tears in the rain in a short time if they keep smiling. Their bottom line will not be affected because people don't care. Remember that now, Bill Gates, who we used to associate ti Satan in the 90s, is now viewed as a hero. Remember that about 1 american out of 160 is working at Amazon, a company with the reputation of treating their employees terribly. Most people hence knows somebody impacted by this, but don't stop shopping at amazon. Remember you could say you grab them by the pussy and be elected president. Remember people gives in mass their data to someone who said "they trust me, the dumb fucks". Remember that when somebody risked his life to unveil a massive conspiracy to spy on the entire world, the person has been hunted as a traitor. Remember that one president can lie about WMD, go to war against the vote of the UN, kill thousands of civiliands, spend 600 billions of dollars there while poverty is rising at home, and suffer no consequence whatsoever. Most people have forgotten already. Hell, my girlfriend don't even remember the name of the guy. But it's you that have to be monitored all the time for CP. And you will be punished harshly if you behave badly, not just according to law, but also according to social contracts, both which powerful people can escape now with money and PR. That's the world we are living right now.
smsm42 5y ago
It's not a mistake, it's a rhetorical trick - to present somebody opposing your actions as a misinformed idiot that just doesn't know what they're talking about, and once proper technobabble is deployed every reasonable person would realize their mistake. And if they don't, they must be much bigger idiot than we thought, there's no other reasonable conclusion.
tinus_hn 5y ago
Quote: > and more than a few are worried about the implications
2Gkashmiri 5y ago
yes. i live in a small 8 million strong "valley" where the government is hellbent on destroying people. https://kashmirobserver.net/2021/07/10/11-govt-employees-including-2-sons-of-hizb-chief-salahuddin-sacked/ https://kashmirobserver.net/2021/07/10/11-govt-employees-inc... they sacked these government employees because they allegedly were engaged in "anti-national activities", no courts, no trial, no hearing. just sentencing that since you are allgedly found to be engaged in such activities, your employment is terminated and there is no recourse. https://www.organiser.org/Encyc/2021/8/3/Govt-s-policy-of-No-passport-for-stone-pelters-unnerve-Kashmir-Netas.html https://www.organiser.org/Encyc/2021/8/3/Govt-s-policy-of-No... here they have decided to not give passports to protesters or give then government jobs because "anti national activities" again the same trope. this might be a small issue but here is a jounalist https://thewire.in/rights/kashmiri-journalist-masrat-zahra-police-assault-father https://thewire.in/rights/kashmiri-journalist-masrat-zahra-p... whose father was beaten to a pulp because she being a decorated journalist had criticized the government, they slapped terrorism charges on her and german government gave her asylum so they did the next best thing, hurt her parents because why not. now if she didnt have her passport because she was already a known suspect of "anti national activities", asylum might be a little difficult so they want that. i know 100% indian government and for that matter pakistan government as well will use this technology for "national interests" in finding out protesters and dissidents and they either do not even go for trials or if they do, the same are sham ones so doesn't matter if they make a show of a trial because the guilty are condemned already.
speleding 5y ago
> A cryptographic hash + file size combo is unlikely to have a false positive within our lifetime The chance of a "technical" false positive is tiny, but they need to look all false positives: If some joker sends you a lewd picture through WhatsApp, WhatsApp by default saves every picture to your photo library, then you are now on the naughty list. Good luck trying to explain yourself out of that one.
- ziml77 5y ago
  WhatsApp saves all received images to your library? That seems hard to believe. I can't imagine that anyone would appreciate having their personal photo collection cluttered with other peoples photos and all of the meme images that people share.
  - speleding 5y ago
    It's a setting, called "Save to camera roll". Default is on.
    - ziml77 5y ago
      Strange. Is that a desirable feature for WhatsApp users? I must be overlooking a use case that makes that a sensible default.
- tyingq 5y ago
  The context there was that a typical CSAM hash like PhotoDNA is NOT a cryptographic hash, but a perceptual one, which DOES have notable false positive issues.
arthur_sav 5y ago
Seriously, the backdoor is in place. Now all they have to do is keep adding hashes to the list and voila. Trump supporters = terrorists = you're in jail because of political games.
lovelyviking 5y ago
>these features are part of an “important mission” for keeping children safe. The other mistake is to think that removing privacy with longing effects on society [1] and thus attacking the very fabric of a free democratic society respecting human rights would not make children any safer. When they grow up they possibly would have to live in concentration camps or Chine-like regime. It is very safe, but not sure it is very pleasant. It's about values. It should be absolutely illegal to install any spyware on personal device without warrant [1] https://news.ycombinator.com/item?id=28084578 https://news.ycombinator.com/item?id=28084578 [2] https://news.ycombinator.com/item?id=28096948 https://news.ycombinator.com/item?id=28096948
Perstone 5y ago
They don't believe that people misunderstand how it works. This is marketing and PR speak for "We are going to do this and we are going to push enough tech jargon out there that most people won't understand nor care about the difference".
mberning 5y ago
You nailed it. Apple is so flexible with their morals when money is on the line you know this will be abused by some authoritarian government. They will ask apple to scan for some content, apple will say no, the government will ban apple from selling their products, and suddenly apple will change their mind. Next thing you know a huge swath of journalists or dissidents is being disappeared to reeducation camps. Apple is working overtime to wrestle the “most evil company in the world” title from google and amazon.
queuebert 5y ago
Isn't perceptual hashing necessary for this application to prevent simple modifications (e.g. resizing, filters) from changing the signature?
ben_w 5y ago
Yeah, this is my reading of the situation too. From a technical point of view, what Apple is doing seems like a reasonable approach to me; what I absolutely do not trust is any claim that it will never ever be expanded to include, say, governments requiring similar tech be applied to all messages before they are encrypted in order to look for dissidents… and then generally at opposition parties. (And of course there are enough competent technologists available for making custom encrypted chat apps that the hostile governments could only achieve their goals by putting the AI in the frame buffer, which opens the door to subliminal attacks against political enemies — targeting the AI with single frames such that the device user doesn’t even notice anything). But I know I have paradoxical values here: On the one hand, I regard abusive porn as evil. I want it stopped. On the other, I have witnessed formerly-legal acts become illegal, and grew up in a place where there are sexual acts you can legally perform yet not legally possess images of. I think the governments (and people in general) follow a sense of disgust-minimisation, even when this is at the expense of harm-minimisation. And any AI which can genuinely detect a category of image can also be used to generate novel examples that are also in that category without directly harming a real human. I don’t have a strong grasp on what I expect the future to look like. My idea of what “the singularity” is, is that rather than a single point in the future where all tech gets invented at once, it’s an event horizon beyond which we can no longer make reasonably accurate predictions. I think this horizon is currently 2025-2035, and that tech is changing the landscape of what morality looks like so fast that I can’t see past that.
squarefoot 5y ago
It's just a matter of time until someone can create a "ThisChildPornDoesNotExist" site, then send links to targets to have their lives ruined. I am extremely worried by humans policing the Intertubes, and even more worried by algorithms doing so because they make the perfect excuse to do sloppy checking. See Youtube DMCA related takedowns for multiple examples.
- dragonwriter 5y ago
  > It's just a matter of time until someone can create a "ThisChildPornDoesNotExist" site, then send links to targets to have their lives ruined. “Their” in this case referring to the sender, which makes the particular problem somewhat self-limiting, right?
  - squarefoot 5y ago
    Probably so, but a more smart malware that grabs fake images from an encrypted server somewhere, then plants them in the victims PC/phone/whatever, then deletes itself, isn't rocket science.
lowbloodsugar 5y ago
Apple: our staff can look at your tits if you take a sexy picture and look young enough or some other magic button. Hm. Seems like a good attack would be take a celebrity, photoshop some nakedness, and put that in the database. Output: real photo of said celebrity. This is not acceptable.

sneak 5y ago

> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. The reality is more nuanced: People understand exactly how it works, and how it works is that it is turn-key onboard spyware, that Apple pinky-swears isn't being used wrong today. It's not a mistake. It's gaslighting.

ASalazarMX 5y ago

Sometimes the slippery slope is not a fallacy, but the natural consequence of seeking more control. This feature can be horribly misused in the future, for example, to find people who have sent/received/downloaded Tiannanmen pictures, because instant messaging pictures can be automatically added to your albums. You can't trust a feature whose only impediment from being abused in the future is such pinky promise.

eurasiantiger 5y ago
Add hashes of police uniforms and now you cannot take pictures of law enforcement. Add hashes of politicians. Businessmen in suits. Army uniforms. At the end you have a weapon that can only shoot civilians.
- midev 5y ago
  Do you know how discovery works in court? I'm guessing not. Once a hash matches, law enforcement would need a subpoena to access the raw image. If a person were to be arrested, that evidence would be turned over to the defense. It would be very obvious that a picture of a police officer was not child pornography. Are you under the impression people are jailed for hash collisions? Because that's not the case at all....
  - eurasiantiger 5y ago
    Have you heard of the chilling effect?
    - midev 5y ago
      Yeah, which is irrelevant to the discussion. This thread is about photos of law enforcement and politicians used to convict people on CP charges. Let's stick to the topic of discovery and how courts work.
      - eurasiantiger 5y ago
        Let’s conveniently ignore all the facts that speak against the planned dystopia, and focus on the positives of having all facets of your life constrained and controlled by authorities.
        midev 5y ago
        I'm happy to have that discussion, I think it's critical and we need to update laws to protect privacy. You still don't get to be technically incorrect, though. You still need to know how courts work, if you want to participate in that discussion.

mickotron 5y ago

They make the premise so innocuous that it would seem unreasonable to object, eg who would object to keeping kids safe? However it's what comes after this, the slippery slope of what to enforce and not. The genie doesn't go back in the bottle.

sharken 5y ago
Exactly. It's the same argument when Corona contact-tracing was discussed and later implemented. First using paper forms and later using apps. Why would you object to limiting the spread of the Coronavirus ? Fast forward to the present where law enforcement in Germany, Australia and Singapore have used contact-tracing for other purposes. So no to this plan from Apple, they must discard all plans of client-side scanning and instead increase their privacy stance.
- mung 5y ago
  Could you please expand on the other uses of contact tracing in those countries?
  - sharken 5y ago
    This is a compilation of what others have posted: - Germany https://www.golem.de/news/hamburg-polizei-nutzt-corona-kontaktlisten-nach-straftat-2007-149482.html https://www.golem.de/news/hamburg-polizei-nutzt-corona-konta... https://www.ccc.de/de/updates/2021/luca-app-ccc-fordert-bundesnotbremse https://www.ccc.de/de/updates/2021/luca-app-ccc-fordert-bund... - Australia Australia, WA, check-in data: https://www.abc.net.au/news/2021-06-15/safewa-app-sparks-urgent-law-change-after-police-access-data/100201340 https://www.abc.net.au/news/2021-06-15/safewa-app-sparks-urg... Australia, WA, border pass data: https://www.abc.net.au/news/2021-06-17/g2g-app-data-accessed-for-investigations-as-well-wa-police-admit/100222804 https://www.abc.net.au/news/2021-06-17/g2g-app-data-accessed... Australia, VIC, check-in data: https://www.theage.com.au/politics/victoria/police-sought-access-to-qr-check-in-data-intended-for-contact-tracing-20210621-p582x4.html https://www.theage.com.au/politics/victoria/police-sought-ac... Australia, QLD, check-in data: https://www.abc.net.au/news/2021-06-29/queensland-coronavirus-check-in-app-police-data-search/100249624 https://www.abc.net.au/news/2021-06-29/queensland-coronaviru... - Singapore https://fortune.com/2021/02/01/singapore-covid-data-tracetogether-use-law-criminal/ https://fortune.com/2021/02/01/singapore-covid-data-tracetog... https://www.straitstimes.com/singapore/politics/police-can-access-tracetogether-data-only-through-person-involved-in-criminal https://www.straitstimes.com/singapore/politics/police-can-a... https://www.straitstimes.com/singapore/proposed-restrictions-to-safeguard-personal-contact-tracing-data-will-override-all-other https://www.straitstimes.com/singapore/proposed-restrictions... https://www.straitstimes.com/singapore/politics/bill-limiting-use-of-tracetogether-for-serious-crimes-passed-with-govt-assurances https://www.straitstimes.com/singapore/politics/bill-limitin...

userbinator 5y ago

Also, there's a "we can't show you the exact CP image that one of yours matched, because that would also be illegal" catch-22 that basically gives them the power to shroud this whole system in legally-obligated secrecy. Secrecy which will no doubt be abused to hide much more.

bryanrasmussen 5y ago
I do worry this is basically automating and scaling up the whole "idiot working at the mall photo development booth turns family in for child porn because they took pictures of 3 year old in bath" story that shows up every now and then.
- mlindner 5y ago
  It's exactly that though. Except instead of the "idiot" it's "completely dumb neural network".
- bryanrasmussen 5y ago
  reading through Daring Fireball's technical discussion of it https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope https://daringfireball.net/2021/08/apple_child_safety_initia... I suppose does much to allay my worries though, although my natural distrust of authority and the powerful still makes me not like anything like this - no matter the spin.
- deleted 5y ago
  [deleted]
- Ancapistani 5y ago
  I spent about a year working a retail photo processing lab - I can assure you that what you describe is exceedingly rare. I saw “questionable” images on a probably weekly basis. Maybe once a month something would come through that I thought warranted bringing my supervisor over to provide a second set of eyes because I didn’t think it warranted calling the police over but wasn’t entirely sure. An example of that that comes to mind was a series of photos that appeared to show a young woman bound and gagged to a pipe in a basement area - but she was obviously of age, her eyes in the photos didn’t look fearful, and the rest of the roll showed her free and apparently happy. Finally, I was the person who accepted the roll of film and knew that she was the one who dropped it off. We didn’t call anyone on that, but I figured it wouldn’t hurt to get a second opinion. When she came to pick them up, I did ask her to review them and make sure everything was OK with them. Twice in about a year we have photos come through that were obviously what’s now called “CSAM”. In both cases I called the police without consulting management then told them afterward. Also in both cases, the negatives were put into the store’s safe and the owners were arrested on-site when they came to pick them up. All of this is to say - photo lab techs see some shit. If they called the police every time there was a picture of a kid in the bath, police brutality would be at an all time low because they’d not have time to respond to anything else. :)

midev 5y ago

> people misunderstand how it works You can literally read any thread on HN or Reddit to see this is the case. Thousands of comments about facial recognition and AI, completely misunderstanding how it works. The EFF article itself was FUD. The EFF article starts right off claiming a backdoor, which is completely incorrect. Taking hashes of client-side content is not a backdoor, and the EFF should be embarrassed for not understanding that.

rdedev 5y ago
This client side scanning of local files can be easily extended to scan the whole device. That's why the eff was calling it a backdoor. All it takes is for the right govt to put the right preassure on apple. In a way it's kinda worse than a backdoor since your phone is actively reporting any suspicious looking files to the authority all the time on its own. The govt dosent need to actively look for it. Just add the appropriate has of what it wants discovered to the db and wait
- midev 5y ago
  > This client side scanning of local files can be easily extended to scan the whole device Correct, Apple can access unencrypted files stored on your device. > That's why the eff was calling it a backdoor Accessing unencrypted files on a device is not a backdoor > actively reporting any suspicious looking files to the authority all the time on its own Yep, that's a problem. Not a backdoor, though.
  - rblatz 5y ago
    What is your deal. You are out here arguing in bad faith all over this thread. What is your motivation?
    - midev 5y ago
      What's the bad faith? What's single thing have I said that is incorrect? It's not breaking E2E encryption. No one is locking people up for political photos. It's not using facial recongition. I don't like FUD driving discussions. I think we need to have serious discussions about this technology. I think people that don't understand it shouldn't participate....

tasogare 5y ago

> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. They don't believe this. The lack of understanding trick is often used by governments and corporations because it robs the opposite party the choice to oppose, as opposition is framed as lack of understanding instead of a political choice. It's a semi-elaborate way of calling idiots people who disagree. This is why governments "educate" on topics they don't people to oppose.

Lammy 5y ago

> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. I've read too many contentious-change memos to believe it's a mistake. These things are authored by committee with Legal always present, and everything is written expecting it to leak. It's just a passive way to manufacture consent by convincing at least some segment of people to doubt their own thought processes. I'd call it "gaslighting" if that word wasn't already massively misused. See also: iPhone 4 owners who were "holding it wrong".

smnrchrds 5y ago

And HN's mistake is thinking any of this will matter. Apple only cares about money, always has, always will. And I'm willing to bet they are going to continue beating their revenue and profit record quarter after quarter. When their actions do not affect the one thing they care about, why should they care what anyone says?

jacquesm 5y ago
Ah ok, so as long as it is profitable ethics don't matter. Got it.
- kyriakos 5y ago
  Companies don't have to be ethical, they just have to be legal.
  - jacquesm 5y ago
    This is the sort of reasoning that got us into all this crap in the first place, and it certainly won't get us out of it.
    - chronic2703 5y ago
      > This is the sort of reasoning that got us into all this crap in the first place I don’t consider us in “crap”. I welcome Apple’s decision.
    - kyriakos 5y ago
      Not saying its good but these are the facts. Companies try to maximise profit within the boundaries of the law.
  - quonn 5y ago
    So just like everyone else? We hold people to certain standards and can do the same for companies.

pinum 5y ago

About that "manual review"... Refer to pages 10 and 11 of the technical paper. >The server then uses the decryption key to decrypt the inner encryption layer and extract the NeuralHash and visual derivatives for the CSAM matches. This "visual derivative" term shows up repeatedly. To me, the implication seems to be that Apple doesn't look at the actual suspected image before deciding whether to proceed with a report. Instead, I infer that they only verify whether (as the device reports) the image's neuralhash is indeed present in the NCMEC database. If my understanding is correct, their "manual review" process actually provides no protection at all against collisions or erroneous database entries. Further supporting this, on page 4: >Apple reviews each report to confirm there is a match It only refers to a match, not about whether the image appears to be illegal. This makes perfect sense from Apple's perspective- who would want to be in the business of reviewing reports of probably-illegal images?- but it means that the references to a manual review safeguard would seem to be false reassurance. Maybe I'm misunderstanding the paper.

smsm42 5y ago
I don't see any reasonable way here. Either somebody looks at the images - i.e. they have some underpaid poorly trained grunt to look at horrible abuse images which is probably illegal to look at whole day, having about 3 seconds to make a decision on each - or what they're doing is just a sham, "Did the computer say 'it matches'? Yes it did! Review complete, match confirmed!" I don't see any non-horrible way of doing non-sham reviews here.
- peakaboo 5y ago
  People keep trading away privacy for convenience. We won't have any privacy left if we continue to go down that road. People need to start looking at these tech giants with extreme scepticism. Like we do if companies are from China. We need to think about our American companies the exact same way, and stop trusting them to care about our privacy.
  - ptero 5y ago
    > We won't have any privacy left if we continue to go down that road. If people look at today's state (phone movement tracking, voice capture, cameras and face ID, etc.) from the mental model of 30 years ago, they would see neither privacy nor freedom. At all. Sadly, it is not the case of preserving what we still have, if we want freedom and privacy we will have to reacquire them. And a lot of the price will have to be paid in blood.

deleted 5y ago

[deleted]

mctub 5y ago

This still means iOS 15 is adversarial and *Apple lacks credible neutrality*.

xxs 5y ago

>"A cryptographic hash + file size combo" SHA already uses the length in the output hash. Having the explicit length is quite superfluous

verygoodname 5y ago
> Having the explicit length is quite superfluous Not entirely. It makes it so that, to achieve a "full" collision, you have to ensure that the sets of data collide both in SHA hash and in length, helping to prevent attacks that rely on appending/prepending/removing data (for example, "length extension attacks" involve manipulation of the hash by appending data). TL;DR: It is harder to find a collision SHA(B) for SHA(A) if you add the additional constraint that the length of B must match the length of A.
- tialaramex 5y ago
  This is completely wrong. The known collision attacks for the MD-family and SHA-1 all in fact produce collisions with the exact same length. The method used necessarily does this.
  - verygoodname 5y ago
    > This is completely wrong. Which part? The fact that storing "length" along with a hash is not superfluous? You can probably find many things which have a SHA hash of "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb" (infinite things, if we assume arbitrary-sized inputs), but you can only find ONE thing which has that hash and has length 1. I just made it impossible (not just unlikely) for you to find a collision. > The known collision attacks for the MD-family and SHA-1 all in fact produce collisions with the exact same length. Emphasis mine. And note that I did not claim otherwise in my comment.
    - tialaramex 5y ago
      > Which part? The fact that storing "length" along with a hash is not superfluous? The part where you make a false claim out of ignorance. > You can probably find many things which have a SHA hash of "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb" No reason I should go looking for such things. You're the one making the false claims, if you have found "many things" with that hash then list them to prove your point, otherwise go away.

ren_engineer 5y ago

A good way to think about any potential law like this is to imagine your worst enemy or most hated politician was given power to abuse said law, and then think about all the ways they could use it to make your life miserable always think about the worst case scenario when it comes to privacy, because that will be the end result

telxoss 5y ago
To me we don't even consider the actual worse case scenario. Is there any doubt 50 years from now all this will be on algorithmic autopilot for digital authoritarianism? All societies will have Chinese style monitoring but to an unimaginable degree even to people in China right now. People outside IT were barely even using the internet 25 years ago. We are at the very start of this and already too far gone. You just have to enjoy these times and what we still have.

jimbob45 5y ago

I disagree. Google announced the same thing for Google Drive and didn’t receive this level of pushback. If you’re Apple, the only way that’s possible is if you explained it badly (i.e. differently than Google did). I’m guessing their real prerogative is to keep unlawful content from ever reaching their servers. You wouldn’t know that from reading their press releases though.

chaostheory 5y ago
The difference is that Google doesn’t advertise itself as a bastion of privacy. Google spying on you is expected. More consumers also pay for iCloud vs Google Drive. Expectations change when you’re paying for a service.
- okamiueru 5y ago
  Isn't the difference that with google drive, they scan the files you upload to their servers? And, this latest apple debacle is on scanning files locally on the device itself?
  - ziml77 5y ago
    If Apple is to be believed (and doesn't change anything later) then they're doing the same thing. They're scanning the things that are being uploaded to iCloud. They're just doing it locally instead of doing it after they're on the iCloud servers.
john_minsk 5y ago
Why wouldn't they show a pop-up on the phone highlighting which images may be problematic and that report is being sent to the government? I might change my mind on uploading these things to my iCloud. Can't wait for all examples of false positives that will surface. Why wouldn't they delete them directly, but rather send report to the government? Why wouldn't they go even further and if image detected on the webpage that is problematic => It doesn't allow it to get to my filesystem either, so that I don't get into trouble if for some reason police decides to search my laptop. All relevant questions. Answer is the only one: so that when police knocks on your door you were blissfully unaware. Quite honestly this is all too sad. Again this argument about CP that everyone will buy, but which doesn't work in the wild. There are some bad people out there, but if these people are into buying or creating CP => they will rarely upload it into iCloud. Especially now, when everyone knows Apple is looking for it. So they will go even deeper into hiding, while everyone else will have to live with devices that are spying on them...

stjohnswarts 5y ago

I see whataboutism (google drive scans! ms scans!) and "here are the technical details of why we can't just look..." . The bottom line is that none of that matters. the 99.999% problem is Apple is spying on you on your. own. device. I don't know how many techies I've had to shoosh and point that out. It doesnt matter what kind of hash or algo or "layers of protection" are being used. What they are doing is inherently wrong and totally the antithesis of their privacy mantra. It will also inevitably be used by governments for ant-humanitarian purposes to imprison people for thought crimes against the state. Unless they (Apple) backpedal this abomination soon I'm definitely minimizing my apple purchases in the near future and moving off the platform entirely long term. Even google isn't so bold as to put their spyware on their devices to police you for the government. Suddenly ads don't seem that bad ...

smsm42 5y ago
The mistake here is thinking iphone is your own device. If they can view anything stored there, remove anything from there, control anything you install and run, disable it at any moment they wish - do you actually own it, or you just granted the use of it as long as Apple approves of it? Apple certainly doesn't ask your permission to do anything on "your own device" - but you have to ask theirs. Is that how ownership works?
- phs318u 5y ago
  Having ditched Apple laptops this year (after 18 years of Mac computers) for Linux (on System76 hw), I've been wondering what it would take to pry the iPhone out of my hands. I think I've found it.

BiteCode_dev 5y ago

No, Apple fully understands the situation, and in fact, you don't see many Apple fanboys in the comments defending the firm for once because even they know it's wrong. Apple just communicates this message because that's what they need to say for their business, that's all. And they can do so because they also know this uproar will disapear like tears in the rain in a short time if they keep smiling. Their bottom line will not be affected because people don't care. Remember that now, Bill Gates, who we used to associate ti Satan in the 90s, is now viewed as a hero. Remember that about 1 american out of 160 is working at Amazon, a company with the reputation of treating their employees terribly. Most people hence knows somebody impacted by this, but don't stop shopping at amazon. Remember you could say you grab them by the pussy and be elected president. Remember people gives in mass their data to someone who said "they trust me, the dumb fucks". Remember that when somebody risked his life to unveil a massive conspiracy to spy on the entire world, the person has been hunted as a traitor. Remember that one president can lie about WMD, go to war against the vote of the UN, kill thousands of civiliands, spend 600 billions of dollars there while poverty is rising at home, and suffer no consequence whatsoever. Most people have forgotten already. Hell, my girlfriend don't even remember the name of the guy. But it's you that have to be monitored all the time for CP. And you will be punished harshly if you behave badly, not just according to law, but also according to social contracts, both which powerful people can escape now with money and PR. That's the world we are living right now.

smsm42 5y ago

It's not a mistake, it's a rhetorical trick - to present somebody opposing your actions as a misinformed idiot that just doesn't know what they're talking about, and once proper technobabble is deployed every reasonable person would realize their mistake. And if they don't, they must be much bigger idiot than we thought, there's no other reasonable conclusion.

tinus_hn 5y ago

Quote: > and more than a few are worried about the implications

2Gkashmiri 5y ago

yes. i live in a small 8 million strong "valley" where the government is hellbent on destroying people. https://kashmirobserver.net/2021/07/10/11-govt-employees-including-2-sons-of-hizb-chief-salahuddin-sacked/ https://kashmirobserver.net/2021/07/10/11-govt-employees-inc... they sacked these government employees because they allegedly were engaged in "anti-national activities", no courts, no trial, no hearing. just sentencing that since you are allgedly found to be engaged in such activities, your employment is terminated and there is no recourse. https://www.organiser.org/Encyc/2021/8/3/Govt-s-policy-of-No-passport-for-stone-pelters-unnerve-Kashmir-Netas.html https://www.organiser.org/Encyc/2021/8/3/Govt-s-policy-of-No... here they have decided to not give passports to protesters or give then government jobs because "anti national activities" again the same trope. this might be a small issue but here is a jounalist https://thewire.in/rights/kashmiri-journalist-masrat-zahra-police-assault-father https://thewire.in/rights/kashmiri-journalist-masrat-zahra-p... whose father was beaten to a pulp because she being a decorated journalist had criticized the government, they slapped terrorism charges on her and german government gave her asylum so they did the next best thing, hurt her parents because why not. now if she didnt have her passport because she was already a known suspect of "anti national activities", asylum might be a little difficult so they want that. i know 100% indian government and for that matter pakistan government as well will use this technology for "national interests" in finding out protesters and dissidents and they either do not even go for trials or if they do, the same are sham ones so doesn't matter if they make a show of a trial because the guilty are condemned already.

speleding 5y ago

> A cryptographic hash + file size combo is unlikely to have a false positive within our lifetime The chance of a "technical" false positive is tiny, but they need to look all false positives: If some joker sends you a lewd picture through WhatsApp, WhatsApp by default saves every picture to your photo library, then you are now on the naughty list. Good luck trying to explain yourself out of that one.

ziml77 5y ago
WhatsApp saves all received images to your library? That seems hard to believe. I can't imagine that anyone would appreciate having their personal photo collection cluttered with other peoples photos and all of the meme images that people share.
- speleding 5y ago
  It's a setting, called "Save to camera roll". Default is on.
  - ziml77 5y ago
    Strange. Is that a desirable feature for WhatsApp users? I must be overlooking a use case that makes that a sensible default.
tyingq 5y ago
The context there was that a typical CSAM hash like PhotoDNA is NOT a cryptographic hash, but a perceptual one, which DOES have notable false positive issues.

arthur_sav 5y ago

Seriously, the backdoor is in place. Now all they have to do is keep adding hashes to the list and voila. Trump supporters = terrorists = you're in jail because of political games.

lovelyviking 5y ago

>these features are part of an “important mission” for keeping children safe. The other mistake is to think that removing privacy with longing effects on society [1] and thus attacking the very fabric of a free democratic society respecting human rights would not make children any safer. When they grow up they possibly would have to live in concentration camps or Chine-like regime. It is very safe, but not sure it is very pleasant. It's about values. It should be absolutely illegal to install any spyware on personal device without warrant [1] https://news.ycombinator.com/item?id=28084578 https://news.ycombinator.com/item?id=28084578 [2] https://news.ycombinator.com/item?id=28096948 https://news.ycombinator.com/item?id=28096948

Perstone 5y ago

They don't believe that people misunderstand how it works. This is marketing and PR speak for "We are going to do this and we are going to push enough tech jargon out there that most people won't understand nor care about the difference".

mberning 5y ago

You nailed it. Apple is so flexible with their morals when money is on the line you know this will be abused by some authoritarian government. They will ask apple to scan for some content, apple will say no, the government will ban apple from selling their products, and suddenly apple will change their mind. Next thing you know a huge swath of journalists or dissidents is being disappeared to reeducation camps. Apple is working overtime to wrestle the “most evil company in the world” title from google and amazon.

queuebert 5y ago

Isn't perceptual hashing necessary for this application to prevent simple modifications (e.g. resizing, filters) from changing the signature?

ben_w 5y ago

Yeah, this is my reading of the situation too. From a technical point of view, what Apple is doing seems like a reasonable approach to me; what I absolutely do not trust is any claim that it will never ever be expanded to include, say, governments requiring similar tech be applied to all messages before they are encrypted in order to look for dissidents… and then generally at opposition parties. (And of course there are enough competent technologists available for making custom encrypted chat apps that the hostile governments could only achieve their goals by putting the AI in the frame buffer, which opens the door to subliminal attacks against political enemies — targeting the AI with single frames such that the device user doesn’t even notice anything). But I know I have paradoxical values here: On the one hand, I regard abusive porn as evil. I want it stopped. On the other, I have witnessed formerly-legal acts become illegal, and grew up in a place where there are sexual acts you can legally perform yet not legally possess images of. I think the governments (and people in general) follow a sense of disgust-minimisation, even when this is at the expense of harm-minimisation. And any AI which can genuinely detect a category of image can also be used to generate novel examples that are also in that category without directly harming a real human. I don’t have a strong grasp on what I expect the future to look like. My idea of what “the singularity” is, is that rather than a single point in the future where all tech gets invented at once, it’s an event horizon beyond which we can no longer make reasonably accurate predictions. I think this horizon is currently 2025-2035, and that tech is changing the landscape of what morality looks like so fast that I can’t see past that.

squarefoot 5y ago

It's just a matter of time until someone can create a "ThisChildPornDoesNotExist" site, then send links to targets to have their lives ruined. I am extremely worried by humans policing the Intertubes, and even more worried by algorithms doing so because they make the perfect excuse to do sloppy checking. See Youtube DMCA related takedowns for multiple examples.

dragonwriter 5y ago
> It's just a matter of time until someone can create a "ThisChildPornDoesNotExist" site, then send links to targets to have their lives ruined. “Their” in this case referring to the sender, which makes the particular problem somewhat self-limiting, right?
- squarefoot 5y ago
  Probably so, but a more smart malware that grabs fake images from an encrypted server somewhere, then plants them in the victims PC/phone/whatever, then deletes itself, isn't rocket science.

lowbloodsugar 5y ago

Apple: our staff can look at your tits if you take a sexy picture and look young enough or some other magic button. Hm. Seems like a good attack would be take a celebrity, photoshop some nakedness, and put that in the database. Output: real photo of said celebrity. This is not acceptable.