18 ms·

Tainting the CSAM client-side scanning database

251 points by raybb 3y ago

miohtama 3y ago
Because client-side scanning is not going to work, and no one wants to government issued black box binary to send their conversations and photos to unnamed police person randomly, the non-compliance is the only way. People just start to use chat programs in the EU that do not comply. This would be Signal, Telegram, others. The EU can fine and fight with Meta/WhatsApp, Apple, others, but that’s about it. The EU bureaucrats do not have power to magically insert spyware in our devices as long as we can install our own software. … which of course means we know what’s going to come next.
- miohtama 3y ago
  Also do not miss the recent good article about the US software development companies who are behind the lobbying of client-side scanning, how is this madness is being funded and by whom. https://balkaninsight.com/2023/09/25/who-benefits-inside-the-eus-fight-over-scanning-for-child-sex-content/ https://balkaninsight.com/2023/09/25/who-benefits-inside-the...
  - robertlagrant 3y ago
    Thanks - interesting. Phrases like this are eternally surprising: > The same month, UK officials privately admitted to tech companies that there is no existing technology able to scan end-to-end encrypted messages without undermining users’ privacy It's as though this is a secret that UK officials knew. Everyone technically minded knows this; it's a battle between those who regulate (the difficulty of whose job generally stops at "write down the rule and fine people if they don't do it") and those who know about current fundamental limitations.
    - seanhunter 3y ago
      Remember this is the same UK government that at one stage back in 2020 was proposing that there could be borderless customs enforcement between Ireland and Northern Ireland (part of the uk) "using drones". Because everyone knows drones do that. I don't think they consider themselves limited by things technology is actually capable of when talking about what they are going to get technology to do.
  - uconnectlol 3y ago
    cloudflare also comes to mind, as they proudly do whatever they can in their power to check off that CSAM box as long as it makes them money somehow.
- Sayrus 3y ago
  And by next you mean in parallel. Spinning E2E encryption as a tool used only by bad actors is very common and pushing browsers to implement governments managed blacklists seems quite in progress. Banning VPNs from social networks (and more) was even suggested in France but quickly removed due to backlash.
- pmontra 3y ago
  Governments will mandate by law that the OS do client side scanning by peeping inside the app directories. Apple, Google, Microsoft will comply and realistically, which phones and computers are we going to use if we don't buy from them? Desktop Linux might become illegal.
  - miohtama 3y ago
    Yes, the EU is already working to make this illegal https://mullvad.net/en/blog/2023/2/1/eu-chat-control-law-will-ban-open-source-operating-systems/ https://mullvad.net/en/blog/2023/2/1/eu-chat-control-law-wil... The best bet is Android fork based on Android open source distribution, without Google.
  - SV_BubbleTime 3y ago
    Right. Per program makes people use different programs. Per OS, ok, maybe people will use a different OS (finally! Year of the Linux desktop!) … ok, we’ll just roll it into the three companies making GPUs. Driver binary side scanning. Maybe hardware frame buffer scanning? All is this is framed around children, but you are an absolute clown if you think that is truly the end goal. It’s always “for the children”.
- OfSanguineFire 3y ago
  Signal advises its users to install the app from the Play Store. While you can still get the .apk from Signal’s website, the developers warn against that. The EU can definitely exert pressure against what is hosted on the Play Store. Telegram famously lacks end-to-end encryption (unless you intentionally use its private-chat feature, which few people do) and shouldn’t be mentioned in the same context as Signal.
  - kzrdude 3y ago
    If you enable private chat, you don't get notifications for new messages, that way the feature is self defeating, it seems to me at least. (Something could also have gone wrong?)
    - CuriousCosmic 3y ago
      Notifications generally go through Google Firebase so by not running private messages through firebase they avoid potential leaks. At least that's my guess
      - kzrdude 3y ago
        That sounds pretty bad, a cloud service needs to see all notifications?
        miohtama 3y ago
        A good question. I am not sure if notifications on iOS, Android, Google Chrome or Firefox are end-to-end encrypted.
        nickphx 3y ago
        You can send whatever values you desire as long as your service worker handles it.
        kube-system 3y ago
- deleted 3y ago
  [deleted]
- jacksnipe 3y ago
  At the moment, it is perfectly possible for e2ee to be compliant.
- reisse 3y ago
  > The EU bureaucrats do not have power to magically insert spyware in our devices as long as we can install our own software. EU bureaucrats have power to force ISPs to block connections to the messengers that would not comply, Chinese-style. They already do block websites they don't like - e. g. in Cyprus local provider CYTA shown me "access blocked due to EU comission regulation #whatever" for most of the Russian media sites.
- JopV 3y ago
  The EU is trying to make it mandatory to allow sideloading. This is bad, but also the opposite of what you are saying.
- deleted 3y ago
  [deleted]
polygamous_bat 3y ago
Any computational method that relies on a function that converts m bits (an image) to n bits (a fingerprint) where m > n will always be vulnerable to such an attack. The smaller n is compared to m, the easier it is to counterfeit something with that signature. It is not new knowledge. The only way to be certain, unfortunately, is for a human to look at it. When the allegation is as serious as CSAM, I would rather be very very certain before accusing someone, rather than taking a shotgun approach and just randomly searching phones. But what do I know.
- mattashii 3y ago
  > Any computational method that relies on a function that converts m bits (an image) to n bits (a fingerprint) where m > n will always be vulnerable to such an attack. The "vulnerable" depends on your definition: SHA-2 and SHA-3 are both still quite safe against preimage attacks, and even second preimage attacks require significant work to pull off for SHA-2, and I am unaware of any meaningful second preimage attack on SHA-3. Of course, SHA isn't built for finding similar images, but for finding exact matches this should be safe enough.
  - lelag 3y ago
    But then bypassing the filter would just require to change a few bits here and there.
    - olliej 3y ago
      No the whole purpose of these “hashes” is that they’re robust to that. The attack model they’re designed for is image manipulation to avoid the hash match, not manipulating manipulating images to trigger the hash. There are numerous papers on doing bit manipulation to cause miss classification.
      - minitech 3y ago
        The context of the comment you’re replying to is a cryptographic hash.
bastawhiz 3y ago
The issue described here, to my understanding, is that you find or create csam and then manipulate it so that its fingerprint collides with another image that you want to be flagged as csam. You then submit the manipulated version of the found or generated image to the authority. First, at what point does the authority go "uh... Where did you get this from?" Practically speaking, the people doing this would have to be authorized law enforcement, no? Like if an ordinary citizen showed up with csam the excuse "oh I found this JPEG in a trunk in my late grandpa's attic" isn't gonna fly. My assumption is that the central authority that compiles the database isn't just taking fingerprints, they want the actual source images. And I'm assuming they actually look at them to ensure they are, in fact, csam. Otherwise anyone could put anything into the database. Now let me be very clear: I hate the idea of my device scanning for csam on principle. But what's the attack here? That someone looks at my device and determines that I do not, in fact, have csam? And that false positive might even be investigated and reveal that the bad actor sought out or generated CSAM for the purposes of perpetrating this! Which is even worse! If the police want to look at your device, they can do that already through any number of other sketchy means that don't involve what's likely weeks of effort. The US searches phones at the border when they're feeling like it. I don't want to defend this stuff, but the potential for abuse in this specific case seems vanishingly small.
- pydry 3y ago
  My understanding was that you generate a csam with a colliding fingerprint and put it online and trick somebody with an iPhone or whatever into clicking on it. Now they're on a watchlist and will get harassed by the police.
  - bastawhiz 3y ago
    So you have to plant the manipulated csam somewhere that it'll be found by law enforcement and added to the database and just hope you did a good enough job to not be tracked?
    - deleted 3y ago
      [deleted]
heisenbit 3y ago
Anyone who thinks the injection of malicious hashes is an unrealistic scenario should take a look at the games now being played with Youtube‘s content protection system which is leveraged by scammers and blackmailers.
- nopurpose 3y ago
  what is happening in Youtube world?
  - Macha 3y ago
    People claiming (via manual reports or via content ID) content that is not theirs to either obtain other people's ad revenue or cause the takedown of content they disagree with
  - jabroni_salad 3y ago
    This one happened just a month ago so it might be what the parent comment is referring to. Basically, somebody made a fake company to make bogus copyright claims against someone to hurt their channel. Youtube refuses to deliver counterclaims unless the YTer puts their government name on it (he originally tried to deliver it via an attorney). Additionally, the other party is actively trying to compromise the YTer's other accounts and identity to damage him, so any new data point given to this person represents risk. I don't really care for this YTer (they made a video essay about someone who doesn't really want to be in the media circus anymore, could just MYOB) but the methods documented in this video can be used to doxx any user that uploads a video. All you gotta do is convince youtube that you own some piece of common non-royalty media that a youtuber uses to gain leverage on them. A lot of this kind of media is old with unfindable owners, so even if YT does want to spend the time to validate, there's nowhere to go with it. https://www.youtube.com/watch?v=hixwIOd_C44 https://www.youtube.com/watch?v=hixwIOd_C44
    - omscs99 3y ago
      What I don’t get is this: why does youtube still have such a monopoly after all this time? It’s had a shitty reputation since I can remember, why don’t creators just band together and take their viewership somewhere less hostile?
solusn 3y ago
[flagged]
- npteljes 3y ago
  Can you point to any society anywhere, where people learned to behave?
- whimsicalism 3y ago
  I am generally against the death penalty, but I am particularly against the death penalty for anything other than murder because of the actions it incentivizes.
jeffrallen 3y ago
Another case of setec astronomy.
pmontra 3y ago
The only thing that can save us is a huge number of false positives and a backslash on the governments in charge, when they'll have to concede for the 10,000th time that it was not CSAM but parents sending their baby pictures to the grandparents.
- jopsen 3y ago
  Looking at the cookie banner stupidity, I don't think a huge number of false positives would generate backlash.
qingcharles 3y ago
How do these databases differentiate between AI generated CSAM and CSAM of real victims? (Since many jurisdictions only criminalize real CP) I know that 99% of people cannot tell an AI image from a real photo since that "Last giant irish greyhound 1902" photo has been going around on social media for weeks, and it is, to me, unbelievably obvious AI.
- chatmasta 3y ago
  I assume the answer to that will be that there is no need to differentiate between them. And honestly, I agree with that argument. Possession of CSAM should be illegal regardless of whether it's "real" or not. But the proposed scanning system is the wrong solution, regardless of any "real or AI" ambiguity, because it's possible to generate false positives with nonsense images that aren't even close to the expected CSAM, real or otherwise.
  - whimsicalism 3y ago
    I am of the opinion that regardless of whether they are both illegal, the penalties for “AI” should be significantly less.
    - anononaut 3y ago
      What's your reasoning?
  - postalrat 3y ago
    Seems like you want to bring all the success of the war on drugs to the war on generated images.
    - chatmasta 3y ago
      No, I don't support any automated scanning system or really any sort of "going out of our way" to find new criminals. The reason I think it's a bad idea to differentiate between real or AI is similar to the arguments against "means testing" for distributing benefits. You don't want to put real victims in a situation where they're deprived of justice because they can't prove that their victimization was "real." Imagine a real CSAM criminal claiming a defense that they "thought it was AI generated." Do you want to give them that out? If protecting those victims comes at a cost of punishing criminals possessing AI-generated CSAM with sentences equally as harsh as those for "real" CSAM, then it's a worthwhile cost to pay. They are still criminals, and they are definitely not innocent (unless they're being framed, but that's a risk with both real and AI images).
cabirum 3y ago
I think it's pretty clear this is not about "CSAM", we have to stop using the term. It's just censorship, plain and simple. Client side means you'll pay from your own pocket for this wrongthing detector to work. It can even be automated so as soon as the detector gets triggered by anything, you'll get locked out of your bank accounts, until further notice I guess. If this thing gets a serious discussion in a parliament, get rid of the parliament.
- ramses0 3y ago
  But parliament wants to seed your camera with mugshots of the FBI's top-ten most wanted list so the instant a false positive appears (directly on the camera, potentially even prior to writing the image to disk, potentially even prior to pressing the snapshot button)... they beacon an alert (or exfiltrate piggybacking via Bluetooth/AirTag/Covid exposure tracking mrchanisms), and bob's you're uncle.
  - SV_BubbleTime 3y ago
    I’m starting to think that a government large enough to get all the things it could want might be a bad idea.
    - timschmidt 3y ago
      I'd generalize this to any hierarchy. That includes corporations, institutions, religions, governments, cults, basically any time there are more than two people in the same room.
  - morkalork 3y ago
    Sounds like there's going to be a market for vintage cell phones built before that happens. Although eventually they'll stop being useful as gWhatever rolls out and telecom providers stop supporting 4/5g.
- INTPenis 3y ago
  What makes the debate confusing is that some people probably believe it is about CSAM. Like the Swedish PM Ylva Johansson for example. She probably believes in what she's doing, and other forces are simply taking advantage of her crusade.
_lvbh 3y ago
Edit: comment deleted
Hizonner 3y ago
It says: > This shows that the database can be tainted with non-CSAM material by an entity that can submit entries to it. Actually, it can easily be tainted by anybody. Take your massaged hash-colliding image, which remember is still visually child porn, and post it on some pedos-R-us forum. The people who maintain the database actively troll those forums. They'll see the image and add the hash to the database for you.
- regularjack 3y ago
  You'd be publishing child porn, which I think is not the wisest thing to be doing.
  - Hizonner 3y ago
    Tainting the database is probably already chargeable as something. The point is that the person doing this doesn't expect to get caught. And frankly they're probably right in that.
    - semiquaver 3y ago
      > Tainting the database is probably already chargeable as something This is definitely true. Good old “intentionally accesses a computer without authorization or exceeds authorized access” in the CFAA. It’s so vague as to be able to make almost _anything_ the government doesn’t like involving computers illegal.
  - mcpackieh 3y ago
    A spy agency or secret police doing this to use the client-side scanning system as a surveillance/censorship system for political media would not care about that.
    - giantrobot 3y ago
      Even if such systems were only used for their stated purpose of finding CSAM, because no one can audit the database of hashes there's no guarantee that non-CSAM images aren't in there. Just being accused of having CSAM is a life ruining event. Even if someone is eventually cleared of charges their life is forever altered. The "We Got Him!!" headlines are front page news, retractions are filed in a basement filing cabinet with a sign that says "Beware of Leopard".
nopurpose 3y ago
If there were N algorithms, is it feasible to create B' image such that its fingerprint matches A in every algorithm?
- huggingmouth 3y ago
  If if isn't, then it becomes technically (and likely) feasible for every client that runs those algorithms to reconstruct A. This is a really stupid idea. Politicians who are demanding this are way out of their depths.
mikotodomo 3y ago
IFL these threads always give vibes like, "oh, thank god that anti CSAM measure wouldn't actually work".
- bloopernova 3y ago
  I think that some people are terrified that their (possibly AI generated) "loli pictures" would be caught by a scanner.
  - giantrobot 3y ago
    What I'm concerned about is a system that flags me for a crime based on a database I can't audit based on mechanisms with an entirely too high false positive rate. Because the database can't be audited by anyone but a select group we have to trust that it only contains actual bad images. I do not trust that such databases don't also contain images that are embarrassing to powerful/connected people. I also do not trust such databases don't contain false positives. The sort of people that are super zealous about a topic aren't simultaneously super rational and objective about that topic. There's a non-zero probability that those databases contain lewd yet entirely legal images that the submitters just didn't like. Because of the false positive rate a photo of my dog might trigger an alarm and then my phone sends an automated message to the police. I'm then told by proponents there will be some manual review. I have to then hope that the local DA doesn't have an election coming up and wants to push a "tough on crime" message so charges me with a crime despite a review. In short these scanning systems require far too much unearned trust. They also present a slippery slope thanks to the incendiary nature of the topic. Today it's CSAM but what undesirable content will the systems be used for tomorrow? Such systems require trust in the stewards of today and tomorrow. Do you want people of the opposite ideology to you in charge of such systems? Do you trust they'll never be abused? Do you trust well meaning people never make mistakes? I do not trust in any of those things. I'm not worried about myself doing actual bad things, I'm worried that demonstrable false positive rates will ruin my life with the mere accusation of doing something bad.
    - astrange 3y ago
varispeed 3y ago
Ok, but why not go for routine home searches then? Is it because on the phone, it is "invisible" and cheap? Does it mean that what stops the government from treating everyone as a potential criminal is cost and inconvenience? People were raising alarms about this years ago and were branded as conspiracy theorists. I guess the EU works well in their propaganda department painting themselves as do gooders, where in fact they are just wannabe Stalinists.
- SV_BubbleTime 3y ago
  >Ok, but why not go for routine home searches then? Is it because on the phone, it is "invisible" and cheap? Underrated comparison. That is apt and the whole “for the children” part is smokescreen.
kypro 3y ago
This is basically what I suggested we do a few years ago if Apple added client-side content scanners. Seems all need to here is obtain a finger print of a CSAM image then you can reverse engineer a non CSAM image to match that finger print. Distribute this image wide enough and you effectively render these algorithms useless. Which is a shame because they could be used for good, but it seems kinda obvious this will expand out to scanning for terrorist materials. And again, while I might be in favour of this in theory I am frequently shocked by what fits this definition. In the EU if you promote the wrong ideas or say something "hateful" you can be arrested and sentenced 10+ years in prison. I think what concerns me is that currently you really only put yourself at risk if you make your wrong think public, e.g tweet about your dislike of Islam on Twitter. These client-side scanners would now enable the authorities to not only go after those who say "racist" things online, but also those who view / save "racist" memes. If I were to guess at a motive, this would be it. Legal note: I obviously don't agree with hate of any kind, but I do believe in the right to express views which may be considered hateful. Edit: Fine with the downvotes – but curious what people are disagreeing with? Would you mind leaving a quick comment if you choose to downvote?
- Hizonner 3y ago
  1. You mixed up generating child porn images that collide with targeted non-child-porn images, in order to taint the database and thereby suppress the target images, with generating non-child-porn images that collide with child porn images, in order to overwhelm the system with false positives. 2. You brought in matters that have nothing to do with hashing or child porn, but tend to generate flame wars. 3. You put sneer quotes around "racist", which makes it sound very much like you don't think any such thing exists.
  - kypro 3y ago
    Thank you. 1. Oh okay. I wasn't very clear there. By, "this is basically what I suggested we do a few years ago" I meant that we should look to exploit the limitations of these fingering printing algorithms to render them impractical. But yeah, my suggestion was that we should do the reverse. Rather than taint the database, taint the content. 2. That wasn't my intention. I was suggesting that there are likely other reasons governments are interested in this technology than, "think of the children". Hence, why I think we should seek to ways to render these algorithms ineffective were they deployed. But yes, please feel free to downvote if you feel my comment is unproductive. On reflection I agree it might be. I'm not really adding much here 3. You know what's annoying? I did that because I didn't to make an assertion either way to be as neutral as possible. I'm using quotes because others deem it racist, and that it's irrelevant what my personal position on that is. If you want my position on whether racism exists though, obviously I believe it does – I'd urge you to just assume good faith in the future.
ls612 3y ago
Didn’t the UK just blink when chat apps said hell no we will leave instead? Does the EU expect to get better results? Apple has very publicly come out and said that their proposal like this was a bad idea and I’d expect Signal to also not comply, the only remaining question is whether WhatsApp would threaten to leave again.
- SV_BubbleTime 3y ago
  Apple proposed the exact same thing themselves though.
nickdothutton 3y ago
Government almost never passes useful/wise legislation with regard to technology, because it thinks “tech” is a problem that the government can “fix”. There are systemic reasons for the legislation almost always being bad/unhelpful/foolish and those reasons are not going to change.
grammers 3y ago
Worst of all the EU Commission is now starting to act like fake news promoters by posting misleading ads (German article): https://netzpolitik.org/2023/politisches-mikrotargeting-eu-kommission-schaltet-irrefuehrende-werbung-fuer-chatkontrolle-auf-x/ https://netzpolitik.org/2023/politisches-mikrotargeting-eu-k... It's outraging.
michaelmrose 3y ago
This attack also works for people who are not in a position to submit entries to the database as you could very well plant the generated matching images on a server set up with a compromised credit card and report it yourself. You could select pictures from targets publically available photos or more insidiously look to compromised accounts cloud storage and generate fake offensive images that match peoples actual baby pictures whether for harm or blackmail.
deleted 3y ago
[deleted]
anigbrowl 3y ago
[...] One could conclude that the abuse centre could thus easily spot the malicious entry in its database after a few of these reports all concerned with the same image, and delete the corresponding fingerprint from the database. If that were the case, this avenue of attack would not be a problem in practice. This assumes, however, that the abuse centre keeps track of such false positives over time to detect such maliciously uploaded fingerprints. This may or may not be the case. Why would it not be the case, unless the abuse center wanted to waste time and resources and have unreliable records? This essay is technically sophisticated and socially naive. Tech people keep spinning up the most unlikely and hard-to-explain reasons for why leveraging technology to interfere with the spread of CSAM is bad, instead of working toward any kind of privacy-respecting solution to target CSAM itself. Constantly emphasizing the scope for government intrusion and privacy violation (legit but at the same time overblown to the point of paranoia) and constantly minimizing or dismissing the real harms of CSAM is a great way to alienate the normal non-technical people you need to have on side.
- uconnectlol 3y ago
  okay, ill bite: yes, the blogger has tunnel vision, that's the only valid point in your post, he also missed the fact that anyone can poison the well, not just privileged entities. however your rhetoric is cancerous, although i'm aware that many people who use it are just sheep and have no idea what they're insinuating. CSAM is a propaganda word: 1. right wing uses it to push their agenda of punishing people for having sex 2. left wing uses it to push their agenda of punishing males or kulaks or whatever 3. police use it to make it look like they're solving a crime so they can get paid for nothing: they can arrest someone for looking at a nude picture of a 17 year old (which the suspect is often unaware of), then in the media and police statements: "he was arrested for the possession of child sexual abuse material". and this is the majority of convictions: people (and even teens) being into teens. people are not naturally sexually attracted to children (13 and under) in more than say 0.0001%. so now that we actually looked at the subject beyond the taboo veil, it seems the discussion is just to stop 14-17 year olds from having sex, which is insane and does not justify one single thing that police do about it. CP being illegal is just completely out of the question. we should be debating re legalizing CP, or changing it to 13 and under at the very least (i don't see any harm in the existence of porn of any age since it does not encourage abuse since most people will not become sexually attracted to children no matter how much you expose them to it). so again, your rhetoric is pure cancer. i don't need a "privacy respecting solution to CSAM". what the hell do you think that would be? do we also need a freedom respecting solution to the murder problem? would you have my legs cut off and say i should just order door dash?
Retr0id 3y ago
The article considers "an entity that is allowed to propose new entries to the CSAM database". You don't even need this! You could target a whole "social cluster" of people without having any special privileges within this system. As an example, lets say you want to attack environmental protesters. For image A, you create a meme about climate change. For image B, you procure something that looks, to humans, like CSAM (as described in the article). Craft B′ such that f(B′) = f(A) (also as described). Now, all you have to do is anonymously publish B′ to a platform that is actively moderated/monitored. Unfortunate users will see it and report it as CSAM, and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database. Now you can start sending out image A, the meme, to your target demographic. You won't be able to post it on "mainstream" platforms with server-side fingerprint scanning, but there are plenty of other avenues for it to spread. If it's a good meme, it will propagate organically through group-chats and DMs, and eventually find its way onto devices with client-side scanning. Apple's proposed device scanning system had a threshold before your device would be flagged, so repeat all these steps a whole bunch of times, until the average meme-savvy environmental protester's device gets flagged for further scrutiny. Unwitting victims who do try to post the meme to a mainstream platform with fingerprint matching may risk getting their accounts flagged and taken down, and they might have no way of knowing what triggered it. This would lead to, of course, automated censorship of environmental protest groups.
- hju22_-3 3y ago
  That's some next level shit. I dig it.
- FireBeyond 3y ago
  > Apple's proposed device scanning system had a threshold before your device would be flagged Which always struck me as odd as it would extremely easy to spin this as “Apple detected CSAM on this device but their policy is only to alert authorities once a set quantity of CSAM is found…”
  - LorenPechtel 3y ago
SV_BubbleTime 3y ago
I’m curious how few people have followed this topic of client side scanning through. The logical conclusion is that someone is going to get a knock on their door and a warrant for search because of an unknown signature match on a file some where at some point? And people seriously think this will stop at being “for the children”?
orbital-decay 3y ago
What are the legal consequences of having an image with the matching fingerprint on your device? Is there some kind of threshold to mitigate false positives? What organization is responsible for processing the matches? Is it enough for a search warrant? (for various countries/jurisdictions)
quitit 3y ago
I am against the idea of scanning for the reason that the author pointed out: It's trivial to repurpose the technology to use it in dystopian ways. I however have precisely zero concerns about impersonating hashes: 1. It's trivial to deal with tainting the database: both secondary hashing and more invasive hashes deal with that problem. 2. It's trivial to deal with impersonated hashes, all positives can be scanned on device in a second round with a different hash method. Those which are still positive can have be checked off device in an automated, yet privacy preserving, way such as by sending a low resolution crop with faces blurred to a system that does a likeness check against the matched CSA image: only the real image will bear likeness. 3. These systems have undisclosed minimums. A person sharing CSAM will be generating positive results at a significant rate. Receiving a few images with faked hashes won't set off alarms, and if a victim is suddenly receiving hundreds of images, that would already be obvious. 4. These articles over sensationalise what occurs when a positive match is found. They vaguely gesture to serious consequences from a single match. In reality even without secondary automated checks, the false positive will be noticed by a human and then discarded. If a person is receiving a lot of false positives that may lead to the individual being informed that they are begin targeted. Overall the idea of scanning chat content seems ineffective, for much of the same reasons why it's ineffective to ban end-to-end encrypted chat: those with criminal intent will just use something else or roll their own.
- Hizonner 3y ago
  > It's trivial to deal with tainting the database: both secondary hashing and more invasive hashes deal with that problem. Sorry, what does "more invasive hashes" mean? > It's trivial to deal with impersonated hashes, all positives can be scanned on device in a second round with a different hash method. The double hashing would definitely help. I doubt anybody's gotten a close collision for more than one hash at a time. It's not inconceivable that they could do it, though. As far as I know, there are only two hash methods that get used for this, and I think both of them are based on sliding a window over a reduced version of the image and doing a DCT, so they're pretty similar. I wouldn't be surprised if somebody could fool both of them enough to at least force you to tighten up your similarity threhsholds. It would be safer to come up with something that worked on completely different principles. No idea how hard that would be. I don't think that the second-stage backup, where you send the image off somewhere, is "privacy preserving" by any standard I'd be comfortable with. Well, OK, actually even completely foolproof client side scanning isn't privacy preserving enough for me, but I mean that's substantially worse. I also don't think it'd be easy to find anybody to run that server. And it'd be a significant structural change from what's deployed now. It'd be much more disruptive than adding a second hash (which is isomorphic to just switching to a single composite hash). It took a long time to get the present system deployed, so that big a change doesn't seem "trivial". > These systems have undisclosed minimums. A person sharing CSAM will be generating positive results at a significant rate. Receiving a few images with faked hashes won't set off alarms, and if a victim is suddenly receiving hundreds of images, that would already be obvious. Minimum what? If you just say "undisclosed minimums", my mind immediately assumes you mean threshold similarity scores. But you seem to mean hit counts to trigger various actions. Anyway, both of those are set by the service that's using the database. Those services vary in their strictness and clue level, so I'm not sure you can say anything very general about them. > These articles over sensationalise what occurs when a positive match is found. In the threat model of the original article, the people doing the review for false positives are most likely the same people who poisoned the database to begin with. They've corrupted it to give them hits on images they want to suppress, even though the system operators don't want to support them in suppressing those images. So that whole "reviewed by a human and discarded" step doesn't happen. In the broader world, I suspect that the thresholds for "delete the file" tend to be very low... too low for there to be human review. The thresholds for "disable the user's account" probably aren't exactly stratospheric, either. You can have damaging consequences well short of midnight police raids. Also, "framing" people isn't necessarily the only thing you might use it for. For example, you could try to use it to drive the cost of running the system up to unsupportable levels.
sterlind 3y ago
This is precisely the kind of trolling 4chan loves to do. It ticks all the boxes: swatting, CP, hacktivism possibly if you squint really hard. And since these hashes are presumably resistant to cropping and scaling, they might be able to target individual symbols, people or image macros, like the rainbow pride flag (you know /pol/ wouldn't be able to resist), politicians etc.
thephyber 3y ago
I’m worried the same kind of attack will make this[1] CSAM removal project useless. If there is no authentication of the submitter and no punishment for fake submissions, then it becomes garbage-in, garbage-out. [1] https://takeitdown.ncmec.org/ https://takeitdown.ncmec.org/
Zuiii 3y ago
I'm completely stumped. I just don't understand how politicians can still believe that client-side scanning would somehow stop CSAM. Couldn't those who possess this material also run those exact same CLIENT scanners (if not directly, than via isolated canary systems and emissions monitoring) to detect when files have been added to the scanners? What about tools that keep shuffling files in ways that are not too perceivable but would defeat those scanners? Just look at copyright infringing material on Youtube (it's still there, just better hidden). Nothing short of a government whitelist database containing all files that are legal to possess, or 100% analog human surveillance will solve CASM. Not AI and certainly not this. I'm forced to agree with others here: This isn't about CSAM at all. It's a thinly disguised frontal attack on democracy. Imagine if the government had this technology when people were organizing to repeal discriminatory laws in the US. Imagine if authoritative governments in the east (or anywhere else) had this technology right now. Take a long hard look at every person or organization who supports this nonsense and pay close attention to them. They are the ones who want to harm us and our children.
MagicMoonlight 3y ago
The database would obviously come from a central organisation like the FBI or NCA and be built from material gathered in investigations. They wouldn’t allow larry from reddit to share his child porn with them and break the database. Anyone with sufficient power to want to target you wouldn’t use a ridiculous exploit like this. They’ll just go to google and say “change the update server for this user to this new address” and then breach your device with the next software update.

251 points by raybb 3y ago

Because client-side scanning is not going to work, and no one wants to government issued black box binary to send their conversations and photos to unnamed police person randomly, the non-compliance is the only way. People just start to use chat programs in the EU that do not comply. This would be Signal, Telegram, others. The EU can fine and fight with Meta/WhatsApp, Apple, others, but that’s about it. The EU bureaucrats do not have power to magically insert spyware in our devices as long as we can install our own software. … which of course means we know what’s going to come next.

miohtama 3y ago
Also do not miss the recent good article about the US software development companies who are behind the lobbying of client-side scanning, how is this madness is being funded and by whom. https://balkaninsight.com/2023/09/25/who-benefits-inside-the-eus-fight-over-scanning-for-child-sex-content/ https://balkaninsight.com/2023/09/25/who-benefits-inside-the...
- robertlagrant 3y ago
  Thanks - interesting. Phrases like this are eternally surprising: > The same month, UK officials privately admitted to tech companies that there is no existing technology able to scan end-to-end encrypted messages without undermining users’ privacy It's as though this is a secret that UK officials knew. Everyone technically minded knows this; it's a battle between those who regulate (the difficulty of whose job generally stops at "write down the rule and fine people if they don't do it") and those who know about current fundamental limitations.
  - seanhunter 3y ago
    Remember this is the same UK government that at one stage back in 2020 was proposing that there could be borderless customs enforcement between Ireland and Northern Ireland (part of the uk) "using drones". Because everyone knows drones do that. I don't think they consider themselves limited by things technology is actually capable of when talking about what they are going to get technology to do.
- uconnectlol 3y ago
  cloudflare also comes to mind, as they proudly do whatever they can in their power to check off that CSAM box as long as it makes them money somehow.
Sayrus 3y ago
And by next you mean in parallel. Spinning E2E encryption as a tool used only by bad actors is very common and pushing browsers to implement governments managed blacklists seems quite in progress. Banning VPNs from social networks (and more) was even suggested in France but quickly removed due to backlash.
pmontra 3y ago
Governments will mandate by law that the OS do client side scanning by peeping inside the app directories. Apple, Google, Microsoft will comply and realistically, which phones and computers are we going to use if we don't buy from them? Desktop Linux might become illegal.
- miohtama 3y ago
  Yes, the EU is already working to make this illegal https://mullvad.net/en/blog/2023/2/1/eu-chat-control-law-will-ban-open-source-operating-systems/ https://mullvad.net/en/blog/2023/2/1/eu-chat-control-law-wil... The best bet is Android fork based on Android open source distribution, without Google.
- SV_BubbleTime 3y ago
  Right. Per program makes people use different programs. Per OS, ok, maybe people will use a different OS (finally! Year of the Linux desktop!) … ok, we’ll just roll it into the three companies making GPUs. Driver binary side scanning. Maybe hardware frame buffer scanning? All is this is framed around children, but you are an absolute clown if you think that is truly the end goal. It’s always “for the children”.
OfSanguineFire 3y ago
Signal advises its users to install the app from the Play Store. While you can still get the .apk from Signal’s website, the developers warn against that. The EU can definitely exert pressure against what is hosted on the Play Store. Telegram famously lacks end-to-end encryption (unless you intentionally use its private-chat feature, which few people do) and shouldn’t be mentioned in the same context as Signal.
- kzrdude 3y ago
  If you enable private chat, you don't get notifications for new messages, that way the feature is self defeating, it seems to me at least. (Something could also have gone wrong?)
  - CuriousCosmic 3y ago
    Notifications generally go through Google Firebase so by not running private messages through firebase they avoid potential leaks. At least that's my guess
    - kzrdude 3y ago
      That sounds pretty bad, a cloud service needs to see all notifications?
      - miohtama 3y ago
        A good question. I am not sure if notifications on iOS, Android, Google Chrome or Firefox are end-to-end encrypted.
        nickphx 3y ago
        You can send whatever values you desire as long as your service worker handles it.
      - kube-system 3y ago
deleted 3y ago
[deleted]
jacksnipe 3y ago
At the moment, it is perfectly possible for e2ee to be compliant.
reisse 3y ago
> The EU bureaucrats do not have power to magically insert spyware in our devices as long as we can install our own software. EU bureaucrats have power to force ISPs to block connections to the messengers that would not comply, Chinese-style. They already do block websites they don't like - e. g. in Cyprus local provider CYTA shown me "access blocked due to EU comission regulation #whatever" for most of the Russian media sites.
JopV 3y ago
The EU is trying to make it mandatory to allow sideloading. This is bad, but also the opposite of what you are saying.
deleted 3y ago
[deleted]

polygamous_bat 3y ago

Any computational method that relies on a function that converts m bits (an image) to n bits (a fingerprint) where m > n will always be vulnerable to such an attack. The smaller n is compared to m, the easier it is to counterfeit something with that signature. It is not new knowledge. The only way to be certain, unfortunately, is for a human to look at it. When the allegation is as serious as CSAM, I would rather be very very certain before accusing someone, rather than taking a shotgun approach and just randomly searching phones. But what do I know.

mattashii 3y ago
> Any computational method that relies on a function that converts m bits (an image) to n bits (a fingerprint) where m > n will always be vulnerable to such an attack. The "vulnerable" depends on your definition: SHA-2 and SHA-3 are both still quite safe against preimage attacks, and even second preimage attacks require significant work to pull off for SHA-2, and I am unaware of any meaningful second preimage attack on SHA-3. Of course, SHA isn't built for finding similar images, but for finding exact matches this should be safe enough.
- lelag 3y ago
  But then bypassing the filter would just require to change a few bits here and there.
  - olliej 3y ago
    No the whole purpose of these “hashes” is that they’re robust to that. The attack model they’re designed for is image manipulation to avoid the hash match, not manipulating manipulating images to trigger the hash. There are numerous papers on doing bit manipulation to cause miss classification.
    - minitech 3y ago
      The context of the comment you’re replying to is a cryptographic hash.

bastawhiz 3y ago

The issue described here, to my understanding, is that you find or create csam and then manipulate it so that its fingerprint collides with another image that you want to be flagged as csam. You then submit the manipulated version of the found or generated image to the authority. First, at what point does the authority go "uh... Where did you get this from?" Practically speaking, the people doing this would have to be authorized law enforcement, no? Like if an ordinary citizen showed up with csam the excuse "oh I found this JPEG in a trunk in my late grandpa's attic" isn't gonna fly. My assumption is that the central authority that compiles the database isn't just taking fingerprints, they want the actual source images. And I'm assuming they actually look at them to ensure they are, in fact, csam. Otherwise anyone could put anything into the database. Now let me be very clear: I hate the idea of my device scanning for csam on principle. But what's the attack here? That someone looks at my device and determines that I do not, in fact, have csam? And that false positive might even be investigated and reveal that the bad actor sought out or generated CSAM for the purposes of perpetrating this! Which is even worse! If the police want to look at your device, they can do that already through any number of other sketchy means that don't involve what's likely weeks of effort. The US searches phones at the border when they're feeling like it. I don't want to defend this stuff, but the potential for abuse in this specific case seems vanishingly small.

pydry 3y ago
My understanding was that you generate a csam with a colliding fingerprint and put it online and trick somebody with an iPhone or whatever into clicking on it. Now they're on a watchlist and will get harassed by the police.
- bastawhiz 3y ago
  So you have to plant the manipulated csam somewhere that it'll be found by law enforcement and added to the database and just hope you did a good enough job to not be tracked?
  - deleted 3y ago
    [deleted]

heisenbit 3y ago

Anyone who thinks the injection of malicious hashes is an unrealistic scenario should take a look at the games now being played with Youtube‘s content protection system which is leveraged by scammers and blackmailers.

nopurpose 3y ago
what is happening in Youtube world?
- Macha 3y ago
  People claiming (via manual reports or via content ID) content that is not theirs to either obtain other people's ad revenue or cause the takedown of content they disagree with
- jabroni_salad 3y ago
  This one happened just a month ago so it might be what the parent comment is referring to. Basically, somebody made a fake company to make bogus copyright claims against someone to hurt their channel. Youtube refuses to deliver counterclaims unless the YTer puts their government name on it (he originally tried to deliver it via an attorney). Additionally, the other party is actively trying to compromise the YTer's other accounts and identity to damage him, so any new data point given to this person represents risk. I don't really care for this YTer (they made a video essay about someone who doesn't really want to be in the media circus anymore, could just MYOB) but the methods documented in this video can be used to doxx any user that uploads a video. All you gotta do is convince youtube that you own some piece of common non-royalty media that a youtuber uses to gain leverage on them. A lot of this kind of media is old with unfindable owners, so even if YT does want to spend the time to validate, there's nowhere to go with it. https://www.youtube.com/watch?v=hixwIOd_C44 https://www.youtube.com/watch?v=hixwIOd_C44
  - omscs99 3y ago
    What I don’t get is this: why does youtube still have such a monopoly after all this time? It’s had a shitty reputation since I can remember, why don’t creators just band together and take their viewership somewhere less hostile?

solusn 3y ago

[flagged]

npteljes 3y ago
Can you point to any society anywhere, where people learned to behave?
whimsicalism 3y ago
I am generally against the death penalty, but I am particularly against the death penalty for anything other than murder because of the actions it incentivizes.

jeffrallen 3y ago

Another case of setec astronomy.

pmontra 3y ago

The only thing that can save us is a huge number of false positives and a backslash on the governments in charge, when they'll have to concede for the 10,000th time that it was not CSAM but parents sending their baby pictures to the grandparents.

jopsen 3y ago
Looking at the cookie banner stupidity, I don't think a huge number of false positives would generate backlash.

qingcharles 3y ago

How do these databases differentiate between AI generated CSAM and CSAM of real victims? (Since many jurisdictions only criminalize real CP) I know that 99% of people cannot tell an AI image from a real photo since that "Last giant irish greyhound 1902" photo has been going around on social media for weeks, and it is, to me, unbelievably obvious AI.

chatmasta 3y ago
I assume the answer to that will be that there is no need to differentiate between them. And honestly, I agree with that argument. Possession of CSAM should be illegal regardless of whether it's "real" or not. But the proposed scanning system is the wrong solution, regardless of any "real or AI" ambiguity, because it's possible to generate false positives with nonsense images that aren't even close to the expected CSAM, real or otherwise.
- whimsicalism 3y ago
  I am of the opinion that regardless of whether they are both illegal, the penalties for “AI” should be significantly less.
  - anononaut 3y ago
    What's your reasoning?
- postalrat 3y ago
  Seems like you want to bring all the success of the war on drugs to the war on generated images.
  - chatmasta 3y ago
    No, I don't support any automated scanning system or really any sort of "going out of our way" to find new criminals. The reason I think it's a bad idea to differentiate between real or AI is similar to the arguments against "means testing" for distributing benefits. You don't want to put real victims in a situation where they're deprived of justice because they can't prove that their victimization was "real." Imagine a real CSAM criminal claiming a defense that they "thought it was AI generated." Do you want to give them that out? If protecting those victims comes at a cost of punishing criminals possessing AI-generated CSAM with sentences equally as harsh as those for "real" CSAM, then it's a worthwhile cost to pay. They are still criminals, and they are definitely not innocent (unless they're being framed, but that's a risk with both real and AI images).

cabirum 3y ago

I think it's pretty clear this is not about "CSAM", we have to stop using the term. It's just censorship, plain and simple. Client side means you'll pay from your own pocket for this wrongthing detector to work. It can even be automated so as soon as the detector gets triggered by anything, you'll get locked out of your bank accounts, until further notice I guess. If this thing gets a serious discussion in a parliament, get rid of the parliament.

ramses0 3y ago
But parliament wants to seed your camera with mugshots of the FBI's top-ten most wanted list so the instant a false positive appears (directly on the camera, potentially even prior to writing the image to disk, potentially even prior to pressing the snapshot button)... they beacon an alert (or exfiltrate piggybacking via Bluetooth/AirTag/Covid exposure tracking mrchanisms), and bob's you're uncle.
- SV_BubbleTime 3y ago
  I’m starting to think that a government large enough to get all the things it could want might be a bad idea.
  - timschmidt 3y ago
    I'd generalize this to any hierarchy. That includes corporations, institutions, religions, governments, cults, basically any time there are more than two people in the same room.
- morkalork 3y ago
  Sounds like there's going to be a market for vintage cell phones built before that happens. Although eventually they'll stop being useful as gWhatever rolls out and telecom providers stop supporting 4/5g.
INTPenis 3y ago
What makes the debate confusing is that some people probably believe it is about CSAM. Like the Swedish PM Ylva Johansson for example. She probably believes in what she's doing, and other forces are simply taking advantage of her crusade.

_lvbh 3y ago

Edit: comment deleted

Hizonner 3y ago

It says: > This shows that the database can be tainted with non-CSAM material by an entity that can submit entries to it. Actually, it can easily be tainted by anybody. Take your massaged hash-colliding image, which remember is still visually child porn, and post it on some pedos-R-us forum. The people who maintain the database actively troll those forums. They'll see the image and add the hash to the database for you.

regularjack 3y ago
You'd be publishing child porn, which I think is not the wisest thing to be doing.
- Hizonner 3y ago
  Tainting the database is probably already chargeable as something. The point is that the person doing this doesn't expect to get caught. And frankly they're probably right in that.
  - semiquaver 3y ago
    > Tainting the database is probably already chargeable as something This is definitely true. Good old “intentionally accesses a computer without authorization or exceeds authorized access” in the CFAA. It’s so vague as to be able to make almost _anything_ the government doesn’t like involving computers illegal.
- mcpackieh 3y ago
  A spy agency or secret police doing this to use the client-side scanning system as a surveillance/censorship system for political media would not care about that.
  - giantrobot 3y ago
    Even if such systems were only used for their stated purpose of finding CSAM, because no one can audit the database of hashes there's no guarantee that non-CSAM images aren't in there. Just being accused of having CSAM is a life ruining event. Even if someone is eventually cleared of charges their life is forever altered. The "We Got Him!!" headlines are front page news, retractions are filed in a basement filing cabinet with a sign that says "Beware of Leopard".

nopurpose 3y ago

If there were N algorithms, is it feasible to create B' image such that its fingerprint matches A in every algorithm?

huggingmouth 3y ago
If if isn't, then it becomes technically (and likely) feasible for every client that runs those algorithms to reconstruct A. This is a really stupid idea. Politicians who are demanding this are way out of their depths.

mikotodomo 3y ago

IFL these threads always give vibes like, "oh, thank god that anti CSAM measure wouldn't actually work".

bloopernova 3y ago
I think that some people are terrified that their (possibly AI generated) "loli pictures" would be caught by a scanner.
- giantrobot 3y ago
  What I'm concerned about is a system that flags me for a crime based on a database I can't audit based on mechanisms with an entirely too high false positive rate. Because the database can't be audited by anyone but a select group we have to trust that it only contains actual bad images. I do not trust that such databases don't also contain images that are embarrassing to powerful/connected people. I also do not trust such databases don't contain false positives. The sort of people that are super zealous about a topic aren't simultaneously super rational and objective about that topic. There's a non-zero probability that those databases contain lewd yet entirely legal images that the submitters just didn't like. Because of the false positive rate a photo of my dog might trigger an alarm and then my phone sends an automated message to the police. I'm then told by proponents there will be some manual review. I have to then hope that the local DA doesn't have an election coming up and wants to push a "tough on crime" message so charges me with a crime despite a review. In short these scanning systems require far too much unearned trust. They also present a slippery slope thanks to the incendiary nature of the topic. Today it's CSAM but what undesirable content will the systems be used for tomorrow? Such systems require trust in the stewards of today and tomorrow. Do you want people of the opposite ideology to you in charge of such systems? Do you trust they'll never be abused? Do you trust well meaning people never make mistakes? I do not trust in any of those things. I'm not worried about myself doing actual bad things, I'm worried that demonstrable false positive rates will ruin my life with the mere accusation of doing something bad.
  - astrange 3y ago

varispeed 3y ago

Ok, but why not go for routine home searches then? Is it because on the phone, it is "invisible" and cheap? Does it mean that what stops the government from treating everyone as a potential criminal is cost and inconvenience? People were raising alarms about this years ago and were branded as conspiracy theorists. I guess the EU works well in their propaganda department painting themselves as do gooders, where in fact they are just wannabe Stalinists.

SV_BubbleTime 3y ago
>Ok, but why not go for routine home searches then? Is it because on the phone, it is "invisible" and cheap? Underrated comparison. That is apt and the whole “for the children” part is smokescreen.

kypro 3y ago

This is basically what I suggested we do a few years ago if Apple added client-side content scanners. Seems all need to here is obtain a finger print of a CSAM image then you can reverse engineer a non CSAM image to match that finger print. Distribute this image wide enough and you effectively render these algorithms useless. Which is a shame because they could be used for good, but it seems kinda obvious this will expand out to scanning for terrorist materials. And again, while I might be in favour of this in theory I am frequently shocked by what fits this definition. In the EU if you promote the wrong ideas or say something "hateful" you can be arrested and sentenced 10+ years in prison. I think what concerns me is that currently you really only put yourself at risk if you make your wrong think public, e.g tweet about your dislike of Islam on Twitter. These client-side scanners would now enable the authorities to not only go after those who say "racist" things online, but also those who view / save "racist" memes. If I were to guess at a motive, this would be it. Legal note: I obviously don't agree with hate of any kind, but I do believe in the right to express views which may be considered hateful. Edit: Fine with the downvotes – but curious what people are disagreeing with? Would you mind leaving a quick comment if you choose to downvote?

Hizonner 3y ago
1. You mixed up generating child porn images that collide with targeted non-child-porn images, in order to taint the database and thereby suppress the target images, with generating non-child-porn images that collide with child porn images, in order to overwhelm the system with false positives. 2. You brought in matters that have nothing to do with hashing or child porn, but tend to generate flame wars. 3. You put sneer quotes around "racist", which makes it sound very much like you don't think any such thing exists.
- kypro 3y ago
  Thank you. 1. Oh okay. I wasn't very clear there. By, "this is basically what I suggested we do a few years ago" I meant that we should look to exploit the limitations of these fingering printing algorithms to render them impractical. But yeah, my suggestion was that we should do the reverse. Rather than taint the database, taint the content. 2. That wasn't my intention. I was suggesting that there are likely other reasons governments are interested in this technology than, "think of the children". Hence, why I think we should seek to ways to render these algorithms ineffective were they deployed. But yes, please feel free to downvote if you feel my comment is unproductive. On reflection I agree it might be. I'm not really adding much here 3. You know what's annoying? I did that because I didn't to make an assertion either way to be as neutral as possible. I'm using quotes because others deem it racist, and that it's irrelevant what my personal position on that is. If you want my position on whether racism exists though, obviously I believe it does – I'd urge you to just assume good faith in the future.

ls612 3y ago

Didn’t the UK just blink when chat apps said hell no we will leave instead? Does the EU expect to get better results? Apple has very publicly come out and said that their proposal like this was a bad idea and I’d expect Signal to also not comply, the only remaining question is whether WhatsApp would threaten to leave again.

SV_BubbleTime 3y ago
Apple proposed the exact same thing themselves though.

nickdothutton 3y ago

Government almost never passes useful/wise legislation with regard to technology, because it thinks “tech” is a problem that the government can “fix”. There are systemic reasons for the legislation almost always being bad/unhelpful/foolish and those reasons are not going to change.

grammers 3y ago

Worst of all the EU Commission is now starting to act like fake news promoters by posting misleading ads (German article): https://netzpolitik.org/2023/politisches-mikrotargeting-eu-kommission-schaltet-irrefuehrende-werbung-fuer-chatkontrolle-auf-x/ https://netzpolitik.org/2023/politisches-mikrotargeting-eu-k... It's outraging.

michaelmrose 3y ago

This attack also works for people who are not in a position to submit entries to the database as you could very well plant the generated matching images on a server set up with a compromised credit card and report it yourself. You could select pictures from targets publically available photos or more insidiously look to compromised accounts cloud storage and generate fake offensive images that match peoples actual baby pictures whether for harm or blackmail.

deleted 3y ago

[deleted]

anigbrowl 3y ago

[...] One could conclude that the abuse centre could thus easily spot the malicious entry in its database after a few of these reports all concerned with the same image, and delete the corresponding fingerprint from the database. If that were the case, this avenue of attack would not be a problem in practice. This assumes, however, that the abuse centre keeps track of such false positives over time to detect such maliciously uploaded fingerprints. This may or may not be the case. Why would it not be the case, unless the abuse center wanted to waste time and resources and have unreliable records? This essay is technically sophisticated and socially naive. Tech people keep spinning up the most unlikely and hard-to-explain reasons for why leveraging technology to interfere with the spread of CSAM is bad, instead of working toward any kind of privacy-respecting solution to target CSAM itself. Constantly emphasizing the scope for government intrusion and privacy violation (legit but at the same time overblown to the point of paranoia) and constantly minimizing or dismissing the real harms of CSAM is a great way to alienate the normal non-technical people you need to have on side.

uconnectlol 3y ago
okay, ill bite: yes, the blogger has tunnel vision, that's the only valid point in your post, he also missed the fact that anyone can poison the well, not just privileged entities. however your rhetoric is cancerous, although i'm aware that many people who use it are just sheep and have no idea what they're insinuating. CSAM is a propaganda word: 1. right wing uses it to push their agenda of punishing people for having sex 2. left wing uses it to push their agenda of punishing males or kulaks or whatever 3. police use it to make it look like they're solving a crime so they can get paid for nothing: they can arrest someone for looking at a nude picture of a 17 year old (which the suspect is often unaware of), then in the media and police statements: "he was arrested for the possession of child sexual abuse material". and this is the majority of convictions: people (and even teens) being into teens. people are not naturally sexually attracted to children (13 and under) in more than say 0.0001%. so now that we actually looked at the subject beyond the taboo veil, it seems the discussion is just to stop 14-17 year olds from having sex, which is insane and does not justify one single thing that police do about it. CP being illegal is just completely out of the question. we should be debating re legalizing CP, or changing it to 13 and under at the very least (i don't see any harm in the existence of porn of any age since it does not encourage abuse since most people will not become sexually attracted to children no matter how much you expose them to it). so again, your rhetoric is pure cancer. i don't need a "privacy respecting solution to CSAM". what the hell do you think that would be? do we also need a freedom respecting solution to the murder problem? would you have my legs cut off and say i should just order door dash?

Retr0id 3y ago

The article considers "an entity that is allowed to propose new entries to the CSAM database". You don't even need this! You could target a whole "social cluster" of people without having any special privileges within this system. As an example, lets say you want to attack environmental protesters. For image A, you create a meme about climate change. For image B, you procure something that looks, to humans, like CSAM (as described in the article). Craft B′ such that f(B′) = f(A) (also as described). Now, all you have to do is anonymously publish B′ to a platform that is actively moderated/monitored. Unfortunate users will see it and report it as CSAM, and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database. Now you can start sending out image A, the meme, to your target demographic. You won't be able to post it on "mainstream" platforms with server-side fingerprint scanning, but there are plenty of other avenues for it to spread. If it's a good meme, it will propagate organically through group-chats and DMs, and eventually find its way onto devices with client-side scanning. Apple's proposed device scanning system had a threshold before your device would be flagged, so repeat all these steps a whole bunch of times, until the average meme-savvy environmental protester's device gets flagged for further scrutiny. Unwitting victims who do try to post the meme to a mainstream platform with fingerprint matching may risk getting their accounts flagged and taken down, and they might have no way of knowing what triggered it. This would lead to, of course, automated censorship of environmental protest groups.

hju22_-3 3y ago
That's some next level shit. I dig it.
FireBeyond 3y ago
> Apple's proposed device scanning system had a threshold before your device would be flagged Which always struck me as odd as it would extremely easy to spin this as “Apple detected CSAM on this device but their policy is only to alert authorities once a set quantity of CSAM is found…”
- LorenPechtel 3y ago

SV_BubbleTime 3y ago

I’m curious how few people have followed this topic of client side scanning through. The logical conclusion is that someone is going to get a knock on their door and a warrant for search because of an unknown signature match on a file some where at some point? And people seriously think this will stop at being “for the children”?

orbital-decay 3y ago

What are the legal consequences of having an image with the matching fingerprint on your device? Is there some kind of threshold to mitigate false positives? What organization is responsible for processing the matches? Is it enough for a search warrant? (for various countries/jurisdictions)

quitit 3y ago

I am against the idea of scanning for the reason that the author pointed out: It's trivial to repurpose the technology to use it in dystopian ways. I however have precisely zero concerns about impersonating hashes: 1. It's trivial to deal with tainting the database: both secondary hashing and more invasive hashes deal with that problem. 2. It's trivial to deal with impersonated hashes, all positives can be scanned on device in a second round with a different hash method. Those which are still positive can have be checked off device in an automated, yet privacy preserving, way such as by sending a low resolution crop with faces blurred to a system that does a likeness check against the matched CSA image: only the real image will bear likeness. 3. These systems have undisclosed minimums. A person sharing CSAM will be generating positive results at a significant rate. Receiving a few images with faked hashes won't set off alarms, and if a victim is suddenly receiving hundreds of images, that would already be obvious. 4. These articles over sensationalise what occurs when a positive match is found. They vaguely gesture to serious consequences from a single match. In reality even without secondary automated checks, the false positive will be noticed by a human and then discarded. If a person is receiving a lot of false positives that may lead to the individual being informed that they are begin targeted. Overall the idea of scanning chat content seems ineffective, for much of the same reasons why it's ineffective to ban end-to-end encrypted chat: those with criminal intent will just use something else or roll their own.

Hizonner 3y ago
> It's trivial to deal with tainting the database: both secondary hashing and more invasive hashes deal with that problem. Sorry, what does "more invasive hashes" mean? > It's trivial to deal with impersonated hashes, all positives can be scanned on device in a second round with a different hash method. The double hashing would definitely help. I doubt anybody's gotten a close collision for more than one hash at a time. It's not inconceivable that they could do it, though. As far as I know, there are only two hash methods that get used for this, and I think both of them are based on sliding a window over a reduced version of the image and doing a DCT, so they're pretty similar. I wouldn't be surprised if somebody could fool both of them enough to at least force you to tighten up your similarity threhsholds. It would be safer to come up with something that worked on completely different principles. No idea how hard that would be. I don't think that the second-stage backup, where you send the image off somewhere, is "privacy preserving" by any standard I'd be comfortable with. Well, OK, actually even completely foolproof client side scanning isn't privacy preserving enough for me, but I mean that's substantially worse. I also don't think it'd be easy to find anybody to run that server. And it'd be a significant structural change from what's deployed now. It'd be much more disruptive than adding a second hash (which is isomorphic to just switching to a single composite hash). It took a long time to get the present system deployed, so that big a change doesn't seem "trivial". > These systems have undisclosed minimums. A person sharing CSAM will be generating positive results at a significant rate. Receiving a few images with faked hashes won't set off alarms, and if a victim is suddenly receiving hundreds of images, that would already be obvious. Minimum what? If you just say "undisclosed minimums", my mind immediately assumes you mean threshold similarity scores. But you seem to mean hit counts to trigger various actions. Anyway, both of those are set by the service that's using the database. Those services vary in their strictness and clue level, so I'm not sure you can say anything very general about them. > These articles over sensationalise what occurs when a positive match is found. In the threat model of the original article, the people doing the review for false positives are most likely the same people who poisoned the database to begin with. They've corrupted it to give them hits on images they want to suppress, even though the system operators don't want to support them in suppressing those images. So that whole "reviewed by a human and discarded" step doesn't happen. In the broader world, I suspect that the thresholds for "delete the file" tend to be very low... too low for there to be human review. The thresholds for "disable the user's account" probably aren't exactly stratospheric, either. You can have damaging consequences well short of midnight police raids. Also, "framing" people isn't necessarily the only thing you might use it for. For example, you could try to use it to drive the cost of running the system up to unsupportable levels.

sterlind 3y ago

This is precisely the kind of trolling 4chan loves to do. It ticks all the boxes: swatting, CP, hacktivism possibly if you squint really hard. And since these hashes are presumably resistant to cropping and scaling, they might be able to target individual symbols, people or image macros, like the rainbow pride flag (you know /pol/ wouldn't be able to resist), politicians etc.

thephyber 3y ago

I’m worried the same kind of attack will make this[1] CSAM removal project useless. If there is no authentication of the submitter and no punishment for fake submissions, then it becomes garbage-in, garbage-out. [1] https://takeitdown.ncmec.org/ https://takeitdown.ncmec.org/

Zuiii 3y ago

I'm completely stumped. I just don't understand how politicians can still believe that client-side scanning would somehow stop CSAM. Couldn't those who possess this material also run those exact same CLIENT scanners (if not directly, than via isolated canary systems and emissions monitoring) to detect when files have been added to the scanners? What about tools that keep shuffling files in ways that are not too perceivable but would defeat those scanners? Just look at copyright infringing material on Youtube (it's still there, just better hidden). Nothing short of a government whitelist database containing all files that are legal to possess, or 100% analog human surveillance will solve CASM. Not AI and certainly not this. I'm forced to agree with others here: This isn't about CSAM at all. It's a thinly disguised frontal attack on democracy. Imagine if the government had this technology when people were organizing to repeal discriminatory laws in the US. Imagine if authoritative governments in the east (or anywhere else) had this technology right now. Take a long hard look at every person or organization who supports this nonsense and pay close attention to them. They are the ones who want to harm us and our children.

MagicMoonlight 3y ago

The database would obviously come from a central organisation like the FBI or NCA and be built from material gathered in investigations. They wouldn’t allow larry from reddit to share his child porn with them and break the database. Anyone with sufficient power to want to target you wouldn’t use a ridiculous exploit like this. They’ll just go to google and say “change the update server for this user to this new address” and then breach your device with the next software update.