4 ms·

For transferring, I mean specifically that it is known that attacks transfer in a general sense e.g., https://www.usenix.org/system/files/sec19-demontis.pdf htt

by mirker 5y ago

For transferring, I mean specifically that it is known that attacks transfer in a general sense e.g., https://www.usenix.org/system/files/sec19-demontis.pdf https://www.usenix.org/system/files/sec19-demontis.pdf It’s not easy in this setting due to the information asymmetry, but if someone wants to dedicate the resources it should be possible. They could get N random models and create images that fool all N. For crowdsourcing, I mean that the crowd is a distribution network and not the attack creator (e.g., they are a self-distributing virus/“worm”). The attacker takes those images (the initial “worm” programs) and uploads them to reddit as a catchy meme for worldwide distribution. Meme sharers wouldn’t be aware that the meme has a hash collision with CP. The entire defense of this sort of thing is obscurity (further obscured by using “magical” machine learning), since nothing is proven about how collision resistant the algorithm is. At Apple’s scale, it’s as careless as rolling your own crypto hash.

scoopertrooper 5y ago
There is no feedback loop to support the development of the attack. Apple in this case would just be filtering and ignoring false positives without notifying the attacker if they were successful or not. I still don't quite see how the 'worm' would work in practice. It's not just a matter of sharing a link to an image, but you have to add it to your photo album. Maybe I'm getting old, but I don't have a single meme-type image in my album. I don't think this can be classed as a security by obscurity problem. Security by obscurity fails because they 'key' (the obscure bit) can't be easily changed if it leaks. But given Apple has both the target and candidate images available to it, it can in effect generate a new key at the expense of having to do additional computation.
- mirker 5y ago
  The way I think about it is that the attacker has plenty of time (years) to either guess or discover the viability of an attack. For example, inspecting packets and reverse engineering, correlating photos with police visits, creating a hash database using black market data, etc. will all be used to determine if an attack is working. Same thing goes with the worm. Posting infected memes is one way and that’ll passively get some downloads. Another way is to text the pictures via bots to people and hope their SMS is iCloud backed up. The point is if someone figures out how to make an attack, they (or entity they sell it to) will almost surely spend great effort in social engineering the distribution. I agree that it’s probably not the right term. My main concern though is that the class of attacks that is effective against one model will likely be effective against the entire class of similar models. So Apple’s main defense is the limited feedback loop caused by keeping part of the modeling private rather than a traditional computational complexity defense, which would allow disclosure of the entire implementation. It’s taken one week for the proof of concept to be demonstrated and most of that was likely boilerplate API code; it would take perhaps even less time to retarget if Apple disclosed the entire implementation with a different model. The demonstrated attack was textbook material, so it’s not unreasonable to believe that all attacks are textbook material.

by mirker 5y ago

scoopertrooper 5y ago

There is no feedback loop to support the development of the attack. Apple in this case would just be filtering and ignoring false positives without notifying the attacker if they were successful or not. I still don't quite see how the 'worm' would work in practice. It's not just a matter of sharing a link to an image, but you have to add it to your photo album. Maybe I'm getting old, but I don't have a single meme-type image in my album. I don't think this can be classed as a security by obscurity problem. Security by obscurity fails because they 'key' (the obscure bit) can't be easily changed if it leaks. But given Apple has both the target and candidate images available to it, it can in effect generate a new key at the expense of having to do additional computation.

mirker 5y ago
The way I think about it is that the attacker has plenty of time (years) to either guess or discover the viability of an attack. For example, inspecting packets and reverse engineering, correlating photos with police visits, creating a hash database using black market data, etc. will all be used to determine if an attack is working. Same thing goes with the worm. Posting infected memes is one way and that’ll passively get some downloads. Another way is to text the pictures via bots to people and hope their SMS is iCloud backed up. The point is if someone figures out how to make an attack, they (or entity they sell it to) will almost surely spend great effort in social engineering the distribution. I agree that it’s probably not the right term. My main concern though is that the class of attacks that is effective against one model will likely be effective against the entire class of similar models. So Apple’s main defense is the limited feedback loop caused by keeping part of the modeling private rather than a traditional computational complexity defense, which would allow disclosure of the entire implementation. It’s taken one week for the proof of concept to be demonstrated and most of that was likely boilerplate API code; it would take perhaps even less time to retarget if Apple disclosed the entire implementation with a different model. The demonstrated attack was textbook material, so it’s not unreasonable to believe that all attacks are textbook material.