4 ms·

As other has pointed out, this is extremely creepy. The easiest solution would have been this one: - The user uses a JS solution to hash the images on the clie

by LukaAl 9y ago

As other has pointed out, this is extremely creepy. The easiest solution would have been this one: - The user uses a JS solution to hash the images on the client, without the image being uploaded - She compiles a form with additional information (e.g: capture her account, reasons for uploading, suspect person sharing the picture) - The picture is saved in the DB as un-verified revenge porn. - The first time someone uploads a picture that matches the hash, the pic is quarantined and the specially trained individual manually check them - A scoring system could be used to check the reliability of the submission. If multiple photos marked revenge porn get rejected, the control becomes ex-post. For even more violations, the user get banned from using the tool and should directly contact Facebook. Submitting the same hash that has been rejected, will count as a "red mark" Now, I understand this system is very complex, what Facebook has done is an MVP and as a product manager, this is what I prefer. But considering the issue (revenge porn, not something I necessarily want to test the impact on retention :-) ). Also, yes, it requires resources, but Facebook has a problem with trust lately, better to do the best... [edited for formatting]

evgen 9y ago
Now figure out a way to do step #1 (user does a hash client-side) without making it trivial for someone else to create a filter that adds enough noise to invalidate step #4 (uploaded pics that match a hash are quarantined.)
- wjh_ 9y ago
  PhotoDNA would be an option, I imagine. Or at least something similar! https://en.wikipedia.org/wiki/PhotoDNA https://en.wikipedia.org/wiki/PhotoDNA
  - moyix 9y ago
    Honestly, I doubt that most of these sorts of algorithms would survive concerted attacks – that's why they tend to be closely guarded. Alex Stamos (Facebook's CISO) implies this is why they can't do it client-side: https://twitter.com/alexstamos/status/928646228472078336 https://twitter.com/alexstamos/status/928646228472078336
    - LukaAl 9y ago
      From when security through obscurity is a good idea?
      - _red 9y ago
        You should go put a $100 bill on the dashboard of your car along with your laptop on the front seat. Report back the results. snark-mode off: We use obscurity every day and its a completely valid layer of security.
        LukaAl 9y ago
        > We use obscurity every day and its a completely valid layer of security. Not sure I agree with that, most of the time when we do that is because we don't want to spend the time to have better security. And then we get burned. To your example of the 100$ bill: at my parents home with the car parked in the garage? No problem to do that at all. Out on the street in SF? No. I don't trust my glass enough as a security measure. But I don't leave money at all, is not security through obscurity. But we are going OT. The problem that is raised is that they need necessarily security through obscurity. And we have two problems: - How really robust are these algorithms? How long before we will see people abusing them? - Have you thought hard enough about how this system could work? E.g: have a partial hashing made client-side and the final one on the server? Or a situation where the server code is open-sourced without the model to calculate the hash? That would allow for external review without disclosing the hash. Yes, you still need warranties that Facebook is using that code, but you could have a trusted third party certifying the program. My point is, the person who designed this program didn't really understand the problem. The problem is not revenge porn. The problem is Facebook reputation. And this solution is totally deaf.
        stevenwoo 9y ago
        Facebook is pretty low on companies I would trust, on the other hand we know that NSA employees who have the highest level of security clearance were using their ability to intercept everything to stalk their exes and pass around their exes nude photos according to Snowden. Clearly nobody can be trusted, and as you write, a solution for this must incorporate this fact.
- LukaAl 9y ago
  Agree, that's a problem but there are options to solve it. Look at PhotoDNA by Microsoft [0]. But it is a second step. First, you need the reporting properly done. [0] https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna
- jsjohnst 9y ago
  This is already a solved problem at FB (PhotoDNA does this for them for CP images).
- tree_of_item 9y ago
  How does giving Facebook the image solve that problem, in a way that can't be done client side?
- dannyw 9y ago
  Why would you need to? We are talking about vengeful ex-es here. It’s possible to bypass this trivially by uploading it to another image sharing service, so the goal should be defection in the general case; not try and protect against the 0.1% that even know what a perceptual hash is.
- deleted 9y ago
  [deleted]
- agumonkey 9y ago
  Good point, that said, with the recent visual ML trend, I'm sure fb or the likes can engineer a solution for that.
the8472 9y ago
> - The user uses a JS solution to hash the images on the client, without the image being uploaded You have to trust facebook in either case, each time you do it. Either to handle your nude pics properly or to serve you javascript that does what they claim it does, every single time. On the other hand an open source desktop application only needs to be audited once and then can be validated based on a hash. In browser crypto is not a solution if you want to minimize the needed trust.
- oh_sigh 9y ago
  Nobody wants to run a desktop application given to them from facebook
  - the8472 9y ago
    > only needs to be audited once and then can be validated based on a hash. I thought I covered that concern, but I neglected to mention that it should be open source so everyone can audit it.
  - statictype 9y ago
    It’s better than sending them NSFW photos for human review
  - hnzix 9y ago
    But it could force autoinstall from the website and autoscan your drives then preupload your nudie pics and tag them with your name. How convenient! If all your friends are using it then it must be safe right?
- zodiac 9y ago
  You could examine network traffic to confirm that not enough data is sent to reconstitute the picture
  - icebraining 9y ago
    You'd have to check every single time, since the code can change at any time.
sadjad 9y ago
You lost me at JS.

by LukaAl 9y ago

evgen 9y ago

Now figure out a way to do step #1 (user does a hash client-side) without making it trivial for someone else to create a filter that adds enough noise to invalidate step #4 (uploaded pics that match a hash are quarantined.)

wjh_ 9y ago
PhotoDNA would be an option, I imagine. Or at least something similar! https://en.wikipedia.org/wiki/PhotoDNA https://en.wikipedia.org/wiki/PhotoDNA
- moyix 9y ago
  Honestly, I doubt that most of these sorts of algorithms would survive concerted attacks – that's why they tend to be closely guarded. Alex Stamos (Facebook's CISO) implies this is why they can't do it client-side: https://twitter.com/alexstamos/status/928646228472078336 https://twitter.com/alexstamos/status/928646228472078336
  - LukaAl 9y ago
    From when security through obscurity is a good idea?
    - _red 9y ago
      You should go put a $100 bill on the dashboard of your car along with your laptop on the front seat. Report back the results. snark-mode off: We use obscurity every day and its a completely valid layer of security.
      - LukaAl 9y ago
        > We use obscurity every day and its a completely valid layer of security. Not sure I agree with that, most of the time when we do that is because we don't want to spend the time to have better security. And then we get burned. To your example of the 100$ bill: at my parents home with the car parked in the garage? No problem to do that at all. Out on the street in SF? No. I don't trust my glass enough as a security measure. But I don't leave money at all, is not security through obscurity. But we are going OT. The problem that is raised is that they need necessarily security through obscurity. And we have two problems: - How really robust are these algorithms? How long before we will see people abusing them? - Have you thought hard enough about how this system could work? E.g: have a partial hashing made client-side and the final one on the server? Or a situation where the server code is open-sourced without the model to calculate the hash? That would allow for external review without disclosing the hash. Yes, you still need warranties that Facebook is using that code, but you could have a trusted third party certifying the program. My point is, the person who designed this program didn't really understand the problem. The problem is not revenge porn. The problem is Facebook reputation. And this solution is totally deaf.
        stevenwoo 9y ago
        Facebook is pretty low on companies I would trust, on the other hand we know that NSA employees who have the highest level of security clearance were using their ability to intercept everything to stalk their exes and pass around their exes nude photos according to Snowden. Clearly nobody can be trusted, and as you write, a solution for this must incorporate this fact.
LukaAl 9y ago
Agree, that's a problem but there are options to solve it. Look at PhotoDNA by Microsoft [0]. But it is a second step. First, you need the reporting properly done. [0] https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna
jsjohnst 9y ago
This is already a solved problem at FB (PhotoDNA does this for them for CP images).
tree_of_item 9y ago
How does giving Facebook the image solve that problem, in a way that can't be done client side?
dannyw 9y ago
Why would you need to? We are talking about vengeful ex-es here. It’s possible to bypass this trivially by uploading it to another image sharing service, so the goal should be defection in the general case; not try and protect against the 0.1% that even know what a perceptual hash is.
deleted 9y ago
[deleted]
agumonkey 9y ago
Good point, that said, with the recent visual ML trend, I'm sure fb or the likes can engineer a solution for that.

the8472 9y ago

> - The user uses a JS solution to hash the images on the client, without the image being uploaded You have to trust facebook in either case, each time you do it. Either to handle your nude pics properly or to serve you javascript that does what they claim it does, every single time. On the other hand an open source desktop application only needs to be audited once and then can be validated based on a hash. In browser crypto is not a solution if you want to minimize the needed trust.

oh_sigh 9y ago
Nobody wants to run a desktop application given to them from facebook
- the8472 9y ago
  > only needs to be audited once and then can be validated based on a hash. I thought I covered that concern, but I neglected to mention that it should be open source so everyone can audit it.
- statictype 9y ago
  It’s better than sending them NSFW photos for human review
- hnzix 9y ago
  But it could force autoinstall from the website and autoscan your drives then preupload your nudie pics and tag them with your name. How convenient! If all your friends are using it then it must be safe right?
zodiac 9y ago
You could examine network traffic to confirm that not enough data is sent to reconstitute the picture
- icebraining 9y ago
  You'd have to check every single time, since the code can change at any time.

sadjad 9y ago

You lost me at JS.