4 ms·

Picking up the fight against deepfakes, voice cloning and generative AI

23 points by matyask 2y ago

deleted 2y ago
[deleted]
metadat 2y ago
Does the watermarking survive aggressive re-encodes? According to this page, it. seems unlikely, because if that's been solved then you would market it. Also, the source images are quite low resolution, what's up with that? What is actually new or novel here?
janalsncm 2y ago
I appreciate the spirit of this, but this is likely to be defeated by anyone with some basic technical understanding of how training works. For example a denoising autoencoder will probably be sufficient. See the Glaze tool which was also cracked https://arstechnica.com/tech-policy/2024/07/glaze-a-tool-protecting-artists-from-ai-bypassed-by-attack-as-demand-spikes/ https://arstechnica.com/tech-policy/2024/07/glaze-a-tool-pro... That said, it’s more like a booby trap that might catch some people off guard who didn’t think to look for it.
- matyask 2y ago
  Hi, one of the founders here, "robustness" is our next step. We are currently training a few models based on cross attention and invariant domain learning that promise to be quite resistant to noising and denoising. The purpose of fingerprint watermarking as we have released it is to act like a signature for validity, some companies already do this but none of them have public products that you can just use. The nest encoder and decoder will use a unique signature for every user. The limitation of Glaze and models like it is that they target specific models/approaches. That being said we are also running experiments on adversarial watermarking that aim to deliver the same image quality as glaze with less computation and more resistance. We want to secure digital identity and copyrights. Watermarking is our initial approach.
  - mrbungie 2y ago
    I'm still trying to understand how a watermarking method may resist simple methods as (1) upscaling and downscaling plus noise, or (2) taking a photo of the image plus compression plus noise, (3) any simple convolutional filter or a combination of those.
    - matyask 2y ago
      1) With an invariant domain representation, 2) and 3) I am not certain yet, but I will find out. Although when it comes to signing with a watermark for validity you would not want the id (50 bits) to be preserved or copied. These are separate use cases / applications.
  - janalsncm 2y ago
    Glaze techniques might be specific to a model but the techniques to defeat it are not. More specifically, I don’t see any reason to believe that I cannot learn a function from A to B where A is the distribution of watermarked files and B is the distribution of non-watermarked files. It’s a trivial exploit.
trod123 2y ago
Interesting company, and something that is surely good to watch out for in the future. It is also very needed use case now that generative AI is just about everywhere. There is some cause to be skeptical of the claims on the website, because the site doesn't provide enough information to validate the claims with any rational basis, and there are a number of other companies that have done the same and failed to deliver. There have also been a few court cases now where similar use cases for attribution have been used inappropriately, and resulted in false claims/charges causing great miscarriages of justice (under facial recognition). Hopefully you performed an adequate adversarial review, and have a demonstrable way to rationally prove there are no false positives at a bare minimum.
- jrm4 2y ago
  I'll bet a dollar that none of this ends up being particularly viable. Feels like an "anti-Photoshop" company; whatever means you might like or want to exist re defending this sort of thing will lose in an arms race. This is not to say that there's nothing that can be done, but I just doubt those solutions will ever be technical.
  - matyask 2y ago
    We for sure don't aim to be anti-AI. It's difficult to see how a non technical solution will be enforceable/effective. Watermarking is our initial approach to securing digital identity and copyrights.
- matyask 2y ago
  Hi there, we have published a demonstration of fingerprint watermarking on audio and images that anyone could just try. In our next phase we will release our first consumer product that will provide value. We have conducted thorough testing on our fingerprint encoding and decoding. We will for sure aim to explain this better when we give users the ability to encode unique signatures. If there are any other particular claim you have not found enough information about then I am happy to elaborate. Could you by any chance link any of those court cases?
gmerc 2y ago
More snakeoil. Great. It’s a fundamental misalignment with the threat model - misinformation and deepfakes being scaled by nation state actors and major political parties to its infinite resources on one since and the reliance on a fully secured providence chain on the other that makes it pure snakeoil.
- matyask 2y ago
  I think you are making assumptions on provenance. The resource disparity you point out is obvious, but your phrasing somewhat binary. We have released a demo not a consumer product yet. I would be happy to hear your more expanded thoughts, especially on the threat model you have in your mind.
masly 2y ago
This reminds me of the struggle mapmakers used to have where other map makers would copy their maps. So to counter it and be able to prove the copying, they would add tiny unnoticeable but fake places. https://vinepair.com/cocktail-chatter/cartographers-fake-places-maps/ https://vinepair.com/cocktail-chatter/cartographers-fake-pla...

23 points by matyask 2y ago

deleted 2y ago

[deleted]

Does the watermarking survive aggressive re-encodes? According to this page, it. seems unlikely, because if that's been solved then you would market it. Also, the source images are quite low resolution, what's up with that? What is actually new or novel here?

janalsncm 2y ago

I appreciate the spirit of this, but this is likely to be defeated by anyone with some basic technical understanding of how training works. For example a denoising autoencoder will probably be sufficient. See the Glaze tool which was also cracked https://arstechnica.com/tech-policy/2024/07/glaze-a-tool-protecting-artists-from-ai-bypassed-by-attack-as-demand-spikes/ https://arstechnica.com/tech-policy/2024/07/glaze-a-tool-pro... That said, it’s more like a booby trap that might catch some people off guard who didn’t think to look for it.

matyask 2y ago
Hi, one of the founders here, "robustness" is our next step. We are currently training a few models based on cross attention and invariant domain learning that promise to be quite resistant to noising and denoising. The purpose of fingerprint watermarking as we have released it is to act like a signature for validity, some companies already do this but none of them have public products that you can just use. The nest encoder and decoder will use a unique signature for every user. The limitation of Glaze and models like it is that they target specific models/approaches. That being said we are also running experiments on adversarial watermarking that aim to deliver the same image quality as glaze with less computation and more resistance. We want to secure digital identity and copyrights. Watermarking is our initial approach.
- mrbungie 2y ago
  I'm still trying to understand how a watermarking method may resist simple methods as (1) upscaling and downscaling plus noise, or (2) taking a photo of the image plus compression plus noise, (3) any simple convolutional filter or a combination of those.
  - matyask 2y ago
    1) With an invariant domain representation, 2) and 3) I am not certain yet, but I will find out. Although when it comes to signing with a watermark for validity you would not want the id (50 bits) to be preserved or copied. These are separate use cases / applications.
- janalsncm 2y ago
  Glaze techniques might be specific to a model but the techniques to defeat it are not. More specifically, I don’t see any reason to believe that I cannot learn a function from A to B where A is the distribution of watermarked files and B is the distribution of non-watermarked files. It’s a trivial exploit.

trod123 2y ago

Interesting company, and something that is surely good to watch out for in the future. It is also very needed use case now that generative AI is just about everywhere. There is some cause to be skeptical of the claims on the website, because the site doesn't provide enough information to validate the claims with any rational basis, and there are a number of other companies that have done the same and failed to deliver. There have also been a few court cases now where similar use cases for attribution have been used inappropriately, and resulted in false claims/charges causing great miscarriages of justice (under facial recognition). Hopefully you performed an adequate adversarial review, and have a demonstrable way to rationally prove there are no false positives at a bare minimum.

jrm4 2y ago
I'll bet a dollar that none of this ends up being particularly viable. Feels like an "anti-Photoshop" company; whatever means you might like or want to exist re defending this sort of thing will lose in an arms race. This is not to say that there's nothing that can be done, but I just doubt those solutions will ever be technical.
- matyask 2y ago
  We for sure don't aim to be anti-AI. It's difficult to see how a non technical solution will be enforceable/effective. Watermarking is our initial approach to securing digital identity and copyrights.
matyask 2y ago
Hi there, we have published a demonstration of fingerprint watermarking on audio and images that anyone could just try. In our next phase we will release our first consumer product that will provide value. We have conducted thorough testing on our fingerprint encoding and decoding. We will for sure aim to explain this better when we give users the ability to encode unique signatures. If there are any other particular claim you have not found enough information about then I am happy to elaborate. Could you by any chance link any of those court cases?

gmerc 2y ago

More snakeoil. Great. It’s a fundamental misalignment with the threat model - misinformation and deepfakes being scaled by nation state actors and major political parties to its infinite resources on one since and the reliance on a fully secured providence chain on the other that makes it pure snakeoil.

matyask 2y ago
I think you are making assumptions on provenance. The resource disparity you point out is obvious, but your phrasing somewhat binary. We have released a demo not a consumer product yet. I would be happy to hear your more expanded thoughts, especially on the threat model you have in your mind.

masly 2y ago

This reminds me of the struggle mapmakers used to have where other map makers would copy their maps. So to counter it and be able to prove the copying, they would add tiny unnoticeable but fake places. https://vinepair.com/cocktail-chatter/cartographers-fake-places-maps/ https://vinepair.com/cocktail-chatter/cartographers-fake-pla...