6 ms·

> It's an endless cat-and-mouse game. This is often stated but I think it has to be obviously wrong. This isn't a traditional interactive game such as malware

> It's an endless cat-and-mouse game. This is often stated but I think it has to be obviously wrong. This isn't a traditional interactive game such as malware & anti-malware. You have existing sensors which operate under the constraint of [ real world -> theoretic pixel space -> optics & aberrations & sensor noise -> compression ]. And a single adversary which attempts to fake this chain. The detection of fakes isn't even an adversary in this game, it's merely a detection of deviation of the faking process. At some point, probably soon, the faking process will reach a point where any deviation will be drowned out by the noise aspect of optics & sensors & compression.

halpmeh 4y ago
One method of generating things via neural networks is called a generative adversarial network. It works by having two models. One that generates content and one that detects fake content. You train them both in parallel. As the fake detector gets better, so does the generative model at generating fakes. It’s literally a cat-and-mouse game. If someone came up with a scheme to reliably detect your fakes, you could add it to your discriminator model and retrain the generator to improve the fake generation.
- rogers18445 4y ago
  My understanding is that it's not quite that simple. GANs have stability problems (and as a result somewhat out of favor atm) and if the fake detection mechanism isn't a differentiable function itself no training can happen.
  - sigmoid10 4y ago
    The fake detection mechanism (aka discriminator) is usually just another neural network and I bet that's the case here as well. So it must be differentiable and thus, if anyone ever gets a hold of it, it could be easily used to train a generator that will eventually fool the discriminator.
    - cudgy 4y ago
      So, basically a NOT operator on a neural network. Does this even require a differentiation?
      - nl 4y ago
        A NN is trained to make something indistinguishable from reality as possible - so it can't tell the difference. The inverse of that will just claim reality is fake too. What you need a a deep fake and a real video of the same thing, then train on the difference. Clearly this is impossible - which is what makes the problem hard.
        sigmoid10 4y ago
        You actually don't need that. You only need a set of real videos and a generator for fake ones. Then train the discriminator to tell these two classes apart and make use of its differentiability to update the generator in tandem with the discriminator.
        anankaie 4y ago
        Would you run into class balance issues if you do not have a growing pool of initial videos?
        sigmoid10 4y ago
        Not if you have a balanced dataset. If not, you're already getting into trouble with GANs, which is why they were superseded by diffusers on image generation tasks.
      - rusticpenn 4y ago
        Not really a not operator. It’s more like tuning the generated models until the fake detector cannot detect it.
      - anankaie 4y ago
        It depends on how you set it up. You can use a very non-expressive valuation that assigns a score, but then you need to use reinforcement learning techniques to transform that score to a model update. Alternatively, if your valuation represents a differentiable metric function from your target you have a way of going directly from your output to a model update. The second way requires dramatically fewer update steps (usually) than the first. Thus - having your adversarial target be differentiable definitely helps, though it is possible to do even absent such a criterion.
  - jasonjmcghee 4y ago
    It is accurate that GANs have stability problems, but they are absolutely being used today for solving problems similar to this (an output needs to be "improved"). Stable Diffusion produces faces- especially eyes that are malformed. You'll often see "restore faces" in online services which feeds the end result into a GFPGAN which is used to restore faces.
    - anticensor 4y ago
      Diffusors are not GANs.
      - homarp 4y ago
        indeed. that is why you use GAN to fix the face generated https://www.reddit.com/r/StableDiffusion/comments/x33rs4/how_do_i_improve_faces/ https://www.reddit.com/r/StableDiffusion/comments/x33rs4/how...
  - snowpid 4y ago
    While Gradient Descent needs differentiable functions, there are evolutionary algorithms that do not need this and can also train neural networks.
- GauntletWizard 4y ago
  There's an excellent sci-fi exploration of the concept and how humans interact with it in Neal Stephensons "Jipi and the Paranoid Chip"; well worth the read.
  - bornfreddy 4y ago
    Thank you for posting this! Link: http://readfrom.net/neal-stephenson/612819-jipi_and_the_paranoid_chip.html http://readfrom.net/neal-stephenson/612819-jipi_and_the_para...
    - peddling-brink 4y ago
      This link gave me a fake "this phone has been hacked" alert. I recommend avoiding.
dheera 4y ago
Hot take: What if we just accepted that any video might be fake, just like any photograph might be fake? And that we accept that the only thing that's assured to be real is face-to-face, and live with that reality?
- not2b 4y ago
  It's already the legal standard in many places that a court doesn't just accept evidence like a photo or a video. Typically the photographer testifies under penalty of perjury that they were there and can be questioned about the circumstances in which they took the picture/video and any post-processing that might distort what we see. So we can say that either it's real or we can identify the specific people who might be lying.
  - dredmorbius 4y ago
    Unfortunately, in media and the public discourse / memosphere, that's a standard notoriously difficult to establish or enforce. People, and crowds, will respond to first impressions.
    - dheera 4y ago
      What if we put out enough deepfakes that they get desensitized to it and just stop believing stuff on screens altogether?
      - dredmorbius 4y ago
        Past experience suggests that that likely won't be too effective. "Big lie" propaganda is profoundly effective, even amongst those who are painfully aware of its existence and methods as individuals, and at the population level, there seem to be virtually no defences. Rather, there'll need to be the emergence of credible and trustworthy channels which verify messaging and content before it's widely disseminated. That largely means re-implementing the sort of media gatekeepers we've seen in the past. Though the challenge of bad-faith actors emerging in such roles at considerable scale points to further challenges, even with that model. Under a regime in which free speech is considered a fundamental right, any sort of preemptive management by government mandate becomes intensely difficult.

halpmeh 4y ago

One method of generating things via neural networks is called a generative adversarial network. It works by having two models. One that generates content and one that detects fake content. You train them both in parallel. As the fake detector gets better, so does the generative model at generating fakes. It’s literally a cat-and-mouse game. If someone came up with a scheme to reliably detect your fakes, you could add it to your discriminator model and retrain the generator to improve the fake generation.

rogers18445 4y ago
My understanding is that it's not quite that simple. GANs have stability problems (and as a result somewhat out of favor atm) and if the fake detection mechanism isn't a differentiable function itself no training can happen.
- sigmoid10 4y ago
  The fake detection mechanism (aka discriminator) is usually just another neural network and I bet that's the case here as well. So it must be differentiable and thus, if anyone ever gets a hold of it, it could be easily used to train a generator that will eventually fool the discriminator.
  - cudgy 4y ago
    So, basically a NOT operator on a neural network. Does this even require a differentiation?
    - nl 4y ago
      A NN is trained to make something indistinguishable from reality as possible - so it can't tell the difference. The inverse of that will just claim reality is fake too. What you need a a deep fake and a real video of the same thing, then train on the difference. Clearly this is impossible - which is what makes the problem hard.
      - sigmoid10 4y ago
        You actually don't need that. You only need a set of real videos and a generator for fake ones. Then train the discriminator to tell these two classes apart and make use of its differentiability to update the generator in tandem with the discriminator.
        anankaie 4y ago
        Would you run into class balance issues if you do not have a growing pool of initial videos?
        sigmoid10 4y ago
        Not if you have a balanced dataset. If not, you're already getting into trouble with GANs, which is why they were superseded by diffusers on image generation tasks.
    - rusticpenn 4y ago
      Not really a not operator. It’s more like tuning the generated models until the fake detector cannot detect it.
    - anankaie 4y ago
      It depends on how you set it up. You can use a very non-expressive valuation that assigns a score, but then you need to use reinforcement learning techniques to transform that score to a model update. Alternatively, if your valuation represents a differentiable metric function from your target you have a way of going directly from your output to a model update. The second way requires dramatically fewer update steps (usually) than the first. Thus - having your adversarial target be differentiable definitely helps, though it is possible to do even absent such a criterion.
- jasonjmcghee 4y ago
  It is accurate that GANs have stability problems, but they are absolutely being used today for solving problems similar to this (an output needs to be "improved"). Stable Diffusion produces faces- especially eyes that are malformed. You'll often see "restore faces" in online services which feeds the end result into a GFPGAN which is used to restore faces.
  - anticensor 4y ago
    Diffusors are not GANs.
    - homarp 4y ago
      indeed. that is why you use GAN to fix the face generated https://www.reddit.com/r/StableDiffusion/comments/x33rs4/how_do_i_improve_faces/ https://www.reddit.com/r/StableDiffusion/comments/x33rs4/how...
- snowpid 4y ago
  While Gradient Descent needs differentiable functions, there are evolutionary algorithms that do not need this and can also train neural networks.
GauntletWizard 4y ago
There's an excellent sci-fi exploration of the concept and how humans interact with it in Neal Stephensons "Jipi and the Paranoid Chip"; well worth the read.
- bornfreddy 4y ago
  Thank you for posting this! Link: http://readfrom.net/neal-stephenson/612819-jipi_and_the_paranoid_chip.html http://readfrom.net/neal-stephenson/612819-jipi_and_the_para...
  - peddling-brink 4y ago
    This link gave me a fake "this phone has been hacked" alert. I recommend avoiding.

dheera 4y ago

Hot take: What if we just accepted that any video might be fake, just like any photograph might be fake? And that we accept that the only thing that's assured to be real is face-to-face, and live with that reality?

not2b 4y ago
It's already the legal standard in many places that a court doesn't just accept evidence like a photo or a video. Typically the photographer testifies under penalty of perjury that they were there and can be questioned about the circumstances in which they took the picture/video and any post-processing that might distort what we see. So we can say that either it's real or we can identify the specific people who might be lying.
- dredmorbius 4y ago
  Unfortunately, in media and the public discourse / memosphere, that's a standard notoriously difficult to establish or enforce. People, and crowds, will respond to first impressions.
  - dheera 4y ago
    What if we put out enough deepfakes that they get desensitized to it and just stop believing stuff on screens altogether?
    - dredmorbius 4y ago
      Past experience suggests that that likely won't be too effective. "Big lie" propaganda is profoundly effective, even amongst those who are painfully aware of its existence and methods as individuals, and at the population level, there seem to be virtually no defences. Rather, there'll need to be the emergence of credible and trustworthy channels which verify messaging and content before it's widely disseminated. That largely means re-implementing the sort of media gatekeepers we've seen in the past. Though the challenge of bad-faith actors emerging in such roles at considerable scale points to further challenges, even with that model. Under a regime in which free speech is considered a fundamental right, any sort of preemptive management by government mandate becomes intensely difficult.