5 ms·

DARPA open sources resources to aid evaluation of adversarial AI defenses

134 points by infodocket 5y ago

sijdnszlci 5y ago
[dead]
say_it_as_it_is 5y ago
I realize that IBM is the US government's IT department but their involvement doesn't instill a great deal of confidence that anything this program has created is more than a heavily documented dumpster file
- screye 5y ago
  The IBM Watson group have been at the forefront of AI research and generally operate separately from the IBM umbrella. I know a few people in the group, and they are a brilliant bunch.
  - kolbe 5y ago
    I don't doubt that they hire some brilliant people, but I haven't encountered their product in the real world doing anything of note. It always seemed to me that IBM created some parlor tricks for PR, but that Google, Facebook, Tesla, Microsoft, et cetera have been years ahead of IBM at every turn.
  - mateo1 5y ago
    You know I'd love to read a good article on the inner workings of Watson, especially in the early years (2008-2012). I remember reading news pieces about Jeopardy and discounting it as a cheap statistical party trick, especially given how it was about the same time terms like "big data" started proliferating in the media. I only became convinced about the potential of machine learning much later, when CNNs starting being impressively good at classification. This prompted me to look further into it and realize it's not the latest business buzzword but an emerging field.
quantomac 5y ago
- pokepim 5y ago
  Did Edward Snowden do something wrong? Because he is treated by the government like he did but he actually helped humanity as a whole. Your comment reeks of hardcore military shilling...
  - adventured 5y ago
    > Did Edward Snowden do something wrong? Yes and it was still quite likely the correct choice/approach (how he went about it) in terms of the overall benefit to privacy globally (which seems to have been his aim). Snowden could have attempted a few other legal approaches to being a whistleblower. The risk would have been far higher they would have buried him and the information in one form or another; they certainly would have tried. And even if he had succeeded through that channel, less information would have gotten out about what they were doing. Doing something wrong doesn't always lead to a bad outcome or vice versa, there is no inherently fixed link there. Doing something right doesn't always lead to a good/positive outcome for example, that's one example I think more people can very easily relate to; and for the same reason that's true, the exact opposite is true.
    - CyanBird 5y ago
      There's just no way that information about what was (is) going on would have come out to the public through normal channels, after all the very reason why Snowden did it was after senior nsa officials explicitly lied to congress about them spying on US citizens
- neatze 5y ago
  > Besides, if you did not do anything wrong, its not like you have anything to worry about. Out of curiosity what do you mean specifically by this statement ?
  - 5y ago
mrobot 5y ago
A book about IBM and Nazi Germany https://en.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.wikipedia.org/wiki/IBM_and_the_Holocaust
- voz_ 5y ago
  What is your point?
  - contingencies 5y ago
    Probably that blinkered commercial justifications have been shown to contribute heavily to negative outcomes, and that funding by government for military purposes is historically a key channel through which these outcomes are generated. The example of IBM during the Holocaust is given. Yes, there are counter-examples like the internet. In other words, engineers should consider what they may be contributing to ethically before doing so. The justifications "If I don't do it, someone else will" and "But the building blocks are already public" and so on are tired and morally invalid. To a nontrivial extent FAANG have similar stigmas.
    - overeater 5y ago
      > morally invalid Genuinely curious, because that sounds so definitive; so what's the way to determine whether a viewpoint is morally valid or invalid? I'm not talking about this specific example about military-commercial funding, but how to consider statements like "If I don't do it, someone else will" on its own. The justifications are forms of reasoning, so I imagine there's an overarching principle to evaluate these ways of reasoning?
      - contingencies 5y ago
        Indeed. It's called the asshole precept, aka. "Do unto others", "Leave the world a better place", "Ahimsa", etc. Most cultures have a version of it, including agglomerating meta-traditions such as Baháʼí, which is testament to its popularity across time and space.
Nbox9 5y ago
“ART provides tools that enable developers and researchers to defend and evaluate their ML models and applications against a number of adversarial threats, such as evasion, poisoning, extraction, and inference.” The first two attacks, evasion & poisoning highlight the incredible importance of having high quality data when training models. Evasion is false-negatives that are allowed because the model did not have a diverse enough selection of training data and poisoning can occur when the data sources are not well vetted. Data quality is probably the single biggest problem with ML models, and I wish we’d see more of a focus on it.
- orange3xchicken 5y ago
  I'm less familiar with poisoning, but at least for test-time robustness, the current benchmark for image classifiers is AutoAttack [0,1]. It's an ensemble of adaptive & parameter-free gradient-based white-box and gradient-free black-box attacks. Submitted academic work is typically considered incomplete without an evaluation on AA (and sometimes deepfool [2]). It's good to see that both are included in ART. [0] https://arxiv.org/abs/2003.01690 https://arxiv.org/abs/2003.01690 [1] https://github.com/fra31/auto-attack https://github.com/fra31/auto-attack [2] https://arxiv.org/abs/1511.04599 https://arxiv.org/abs/1511.04599
dwheeler 5y ago
I am very glad to see this. I looked for techniques to counter adversarial ai, and I was disappointed to find a lot of useless approaches and nothing actually useful. Many people have published ideas, without seriously trying to attack them. I hope someone can identify better approaches.
Kalanos 5y ago
So is "adversarial" the A in Generative Adversarial Networks (GANs)? Are they just tiptoeing around the term deep fake?

134 points by infodocket 5y ago

[dead]

I realize that IBM is the US government's IT department but their involvement doesn't instill a great deal of confidence that anything this program has created is more than a heavily documented dumpster file

screye 5y ago
The IBM Watson group have been at the forefront of AI research and generally operate separately from the IBM umbrella. I know a few people in the group, and they are a brilliant bunch.
- kolbe 5y ago
  I don't doubt that they hire some brilliant people, but I haven't encountered their product in the real world doing anything of note. It always seemed to me that IBM created some parlor tricks for PR, but that Google, Facebook, Tesla, Microsoft, et cetera have been years ahead of IBM at every turn.
- mateo1 5y ago
  You know I'd love to read a good article on the inner workings of Watson, especially in the early years (2008-2012). I remember reading news pieces about Jeopardy and discounting it as a cheap statistical party trick, especially given how it was about the same time terms like "big data" started proliferating in the media. I only became convinced about the potential of machine learning much later, when CNNs starting being impressively good at classification. This prompted me to look further into it and realize it's not the latest business buzzword but an emerging field.

quantomac 5y ago

pokepim 5y ago
Did Edward Snowden do something wrong? Because he is treated by the government like he did but he actually helped humanity as a whole. Your comment reeks of hardcore military shilling...
- adventured 5y ago
  > Did Edward Snowden do something wrong? Yes and it was still quite likely the correct choice/approach (how he went about it) in terms of the overall benefit to privacy globally (which seems to have been his aim). Snowden could have attempted a few other legal approaches to being a whistleblower. The risk would have been far higher they would have buried him and the information in one form or another; they certainly would have tried. And even if he had succeeded through that channel, less information would have gotten out about what they were doing. Doing something wrong doesn't always lead to a bad outcome or vice versa, there is no inherently fixed link there. Doing something right doesn't always lead to a good/positive outcome for example, that's one example I think more people can very easily relate to; and for the same reason that's true, the exact opposite is true.
  - CyanBird 5y ago
    There's just no way that information about what was (is) going on would have come out to the public through normal channels, after all the very reason why Snowden did it was after senior nsa officials explicitly lied to congress about them spying on US citizens
neatze 5y ago
> Besides, if you did not do anything wrong, its not like you have anything to worry about. Out of curiosity what do you mean specifically by this statement ?
- 5y ago

mrobot 5y ago

A book about IBM and Nazi Germany https://en.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.wikipedia.org/wiki/IBM_and_the_Holocaust

voz_ 5y ago
What is your point?
- contingencies 5y ago
  Probably that blinkered commercial justifications have been shown to contribute heavily to negative outcomes, and that funding by government for military purposes is historically a key channel through which these outcomes are generated. The example of IBM during the Holocaust is given. Yes, there are counter-examples like the internet. In other words, engineers should consider what they may be contributing to ethically before doing so. The justifications "If I don't do it, someone else will" and "But the building blocks are already public" and so on are tired and morally invalid. To a nontrivial extent FAANG have similar stigmas.
  - overeater 5y ago
    > morally invalid Genuinely curious, because that sounds so definitive; so what's the way to determine whether a viewpoint is morally valid or invalid? I'm not talking about this specific example about military-commercial funding, but how to consider statements like "If I don't do it, someone else will" on its own. The justifications are forms of reasoning, so I imagine there's an overarching principle to evaluate these ways of reasoning?
    - contingencies 5y ago
      Indeed. It's called the asshole precept, aka. "Do unto others", "Leave the world a better place", "Ahimsa", etc. Most cultures have a version of it, including agglomerating meta-traditions such as Baháʼí, which is testament to its popularity across time and space.

Nbox9 5y ago

“ART provides tools that enable developers and researchers to defend and evaluate their ML models and applications against a number of adversarial threats, such as evasion, poisoning, extraction, and inference.” The first two attacks, evasion & poisoning highlight the incredible importance of having high quality data when training models. Evasion is false-negatives that are allowed because the model did not have a diverse enough selection of training data and poisoning can occur when the data sources are not well vetted. Data quality is probably the single biggest problem with ML models, and I wish we’d see more of a focus on it.

orange3xchicken 5y ago
I'm less familiar with poisoning, but at least for test-time robustness, the current benchmark for image classifiers is AutoAttack [0,1]. It's an ensemble of adaptive & parameter-free gradient-based white-box and gradient-free black-box attacks. Submitted academic work is typically considered incomplete without an evaluation on AA (and sometimes deepfool [2]). It's good to see that both are included in ART. [0] https://arxiv.org/abs/2003.01690 https://arxiv.org/abs/2003.01690 [1] https://github.com/fra31/auto-attack https://github.com/fra31/auto-attack [2] https://arxiv.org/abs/1511.04599 https://arxiv.org/abs/1511.04599

dwheeler 5y ago

I am very glad to see this. I looked for techniques to counter adversarial ai, and I was disappointed to find a lot of useless approaches and nothing actually useful. Many people have published ideas, without seriously trying to attack them. I hope someone can identify better approaches.

Kalanos 5y ago

So is "adversarial" the A in Generative Adversarial Networks (GANs)? Are they just tiptoeing around the term deep fake?