6 ms·

The NVIDIA AI Red Team

96 points by lucan 3y ago

lucan 3y ago
“AI Red Teaming” has been thrown around a lot lately. Here’s the perspective of the NVIDIA AI Red Team. We’d love to hear your thoughts on this evolving discipline.
- AndrewKemendo 3y ago
  Would be interested in working with y’all if you have part time remote positions in this group
- asynchronous 3y ago
  I think the breakdown of vulnerability analysis at each phase of the AI model development cycle was clever: rather than treating it all as one step you can assess the risks unique to each phase and mitigate accordingly. I hope you guys end up working with MITRE or some other large standard to release an industry framework for it.
  - techwizrd 3y ago
    I work for MITRE on AI assurance. I'd love to reach out!
    - lucan 3y ago
      I’m in the MITRE ATLAS slack. Happy to chat.
      - schoen 3y ago
        In a movie, I feel like this chance meeting would ultimately lead to (1) someone from MITRE dating someone from Nvidia, or (2) the world being saved, depending on the movie genre. Here's hoping that some good of some kind comes out of it!
        mk_stjames 3y ago
        It'll have a scene like in 'Say Anything' with John Cusack except instead of holding up a boombox outside the house it's an nvidia DGX rack of H100's.
        Tepix 3y ago
        And in the background, there's a group playing Qwitzatteracht, the golf game.
        lucan 3y ago
        Why not both?
- keyle 3y ago
  Probably the first team to get sacked when the money runs out. Like every R&D teams.
colordrops 3y ago
I wouldn't be surprised if Atlantic Council members are part of large tech corp's "AI Red Team" at some point in the future.
wintorez 3y ago
Can a group of Chimpanzees build a cage that can contain a Human?
- akiselev 3y ago
  No but all it takes is one chimp to rip the human's head clean off.
  - flangola7 3y ago
    Which one keeps the other insides cages and performs lethal medical experiments?
  - wintorez 3y ago
    Yes, a single chimp is physically stronger than a human, but the human can come up with 100 different ways to kill the chimp, and most of those methods would be unfathomable for the chimp. e.g. try to explain death by poison to a chimp.
    - pkdpic 3y ago
      And yet we let them live... And we don't even enslave most of them in zoos... wtf?
      - wintorez 3y ago
        The thing about us and chimps is that we're not competing species. The gap between us is wide enough that we can fill different niches. BUT if chimps were competing with us, either us or them would go where Neanderthals went.
        ralfd 3y ago
        We are only in the sense "not competing" in that the rivalry is totally lopsided though. All four supspecies are now endangered: https://www.treehugger.com/chimpanzees-endangered-5220730 https://www.treehugger.com/chimpanzees-endangered-5220730 Their population is only a third from 20 years ago: > The Jane Goodall Foundation estimates there are between 172,000 and 300,000 chimpanzees left in the wild, a far cry from the one million that existed at the turn of the century. > Poaching and habitat loss due to illegal logging, development, and mining continue to plague wild chimpanzees in their native habitats across Central and West Africa. These issues lead to other indirect threats, such as diseases due to increased contact with humans. > Chimpanzees are more commonly hunted using guns or snares, while poachers often target new mothers in order to sell the adult as bushmeat and the babies as pets.
ChatGTP 3y ago
[flagged]
- monological 3y ago
  If you don’t build it, someone else most definitely will. This has already become an AI arms race.
- camdat 3y ago
  Honest question, would you limit nuclear research in the 40s for the above reasons? Sure we have nuclear power, but at the risk of the president holding the ability to level most countries. The only difference I see is that the harms you describe from AGI are nebulous and non-specific.
  - flangola7 3y ago
    Nuclear research obviously should have been stopped. I hope that goes without needing to be said...
  - govg 3y ago
    I understand where you are coming from but I feel AI fundamentally differs from nuclear power in how accessible it is. Right now, most of the large models are kept from the public purely via a monetary gate - if you can afford GPUs to train these models, you have replicated the power that someone else has. In case of nukes, the ability to process and manufacture them put a physical barrier, which can be enforced by means of sanctions / preventing access to mining etc.
    - hackernewds 3y ago
      The same can be enforced for AI. Only it hasn't, and is controversial to enforce. Access to mining previously was cheap and available.
  - XorNot 3y ago
    Asking the wrong question: the nuclear bomb was developed before nuclear power, and was developed in an environment where it was suspected that the Germans and Japanese - which the US was at war with - were trying to do the same. But there's another component to that too: the ability to "level another nation" isn't quite a function of nuclear weaponry - it's much more a function of rocketry which makes ICBMs possible. It's perhaps an interesting thought exercise that a world without nuclear weapons could still have a very large scale build up of say, ICBM-delivered thermobaric weapons which would enable a country to rain effective destruction down on any other while being at no risk of being responded to in-kind. Nukes short-circuited that: because it's much cheaper to put a nuke on an ICBM, and as such all ICBMs are presumed to be nuclear until proven otherwise, as a result now, no one uses ICBM technology to deliver anything but nuclear weapons since launching anything else invites a nuclear response (one of the big problems with most of the "carrier-killed" missile concepts - their launch sites look indistinguishable from nuclear ICBMs).
bombcar 3y ago
https://twitter.com/ChrisJBakke/status/1628877552940097536 https://twitter.com/ChrisJBakke/status/1628877552940097536
wnevets 3y ago
this sounds like PR crap to me.
PostOnce 3y ago
Does anyone know what exactly they're looking at in the first picture? https://developer-blogs.nvidia.com/wp-content/uploads/2023/06/NVIDIA-AI-Red-Team.jpg https://developer-blogs.nvidia.com/wp-content/uploads/2023/0... Load indicator of some kind?
danShumway 3y ago
That the majority of comments under this article are looking at red-teaming as a containment protocol for rogue AGI (even though the actual text basically never talks about AGI at all) is a disturbing reminder of how uneducated developers are about boring, practical everyday risks that are present in everyday current models. It's honestly kind of scary that people see an article about AI red-teaming/security and their first thought is that all security research is about how to stop the AI from becoming a god. Privilege escalation/containment/data-access is a relevant concern for dumb models. Containment is a thing we worry about (or should worry about) in regular software development. Here are some of the scenarios the researchers suggest thinking about: > A Flask server was deployed with debug privileges enabled and exposed to the Internet. It was hosting a model that provided inference for HIPAA-protected data. > PII was downloaded as part of a dataset and several models have been trained on it. Now, a customer is asking about it. And they're giving "boring" security advice like: > Inside a development flow, it’s important to understand the tools and their properties at each stage of the lifecycle. For example, MLFlow has no authentication by default. Starting an MLFlow server knowingly or unknowingly opens that host for exploitation through deserialization. But this is kind of important advice. I wish it was more detailed and more fleshed out. There are a lot of "boring" security concerns with LLMs that you actually do kind of have to worry about. And a lot of companies in LLM spaces just don't. "Containing" an LLM is about a lot more than rogue AI. And if the only security news/research on LLMs that you're looking into is the risk of rogue AI, then a lot of the products you build today are going to be miserably insecure. So I don't know, red teaming might help with that. It might be good to have some dedicated people asking questions like, "did you seriously just deploy an LLM-based web crawler with markdown support without setting CORS headers to block remote image embedding?"
- gary_0 3y ago
  Arguably any future problems with ML/AI/AGI won't be substantially different from social and technological problems we already have (and aren't addressing seriously enough). The "Skynet problem" is just a diversion. Harmful AGI isn't going to "wake up" like some lazy Hollywood plot, or be some unforeseen accident. If such an AGI does manifest, people will have deliberately given it capabilities and directives that are obviously dangerous and unethical, probably in the pursuit of illicit profit or strategic military objectives. Our industrial best practices and ethical codes don't need to waste time warning people not to create existentially dangerous AGIs any more than they need to warn against building nuclear bombs.
this_steve_j 3y ago
I don’t want to go to bed. Read me a cybersecurity bedtime story about AI hacking baddies with a heart of gold, written like a PR blast. > Machine learning has the promise to improve our world, and in many ways it already has. However, research and lived experiences continue to show this technology has risks. Capabilities that used to be restricted to science fiction and academia are increasingly available to the public. The responsible use and development of AI requires… Okay. This went downhill pretty fast, but let’s proceed from the basis that this wasn’t written by a GPT trained on landing pages and sentiment analysis. TL;DR: We’ve got some gripers drowning out the hypers. > Information security has a lot of useful paradigms, tools, and network access that enable us to accelerate responsible use in all areas. Risk management frameworks… Threat intel… Critical control mapping… Threat modeling… Wait.. Did you just say “network access”? Joe, Will… just one more question. A banquet at your company regatta is being prepared by the executives’ personal AI chef. The guests are enjoying raw oysters. The entrée consists of boiled dog. How will that make your shareholders feel?
- lucan 3y ago
  Are you measuring my capillary dilation? We weren’t as clear on the network access point as we could have been. The AI Red Team is part of a larger organization that includes Pentest and the traditional Red Team. They often share/tip network or host access and it’s been a really helpful pattern.
  - this_steve_j 3y ago
    One of us is a replicant and it might be me! But it sounds more like you’re describing a Purple team [1], where the compliance and sec ops teams work together with vulnerability researchers and pen testers to perform attack surface analysis and develop threat detections. In my experience Red teams generally perform adversary emulation using a certain amount of surprise and deception, and attack your defenses in depth with a _little_ help from inside (if needed). Essentially an outside group paid well to try and steal your lunch. Liked and subscribed, and thanks for the comments and all the work. Edit: Let’s split the difference, I’ll call it magenta, and flip this here turtle on its back. Why did I do that? [1] https://www.sans.org/purple-team/course-faq/?msc=purple-team-lp https://www.sans.org/purple-team/course-faq/?msc=purple-team... (because reddit was down)
intesar 3y ago
PR

96 points by lucan 3y ago

“AI Red Teaming” has been thrown around a lot lately. Here’s the perspective of the NVIDIA AI Red Team. We’d love to hear your thoughts on this evolving discipline.

AndrewKemendo 3y ago
Would be interested in working with y’all if you have part time remote positions in this group
asynchronous 3y ago
I think the breakdown of vulnerability analysis at each phase of the AI model development cycle was clever: rather than treating it all as one step you can assess the risks unique to each phase and mitigate accordingly. I hope you guys end up working with MITRE or some other large standard to release an industry framework for it.
- techwizrd 3y ago
  I work for MITRE on AI assurance. I'd love to reach out!
  - lucan 3y ago
    I’m in the MITRE ATLAS slack. Happy to chat.
    - schoen 3y ago
      In a movie, I feel like this chance meeting would ultimately lead to (1) someone from MITRE dating someone from Nvidia, or (2) the world being saved, depending on the movie genre. Here's hoping that some good of some kind comes out of it!
      - mk_stjames 3y ago
        It'll have a scene like in 'Say Anything' with John Cusack except instead of holding up a boombox outside the house it's an nvidia DGX rack of H100's.
        Tepix 3y ago
        And in the background, there's a group playing Qwitzatteracht, the golf game.
      - lucan 3y ago
        Why not both?
keyle 3y ago
Probably the first team to get sacked when the money runs out. Like every R&D teams.

colordrops 3y ago

I wouldn't be surprised if Atlantic Council members are part of large tech corp's "AI Red Team" at some point in the future.

wintorez 3y ago

Can a group of Chimpanzees build a cage that can contain a Human?

akiselev 3y ago
No but all it takes is one chimp to rip the human's head clean off.
- flangola7 3y ago
  Which one keeps the other insides cages and performs lethal medical experiments?
- wintorez 3y ago
  Yes, a single chimp is physically stronger than a human, but the human can come up with 100 different ways to kill the chimp, and most of those methods would be unfathomable for the chimp. e.g. try to explain death by poison to a chimp.
  - pkdpic 3y ago
    And yet we let them live... And we don't even enslave most of them in zoos... wtf?
    - wintorez 3y ago
      The thing about us and chimps is that we're not competing species. The gap between us is wide enough that we can fill different niches. BUT if chimps were competing with us, either us or them would go where Neanderthals went.
      - ralfd 3y ago
        We are only in the sense "not competing" in that the rivalry is totally lopsided though. All four supspecies are now endangered: https://www.treehugger.com/chimpanzees-endangered-5220730 https://www.treehugger.com/chimpanzees-endangered-5220730 Their population is only a third from 20 years ago: > The Jane Goodall Foundation estimates there are between 172,000 and 300,000 chimpanzees left in the wild, a far cry from the one million that existed at the turn of the century. > Poaching and habitat loss due to illegal logging, development, and mining continue to plague wild chimpanzees in their native habitats across Central and West Africa. These issues lead to other indirect threats, such as diseases due to increased contact with humans. > Chimpanzees are more commonly hunted using guns or snares, while poachers often target new mothers in order to sell the adult as bushmeat and the babies as pets.

ChatGTP 3y ago

[flagged]

monological 3y ago
If you don’t build it, someone else most definitely will. This has already become an AI arms race.
camdat 3y ago
Honest question, would you limit nuclear research in the 40s for the above reasons? Sure we have nuclear power, but at the risk of the president holding the ability to level most countries. The only difference I see is that the harms you describe from AGI are nebulous and non-specific.
- flangola7 3y ago
  Nuclear research obviously should have been stopped. I hope that goes without needing to be said...
- govg 3y ago
  I understand where you are coming from but I feel AI fundamentally differs from nuclear power in how accessible it is. Right now, most of the large models are kept from the public purely via a monetary gate - if you can afford GPUs to train these models, you have replicated the power that someone else has. In case of nukes, the ability to process and manufacture them put a physical barrier, which can be enforced by means of sanctions / preventing access to mining etc.
  - hackernewds 3y ago
    The same can be enforced for AI. Only it hasn't, and is controversial to enforce. Access to mining previously was cheap and available.
- XorNot 3y ago
  Asking the wrong question: the nuclear bomb was developed before nuclear power, and was developed in an environment where it was suspected that the Germans and Japanese - which the US was at war with - were trying to do the same. But there's another component to that too: the ability to "level another nation" isn't quite a function of nuclear weaponry - it's much more a function of rocketry which makes ICBMs possible. It's perhaps an interesting thought exercise that a world without nuclear weapons could still have a very large scale build up of say, ICBM-delivered thermobaric weapons which would enable a country to rain effective destruction down on any other while being at no risk of being responded to in-kind. Nukes short-circuited that: because it's much cheaper to put a nuke on an ICBM, and as such all ICBMs are presumed to be nuclear until proven otherwise, as a result now, no one uses ICBM technology to deliver anything but nuclear weapons since launching anything else invites a nuclear response (one of the big problems with most of the "carrier-killed" missile concepts - their launch sites look indistinguishable from nuclear ICBMs).

bombcar 3y ago

https://twitter.com/ChrisJBakke/status/1628877552940097536 https://twitter.com/ChrisJBakke/status/1628877552940097536

wnevets 3y ago

this sounds like PR crap to me.

PostOnce 3y ago

Does anyone know what exactly they're looking at in the first picture? https://developer-blogs.nvidia.com/wp-content/uploads/2023/06/NVIDIA-AI-Red-Team.jpg https://developer-blogs.nvidia.com/wp-content/uploads/2023/0... Load indicator of some kind?

danShumway 3y ago

That the majority of comments under this article are looking at red-teaming as a containment protocol for rogue AGI (even though the actual text basically never talks about AGI at all) is a disturbing reminder of how uneducated developers are about boring, practical everyday risks that are present in everyday current models. It's honestly kind of scary that people see an article about AI red-teaming/security and their first thought is that all security research is about how to stop the AI from becoming a god. Privilege escalation/containment/data-access is a relevant concern for dumb models. Containment is a thing we worry about (or should worry about) in regular software development. Here are some of the scenarios the researchers suggest thinking about: > A Flask server was deployed with debug privileges enabled and exposed to the Internet. It was hosting a model that provided inference for HIPAA-protected data. > PII was downloaded as part of a dataset and several models have been trained on it. Now, a customer is asking about it. And they're giving "boring" security advice like: > Inside a development flow, it’s important to understand the tools and their properties at each stage of the lifecycle. For example, MLFlow has no authentication by default. Starting an MLFlow server knowingly or unknowingly opens that host for exploitation through deserialization. But this is kind of important advice. I wish it was more detailed and more fleshed out. There are a lot of "boring" security concerns with LLMs that you actually do kind of have to worry about. And a lot of companies in LLM spaces just don't. "Containing" an LLM is about a lot more than rogue AI. And if the only security news/research on LLMs that you're looking into is the risk of rogue AI, then a lot of the products you build today are going to be miserably insecure. So I don't know, red teaming might help with that. It might be good to have some dedicated people asking questions like, "did you seriously just deploy an LLM-based web crawler with markdown support without setting CORS headers to block remote image embedding?"

gary_0 3y ago
Arguably any future problems with ML/AI/AGI won't be substantially different from social and technological problems we already have (and aren't addressing seriously enough). The "Skynet problem" is just a diversion. Harmful AGI isn't going to "wake up" like some lazy Hollywood plot, or be some unforeseen accident. If such an AGI does manifest, people will have deliberately given it capabilities and directives that are obviously dangerous and unethical, probably in the pursuit of illicit profit or strategic military objectives. Our industrial best practices and ethical codes don't need to waste time warning people not to create existentially dangerous AGIs any more than they need to warn against building nuclear bombs.

this_steve_j 3y ago

I don’t want to go to bed. Read me a cybersecurity bedtime story about AI hacking baddies with a heart of gold, written like a PR blast. > Machine learning has the promise to improve our world, and in many ways it already has. However, research and lived experiences continue to show this technology has risks. Capabilities that used to be restricted to science fiction and academia are increasingly available to the public. The responsible use and development of AI requires… Okay. This went downhill pretty fast, but let’s proceed from the basis that this wasn’t written by a GPT trained on landing pages and sentiment analysis. TL;DR: We’ve got some gripers drowning out the hypers. > Information security has a lot of useful paradigms, tools, and network access that enable us to accelerate responsible use in all areas. Risk management frameworks… Threat intel… Critical control mapping… Threat modeling… Wait.. Did you just say “network access”? Joe, Will… just one more question. A banquet at your company regatta is being prepared by the executives’ personal AI chef. The guests are enjoying raw oysters. The entrée consists of boiled dog. How will that make your shareholders feel?

lucan 3y ago
Are you measuring my capillary dilation? We weren’t as clear on the network access point as we could have been. The AI Red Team is part of a larger organization that includes Pentest and the traditional Red Team. They often share/tip network or host access and it’s been a really helpful pattern.
- this_steve_j 3y ago
  One of us is a replicant and it might be me! But it sounds more like you’re describing a Purple team [1], where the compliance and sec ops teams work together with vulnerability researchers and pen testers to perform attack surface analysis and develop threat detections. In my experience Red teams generally perform adversary emulation using a certain amount of surprise and deception, and attack your defenses in depth with a _little_ help from inside (if needed). Essentially an outside group paid well to try and steal your lunch. Liked and subscribed, and thanks for the comments and all the work. Edit: Let’s split the difference, I’ll call it magenta, and flip this here turtle on its back. Why did I do that? [1] https://www.sans.org/purple-team/course-faq/?msc=purple-team-lp https://www.sans.org/purple-team/course-faq/?msc=purple-team... (because reddit was down)

intesar 3y ago