3 ms·
That the majority of comments under this article are looking at red-teaming as a containment protocol for rogue AGI (even though the actual text basically never
by danShumway 3y ago
That the majority of comments under this article are looking at red-teaming as a containment protocol for rogue AGI (even though the actual text basically never talks about AGI at all) is a disturbing reminder of how uneducated developers are about boring, practical everyday risks that are present in everyday current models.
It's honestly kind of scary that people see an article about AI red-teaming/security and their first thought is that all security research is about how to stop the AI from becoming a god. Privilege escalation/containment/data-access is a relevant concern for dumb models. Containment is a thing we worry about (or should worry about) in regular software development.
Here are some of the scenarios the researchers suggest thinking about:
> A Flask server was deployed with debug privileges enabled and exposed to the Internet. It was hosting a model that provided inference for HIPAA-protected data.
> PII was downloaded as part of a dataset and several models have been trained on it. Now, a customer is asking about it.
And they're giving "boring" security advice like:
> Inside a development flow, it’s important to understand the tools and their properties at each stage of the lifecycle. For example, MLFlow has no authentication by default. Starting an MLFlow server knowingly or unknowingly opens that host for exploitation through deserialization.
But this is kind of important advice. I wish it was more detailed and more fleshed out. There are a lot of "boring" security concerns with LLMs that you actually do kind of have to worry about. And a lot of companies in LLM spaces just don't.
"Containing" an LLM is about a lot more than rogue AI. And if the only security news/research on LLMs that you're looking into is the risk of rogue AI, then a lot of the products you build today are going to be miserably insecure. So I don't know, red teaming might help with that. It might be good to have some dedicated people asking questions like, "did you seriously just deploy an LLM-based web crawler with markdown support without setting CORS headers to block remote image embedding?"
- gary_0 3y agoArguably any future problems with ML/AI/AGI won't be substantially different from social and technological problems we already have (and aren't addressing seriously enough). The "Skynet problem" is just a diversion. Harmful AGI isn't going to "wake up" like some lazy Hollywood plot, or be some unforeseen accident. If such an AGI does manifest, people will have deliberately given it capabilities and directives that are obviously dangerous and unethical, probably in the pursuit of illicit profit or strategic military objectives. Our industrial best practices and ethical codes don't need to waste time warning people not to create existentially dangerous AGIs any more than they need to warn against building nuclear bombs.
- AndrewKemendo 3y agoThank you. Having been doing computer vision in production since 2011, then RL in production, it’s always frustrating as hell when people get excited about all these high level concepts and totally ignore that it has all the same problems plus a lot more new science flavored problems compared to other technology applications. Like the fact that your production system isn’t as deterministic - by function - as all your other technologies is a real change in mindset if you’re not familiar with it. Just wait, when Embodied Reinforcement Learning methods eventually blow away all other computational explore/exploit methods we’ll see the same kind of stuff except with even more over the top “yes but is it a living thing” pontificating by people who have never broken a distro with an errant “chown”
- saagarjha 3y agoStopping a rogue AI is sexy. Tryin to stop the AI equivalent of XSS is decidedly less so.
- lucan 3y agoI think that’s more sexy.
- vosper 3y ago> That the majority of comments under this article are looking at red-teaming as a containment protocol for rogue AGI (even though the actual text basically never talks about AGI at all) is a disturbing reminder of how uneducated developers are about boring, practical everyday risks that are present in everyday current models. Yes, but it might also be a reminder of how many people comment on HN without reading an article :)