3 ms·
Do you think we can have an open source model whose only role is to classify an incoming prompt as a possible override or injection attack and thereby decide wh
by mercurialsolo 3y ago
Do you think we can have an open source model whose only role is to classify an incoming prompt as a possible override or injection attack and thereby decide whether to execute it or not?
- esjeon 3y agoIf it gets fully open sourced, attackers can use it to find its holes more efficiently using automated tools.
- hgsgm 3y agoThat's open source in general yeah.
- esjeon 3y agoYeah, but software is complex, and we don't have tools to effectively analyze its code. The scanning solutions currently available in the market are really crude, and most of them perform behavioral analysis looking for very basic vulns. In case of AI models, brute-forcing is much easier as their input channels are limited. Also, they are probabilistic by design, so hardening them is much more difficult than conventional SW. Code leak is one thing, things can get really bad if the prod weights are leaked. However, the cost of GPU computation is working as a big deterrence, for now. It's expensive to scan a model for vulnerabilities with massive parallelism. But, it also means it's difficult for developers to verify their models, so manual guesswork is still a valid attack strategy.
- simonw 3y agoI talk about that in the post. I don't think a detection mechanism can be 100% reliable against all future adversarial attacks, which for security I think is unacceptable.
- dangero 3y agoI would not be surprised if this already happens on the OpenAI back end but the attack surface is immense and false positives will damage the platform quality, so it will be hard to solve 100% given we have no concept of how many ways it can be done.