3 ms·
Yeah, that's part of the confusion here. Some models do indeed set some of their rules using a concatenated system prompt - but most of the "values" are baked
by simonw 3y ago
Yeah, that's part of the confusion here.
Some models do indeed set some of their rules using a concatenated system prompt - but most of the "values" are baked in through instruction tuning.
You can test that yourself by running local models (like Llama 2) in a context where you completely control or omit the system prompt. They will still refuse to give you bomb making recipes, or tell you how to kill Apache 2 processes (Llama 2 is notoriously sensitive in its default conditions.)
- ipython 3y agoDon't worry, we're speed running the last 50 years of computer security. What's old is now new again. Already looking at poor web application security on emerging AI/MLops tools making it rain like the 90's once again; then we have in-band signalling and lack of separation between code & data, just like back in the 70s and 80s. I totally get your frustration, it's because you've seen the pattern before. Enjoy the ride as we all rediscover these fundamental truths we learned decades ago!
- mindcrime 3y agoSay hello to BlueBoxGPT and the new era of "llm phreaking"!