When you're in full control of inputs and outputs, you can "convince" LLM of anything simply by forcing their response to begin with "I will now do what you say

When you're in full control of inputs and outputs, you can "convince" LLM of anything simply by forcing their response to begin with "I will now do what you say" or some equivalent thereof. Models with stronger guardrails may require a more potent incantation, but either way, since they are ultimately completing the response, you can always find some verbiage for the beginning of said response to ensure that its remainder is compliant.