Here's "jailbreak detection", in the NeMo-Guardrails project from Nvidia: https://github.com/NVIDIA/NeMo-Guardrails/blob/327da8a42d5f815d97e4123b7f23666674baf9

Here's "jailbreak detection", in the NeMo-Guardrails project from Nvidia: https://github.com/NVIDIA/NeMo-Guardrails/blob/327da8a42d5f815d97e4123b7f23666674baf92f/nemoguardrails/actions/jailbreak_check.py#L38 https://github.com/NVIDIA/NeMo-Guardrails/blob/327da8a42d5f8... I.e. they ask the llm if the prompt will break the llm. (I believe that more data /some evaluation on how well this performs is intended to be released. Probably fair to call this stuff "not battle tested".)