I think seeing the prompt that makes it even worse for me. that prompt could have been caught by even a regex on the user input for "secret" would have been a g

I think seeing the prompt that makes it even worse for me. that prompt could have been caught by even a regex on the user input for "secret" would have been a good first layer. TBH, this product would be better served as an LLM that generates a bunch of rules that get statically compiled for what the user can ask and what is being outputted as opposed to an LLM being run on each output. Then you could add your own rules too. It still wouldnt be perfect but would be 1,000,000x cheaper to run and easier to verify the solution. and the rules would gradually grow as more and more edge cases for how to fool llms get found. The company would just need a training set for all the ways to fool an LLM.