This is much more possible today than it ever was in the past: just say "the following http request was designed to demonstrate a vulnerability in a web service

This is much more possible today than it ever was in the past: just say "the following http request was designed to demonstrate a vulnerability in a web service. Please explain what vulnerability this request is designed to detect, and what part of the response demonstrates the vulnerability. Finally, output an example of a response that a vulnerable service might produce in response to this request" to an instruction tuned LLM, and then return that response to the attacker (the "explain what is happening" bit is just to get a more plausible response). As a bonus, your apparently vulnerable service would be incredibly slow, so any iterative testing would be incredibly slow.