> There's something that feels tremendously off to me about these defenses which are to the effect of "What's so wrong with this? In principle, there's nothing

> There's something that feels tremendously off to me about these defenses which are to the effect of "What's so wrong with this? In principle, there's nothing saying a human couldn't also make one of these errors." The big difference is that (most) humans will learn from this mistake. An LLM, however, is always a blank slate so will make the same mistakes over and over again. And more generally, because of how these models work, it would be very strange if they didn't make the same mistakes as humans.