5 ms·

But it's "fooling" you into thinking that it's been "fooled". There is no tricking. You just didn't prevent the model from presenting this as an output. Get re

But it's "fooling" you into thinking that it's been "fooled". There is no tricking. You just didn't prevent the model from presenting this as an output. Get real! It's not an attack it's just honestly laziness on the part of you since this must be your chatbot that you have trained and deployed. Consider all of the things that a user could input and if you don't want the program outputting to the user a certain type of output then you have to make sure that output can't be a possibility. Chatbots are trained bullshitters. Sociopathic bullshitters. They tell you, the user, what it thinks it should tell you.

dr_zoidberg 4y ago
> ...Consider all of the things that a user could input and if you don't want the program outputting to the user a certain type of output then you have to make sure that output can't be a possibility. But here's the problem: DL systems are by definition made to be good at generalization, so the input space is (could be) infinite. Also if you're using large pretrained models, you don't have power over the examples that it has already seen in that training. You can fine-tune for your use case, but there's the latent posibility that the "old memories" may come back. Which is valuable for the model to learn complex things that you may not have enough data for, but you know, it can also come with a few surprises.