I would not do this at the server level, but instead, at the user level. I use postfix as my mail server, and tell it to use procmail for incoming emails. Ins

I would not do this at the server level, but instead, at the user level. I use postfix as my mail server, and tell it to use procmail for incoming emails. Install procmail, and set up a .procmailrc file in your user account. Procmail is powerful, and let's you run scripts against incoming email, and it also let's you bounce emails. Combining a script that attempts to decode the email, with a bounce instruction, if that decryption fails, should get you what you need.