You just need to be transparent about your reasons. Just add a clause which prevents Patent Trolls from using your app. People are angry at the App store becaus

You just need to be transparent about your reasons. Just add a clause which prevents Patent Trolls from using your app. People are angry at the App store because they are blocking genuine apps. You can't compare your situation to the App Store or Google, they both have a huge user base, the advantage of being a small company is that nobody is going to notice that you asked a Patent Troll to stop using your app, and if they did it would only improve your image.