You are effectively "saying no" to your manager. In your estimation helping your colleagues is more impactful to the company than doing your current task. I do

You are effectively "saying no" to your manager. In your estimation helping your colleagues is more impactful to the company than doing your current task. I don't think there's anything wrong with the attitude. The manager's intention is probably to help you shield yourself from unwanted help requests. But if you actually think you should help that team, then do it!