I have no problem with that. But the idea that I should reject doing things that benefit a large number of people just because they don't simultaneously benefit

I have no problem with that. But the idea that I should reject doing things that benefit a large number of people just because they don't simultaneously benefit me in an optimal way is scary. If you're optimizing for your own personal growth, then you're only going to choose to do something that benefits others when it benefits you more than anything else you might choose to do with your limited time and resources. Now look, I know, maybe I'm just being selfish and optimizing for my personal happiness when I donate to a cause I believe in, maybe Bill Gates just does philanthropy because he 'made crappy products' and needs to feel good about himself. But I don't believe that those choices are optimal for personal growth, and I believe that the world turns on them.