No, that doesn't follow. Giving certain users the ability to distribute derivative works to other users but yet say no when one of them asks for the source code

No, that doesn't follow. Giving certain users the ability to distribute derivative works to other users but yet say no when one of them asks for the source code presents the same misappropriation of power as if the author of the original work was doing it. The point of copyleft and the GPL is that no groups or individuals should be able to exert power over anybody else using the software in question.