I kind of understand your line of thought, but it is implicitly taking sides: if employers can just say "you're only allowed to work on X" to shut down any sort

I kind of understand your line of thought, but it is implicitly taking sides: if employers can just say "you're only allowed to work on X" to shut down any sort of metadiscussion about the conditions under which X is done, that makes any sort of work to change these conditions nearly impossible.