It depends on how you set it up. You can use a very non-expressive valuation that assigns a score, but then you need to use reinforcement learning techniques to

It depends on how you set it up. You can use a very non-expressive valuation that assigns a score, but then you need to use reinforcement learning techniques to transform that score to a model update. Alternatively, if your valuation represents a differentiable metric function from your target you have a way of going directly from your output to a model update. The second way requires dramatically fewer update steps (usually) than the first. Thus - having your adversarial target be differentiable definitely helps, though it is possible to do even absent such a criterion.