The issue is that the lottery is never enough. Effectively, the lottery percentage is removed from discussions and the fight moves on to the non-lottery percent

The issue is that the lottery is never enough. Effectively, the lottery percentage is removed from discussions and the fight moves on to the non-lottery percentage where the same arguments of diversity come back to the non-lottery percentage. This is not hypothetical, this is how the US immigration system is currently. There is a diversity lottery, which is a mostly foregone conclusion right now. And yet, there is race/country based quota in the skill based categories and proponents want quotas despite it being a skill being category. Once you create a lottery, you can't say, "Go do your thing in the lottery, this lane is for merit". So the lottery is a waste. Several attempts to phase out caps/quotas from skill based categories through legislature have failed so far. In the most recent case, the congressional black caucus opposed it as it would disadvantage the currently advantaged immigrants from African countries.