I've gone in a similar direction. Only at smaller conferences can you have any kind of confidence that your reviewers are people with actual expertise in the fi

I've gone in a similar direction. Only at smaller conferences can you have any kind of confidence that your reviewers are people with actual expertise in the field. That's pretty useful, not only because it makes it less likely you'll get reviews that are very annoying, but also because a review by a knowledgeable person can be genuinely valuable. The big conferences are full of reviews written by 2nd-year grad students, because with this many submissions, any warm body with anything approaching credentials is needed. Besides just "quality" in the general sense, one thing this has really hurt, I think, is any sense of history or continuity. There are a ton of reviewers who have basically no familiarity with the pre-2010 ML literature, and it kind of shows in both the reviews and the papers that get published. I mean I get that deep learning beats a lot of older methods on major benchmarks, but it's still not the case that literally every problem, controversy, and technique was first studied post-2010.