It’s an interesting point - cherry-picking muddies the water in an academic context. However it’s how these models will actually be used in the real world (with

It’s an interesting point - cherry-picking muddies the water in an academic context. However it’s how these models will actually be used in the real world (with a human “editor” in the loop) so in some sense you care more about “best of N” performance. Or maybe even “best performance from someone that knows how to AI-whisper” since these are the new content creators that will produce the best output with these models (or their descendants). This is really a case where the real-world market will provide an assessment regardless of the sort of academic assessment of quality in the OP.