5 ms·
If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions. Interestingly enough
by not_a_bot_4sho 3mo ago
If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions.
Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.
- simonw 3mo agoI'm not sure there's any level of iterations that could result in a credible decision that model A clearly draws a better pelican riding a bicycle than model B. What does "better" even mean there?
- not_a_bot_4sho 3mo ago(I came to delete but was too late. So edits are in.) Wow, that's a stark take. I suppose I'm biased towards a scientific viewpoint. All the best.