2 ms·
It's generally one-shot-only - whatever comes out the first time is what I go with. I've been contemplating a more fair version where each model gets 3-5 attem
by simonw 7mo ago
It's generally one-shot-only - whatever comes out the first time is what I go with.
I've been contemplating a more fair version where each model gets 3-5 attempts and then can select which rendered image is "best".
- andriy_koval 7mo agoI think it will make results way better and more representative of model abilities..
- simonw 7mo agoIt would... but the test is inherently silly, so I'm still not sure if it's worth me investing that extra effort in it.
- irthomasthomas 7mo agoTry llm-consortium with --judging-method rank