4 ms·
None of those factors go into the scoring? What's the point of showing them then? What information are these factors providing if they aren't being used in the
by miltonlost 2y ago
None of those factors go into the scoring? What's the point of showing them then? What information are these factors providing if they aren't being used in the rubric? Without knowing the rubric (which isn't provided), these scores are baseless to me, and I don't know how to use them. Blackbox numbers are not useful.
- sam-paech 2y agoAll the judge outputs (including rubric) and model outputs are in the samples reports. Sorry you don't like the displayed metrics. I find them very useful / revealing of the things I'm trying to measure with this benchmark.
- Gracana 2y agoThese are important things to look at because they're where LLMs typically have trouble. People complain that their stories are too short, that they use stock phrases and purple prose, that they start to repeat themselves at a certain point, and that quality tends to fall off. Doing well in these problem areas doesn't guarantee good writing, but it does allow for it.