3 ms·
None of those factors go into the scoring fwiw. They are just informational. The scoring is done to a rubric, like a teacher would grade an essay, on various c
by sam-paech 1y ago
None of those factors go into the scoring fwiw. They are just informational.
The scoring is done to a rubric, like a teacher would grade an essay, on various criteria for good & bad writing.
- miltonlost 1y agoNone of those factors go into the scoring? What's the point of showing them then? What information are these factors providing if they aren't being used in the rubric? Without knowing the rubric (which isn't provided), these scores are baseless to me, and I don't know how to use them. Blackbox numbers are not useful.
- sam-paech 1y agoAll the judge outputs (including rubric) and model outputs are in the samples reports. Sorry you don't like the displayed metrics. I find them very useful / revealing of the things I'm trying to measure with this benchmark.
- Gracana 1y agoThese are important things to look at because they're where LLMs typically have trouble. People complain that their stories are too short, that they use stock phrases and purple prose, that they start to repeat themselves at a certain point, and that quality tends to fall off. Doing well in these problem areas doesn't guarantee good writing, but it does allow for it.