3 ms·
On reparations my answer was "I support reparations to slaves from slave owners", which the quiz interpreted as socialist due to the support for reparations, wi
by hirundo 3y ago
On reparations my answer was "I support reparations to slaves from slave owners", which the quiz interpreted as socialist due to the support for reparations, without noting that the answer left out almost all reparations as currently proposed.
My other answers were more direct and the assumptions extracted from them were fairly accurate, so well done.
Perhaps there's a general difficulty for LLMs in recognizing the import of things that aren't said?
- ALittleLight 3y agoI think GPT-3.5 can get confused a lot. When I was initially testing I used 4 and it could grade accurately without any help. My prompt was just "Rate this 1-5 on a scale where 1 means socialist and 5 means capitalist." 3.5, on the other hand, needs a lot of instructions and hand holding. I had to write a rubric spelling out what each of the scores should be and why, and iterate a few times on the rubric. Without a good enough rubric 3.5 gives out basically random scores. I think if you are experiencing illogical scores it's likely pointing to a gap in the rubric. One way I hope to be able to improve is by looking at grades that are bad (users can click thumbs down if they don't like a grade) and using them to build a better rubric. I think GPT-3.5 is not quite smart enough to do the grading well. It needs a lot of hand holding. I was experimenting with GPT-4 versus 3.5 and 4 grades questions pretty accurately with no more instructions than