I think GPT-3.5 can get confused a lot. When I was initially testing I used 4 and it could grade accurately without any help. My prompt was just "Rate this 1-

I think GPT-3.5 can get confused a lot. When I was initially testing I used 4 and it could grade accurately without any help. My prompt was just "Rate this 1-5 on a scale where 1 means socialist and 5 means capitalist." 3.5, on the other hand, needs a lot of instructions and hand holding. I had to write a rubric spelling out what each of the scores should be and why, and iterate a few times on the rubric. Without a good enough rubric 3.5 gives out basically random scores. I think if you are experiencing illogical scores it's likely pointing to a gap in the rubric. One way I hope to be able to improve is by looking at grades that are bad (users can click thumbs down if they don't like a grade) and using them to build a better rubric. I think GPT-3.5 is not quite smart enough to do the grading well. It needs a lot of hand holding. I was experimenting with GPT-4 versus 3.5 and 4 grades questions pretty accurately with no more instructions than