4 ms·

I can't imagine LLMs are good judges of good writing.

itishappy 1y ago
That's the core of my concern too. Be interested to see what happens if you feed the ranking algorithm a list of the most popular books and a list of the most impactful books. Something tells me this will be a lot more interested in Chuck Tingle than Kafka.
CuriouslyC 1y ago
LLMs are fairly good judges of writing, in fact they're better at evaluating writing than they are at actually writing. I use Gemini as a beta reader, and I've had a lot human beta readers look at the same material, and Gemini consistently gives significantly better than average feedback, though it's stronger at structural and prose evaluation and weaker at emotional and "wishlist" style feedback as you would probably expect.
- riskable 1y ago
  How are you using it as a beta reader? What prompts do you use? I'd love to try it.
  - CuriouslyC 1y ago
    Just dump your manuscript into google's aistudio, and tell it you'd like it to serve as a beta reader/editor, and tell it what your objectives are with your manuscript so it can give you targeted feedback.
riskable 1y ago
I've pasted whole chapters (of my own writing) into ChatGPT and Claude that I know need drastic improvements. Basically, they were first draft, "get the concept down; don't think too hard" paragraphs with occasional run-on sentences and whatnot. This is my very first novel (ever) so of course the initial draft is going to be bad. Both ChatGPT and Claud always say something like, "a few grammar corrections are needed but this is excellent!" So yeah: They're not very good at judging the quality of writing. Even with the "we're trying not to be sycophants anymore" improvements they're still sycophants. For reference, I mostly use these tools to check my grammar. That's something they're actually quite good at. It wasn't until the first draft was done that I decided to try them out for "whole work evaluation".
- vidarh 1y ago
  That sounds at least partly like a prompting issue to me. I have no problem getting scathing critiques out of either, by defining the role I want them to take clearly. Here's part of an initial criticism Claude made of your comment (it also said nice things): "However, the prose suffers from structural inconsistencies. The opening sentence contains an awkward parenthetical insertion that disrupts flow, and the second sentence uses unclear pronoun reference with "This" and "they." The rhythm varies unpredictably between crisp, direct statements and meandering explanations. "The vocabulary choices are generally precise—"sycophants" is particularly apt and memorable—though some phrases like "get the concept down; don't think too hard" feel slightly clunky in their construction." This was the prompt I used: "Imagine you're a literary critic. Critique the following comment based on use of language and effectiveness of communication only. Don't critique the argument itself:" followed by your comment. "Image you're a ..." or "Act as a ..." tends to make a huge difference in the kind of output you get. If you put it in the role of a critic that people expect to be tough, you're less likely to get sycophantic responses, at least in my experience. (If you want to see it get brutal, follow up the first response with a "be harsher" - it got unpleasantly savage)
  - suddenlybananas 1y ago
    Those critiques are nonsensical; the referents of "this" and "they" are completely obvious in the comment and the "clunky" construction is completely fine. There's also no "this" in the second sentence?
    - riskable 1y ago
      Exactly... What Claude output is what the prompt asked for: Critique. Even if there was nothing wrong at all Claude would still find something to critique—even if it has to hallucinate something wrong. That's it's job: To generate critique. It's not really doing any sort of "deep analysis" it's just reading the text and comparing it to what it knows about similar texts from its model. It'll then predict/generate the next word based on prior critiques of similar writings. It doesn't really have an understanding of the text at all.

That's the core of my concern too. Be interested to see what happens if you feed the ranking algorithm a list of the most popular books and a list of the most impactful books. Something tells me this will be a lot more interested in Chuck Tingle than Kafka.

CuriouslyC 1y ago

LLMs are fairly good judges of writing, in fact they're better at evaluating writing than they are at actually writing. I use Gemini as a beta reader, and I've had a lot human beta readers look at the same material, and Gemini consistently gives significantly better than average feedback, though it's stronger at structural and prose evaluation and weaker at emotional and "wishlist" style feedback as you would probably expect.

riskable 1y ago
How are you using it as a beta reader? What prompts do you use? I'd love to try it.
- CuriouslyC 1y ago
  Just dump your manuscript into google's aistudio, and tell it you'd like it to serve as a beta reader/editor, and tell it what your objectives are with your manuscript so it can give you targeted feedback.

riskable 1y ago

I've pasted whole chapters (of my own writing) into ChatGPT and Claude that I know need drastic improvements. Basically, they were first draft, "get the concept down; don't think too hard" paragraphs with occasional run-on sentences and whatnot. This is my very first novel (ever) so of course the initial draft is going to be bad. Both ChatGPT and Claud always say something like, "a few grammar corrections are needed but this is excellent!" So yeah: They're not very good at judging the quality of writing. Even with the "we're trying not to be sycophants anymore" improvements they're still sycophants. For reference, I mostly use these tools to check my grammar. That's something they're actually quite good at. It wasn't until the first draft was done that I decided to try them out for "whole work evaluation".

vidarh 1y ago
That sounds at least partly like a prompting issue to me. I have no problem getting scathing critiques out of either, by defining the role I want them to take clearly. Here's part of an initial criticism Claude made of your comment (it also said nice things): "However, the prose suffers from structural inconsistencies. The opening sentence contains an awkward parenthetical insertion that disrupts flow, and the second sentence uses unclear pronoun reference with "This" and "they." The rhythm varies unpredictably between crisp, direct statements and meandering explanations. "The vocabulary choices are generally precise—"sycophants" is particularly apt and memorable—though some phrases like "get the concept down; don't think too hard" feel slightly clunky in their construction." This was the prompt I used: "Imagine you're a literary critic. Critique the following comment based on use of language and effectiveness of communication only. Don't critique the argument itself:" followed by your comment. "Image you're a ..." or "Act as a ..." tends to make a huge difference in the kind of output you get. If you put it in the role of a critic that people expect to be tough, you're less likely to get sycophantic responses, at least in my experience. (If you want to see it get brutal, follow up the first response with a "be harsher" - it got unpleasantly savage)
- suddenlybananas 1y ago
  Those critiques are nonsensical; the referents of "this" and "they" are completely obvious in the comment and the "clunky" construction is completely fine. There's also no "this" in the second sentence?
  - riskable 1y ago
    Exactly... What Claude output is what the prompt asked for: Critique. Even if there was nothing wrong at all Claude would still find something to critique—even if it has to hallucinate something wrong. That's it's job: To generate critique. It's not really doing any sort of "deep analysis" it's just reading the text and comparing it to what it knows about similar texts from its model. It'll then predict/generate the next word based on prior critiques of similar writings. It doesn't really have an understanding of the text at all.