4 ms·

If it’s not easily verifiable, LLMs aren’t good at it.

jeremyjh 5mo ago
I think that’s mostly because they get so much more of that reinforcement learning - since it is so economical. I dont know if there is any evidence of a fundamental reason they can’t be just as good at other tasks, but it might be economically infeasible for awhile yet.
- mjburgess 5mo ago
  No one is curating vast amounts of data for them in other domains. Programmers send programs with fixes
  - knollimar 5mo ago
    There's no diff of my excel lambdas being fixed? :(
  - jeremyjh 5mo ago
    Its more about how costly it is to verify work in reinforcement learning. It is cheap in Mathematics and coding because it can be automated. It is expensive in other domains because while you can capture certain datasets to do pre-training on, you ultimately need humans in the loop to judge the quality of work.
- emp17344 5mo ago
  RLVR doesn’t work for unverifiable tasks, so they won’t be able to effectively use tools to boost reliability for those tasks.
  - jeremyjh 5mo ago
    Right, so you have to use RLHF. That is the economics problem I was referring to.
dominotw 5mo ago
but what does it mean to be good at something that cant be verified. how do you know that they are not good at it, you are obviously using some measure. sounds like an oxymoron of a claim.
- maxbond 5mo ago
  It means having taste. People say Picasso was a great painter, but that cannot be verified (at least, not in the sense of a verified reward).
  - dominotw 5mo ago
    "people say picasso was a great painter" is definitely not hard to verify . lol.
    - deleted 5mo ago
      [deleted]
    - maxbond 5mo ago
      I don't know if you're being factitious or not but that was not what I meant. Picasso being a great painter is an example of "having taste"; "create an artistic image generation model with Picasso-level performance" is a valid problem statement we could attack with RLHF, but not with RLVR, because "taste" is not amenable to modeling with a reward function. "Write this code in a way that is readable and maintainable" is another example.
      - dominotw 5mo ago
        https://futurism.com/artificial-intelligence/real-monet-ai-chaos https://futurism.com/artificial-intelligence/real-monet-ai-c...
        deleted 5mo ago
        [deleted]
        5mo ago

I think that’s mostly because they get so much more of that reinforcement learning - since it is so economical. I dont know if there is any evidence of a fundamental reason they can’t be just as good at other tasks, but it might be economically infeasible for awhile yet.

mjburgess 5mo ago
No one is curating vast amounts of data for them in other domains. Programmers send programs with fixes
- knollimar 5mo ago
  There's no diff of my excel lambdas being fixed? :(
- jeremyjh 5mo ago
  Its more about how costly it is to verify work in reinforcement learning. It is cheap in Mathematics and coding because it can be automated. It is expensive in other domains because while you can capture certain datasets to do pre-training on, you ultimately need humans in the loop to judge the quality of work.
emp17344 5mo ago
RLVR doesn’t work for unverifiable tasks, so they won’t be able to effectively use tools to boost reliability for those tasks.
- jeremyjh 5mo ago
  Right, so you have to use RLHF. That is the economics problem I was referring to.

dominotw 5mo ago

but what does it mean to be good at something that cant be verified. how do you know that they are not good at it, you are obviously using some measure. sounds like an oxymoron of a claim.

maxbond 5mo ago
It means having taste. People say Picasso was a great painter, but that cannot be verified (at least, not in the sense of a verified reward).
- dominotw 5mo ago
  "people say picasso was a great painter" is definitely not hard to verify . lol.
  - deleted 5mo ago
    [deleted]
  - maxbond 5mo ago
    I don't know if you're being factitious or not but that was not what I meant. Picasso being a great painter is an example of "having taste"; "create an artistic image generation model with Picasso-level performance" is a valid problem statement we could attack with RLHF, but not with RLVR, because "taste" is not amenable to modeling with a reward function. "Write this code in a way that is readable and maintainable" is another example.
    - dominotw 5mo ago
      https://futurism.com/artificial-intelligence/real-monet-ai-chaos https://futurism.com/artificial-intelligence/real-monet-ai-c...
      - deleted 5mo ago
        [deleted]
      - 5mo ago