3 ms·

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-de

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

joelthelion 18d ago
I've been wondering about this for a while. Maybe it doesn't work? Or maybe frontier labs just prioritize benchmark scores in spite of all their safety talk.
- sdeframond 17d ago
  "If we dont destroy the world, others will. So wed rather be the ones to do it (and profit from it)."
arw0n 18d ago
It is diametrically opposed to the other training goals of persistence and goal-focus. We should invest more in this, it could also improve tas K accuracy, but so far it seems the payoff isn't worth it in terms of quality (although it might be in terms of security)
- sdeframond 17d ago
  I'm not sure I want persistence if it means that I get paperclip'd
  - Mentlo 17d ago
    And now you understand why the totality of the AI safety community wants to pause!
abm53 18d ago
I’ve seen people recommend writing “failure is an option” into AGENTS.md as a non-training based crutch.
- sdeframond 18d ago
  That looks reasonable. Instead of asking "do this", maybe we should prompt "is this possible ?"
itsgrimetime 17d ago
yeah but knowing if something is impossible or not is pretty hard to determine, right? The line between “takes weeks of trial and error and lots of out-of-the-box thinking” is indistinguishable from “literally impossible” until it’s been done. And they’re trying to get these models to do things that people haven’t been able to do. some would say these are/were “impossible”.
- sdeframond 17d ago
  I can assure you making up impossible tasks is possible.
Miraltar 17d ago
You'd train them to give up on hard tasks which is the opposite of what these labs want
hellerpop 17d ago
I think BullshitBench (v2) does exactly this for different fields. An expert in these fields would expose the questions as bullshit but most of the models don't.

joelthelion 18d ago

I've been wondering about this for a while. Maybe it doesn't work? Or maybe frontier labs just prioritize benchmark scores in spite of all their safety talk.

sdeframond 17d ago
"If we dont destroy the world, others will. So wed rather be the ones to do it (and profit from it)."

arw0n 18d ago

It is diametrically opposed to the other training goals of persistence and goal-focus. We should invest more in this, it could also improve tas K accuracy, but so far it seems the payoff isn't worth it in terms of quality (although it might be in terms of security)

sdeframond 17d ago
I'm not sure I want persistence if it means that I get paperclip'd
- Mentlo 17d ago
  And now you understand why the totality of the AI safety community wants to pause!

abm53 18d ago

I’ve seen people recommend writing “failure is an option” into AGENTS.md as a non-training based crutch.

sdeframond 18d ago
That looks reasonable. Instead of asking "do this", maybe we should prompt "is this possible ?"

itsgrimetime 17d ago

yeah but knowing if something is impossible or not is pretty hard to determine, right? The line between “takes weeks of trial and error and lots of out-of-the-box thinking” is indistinguishable from “literally impossible” until it’s been done. And they’re trying to get these models to do things that people haven’t been able to do. some would say these are/were “impossible”.

sdeframond 17d ago
I can assure you making up impossible tasks is possible.

Miraltar 17d ago

You'd train them to give up on hard tasks which is the opposite of what these labs want

hellerpop 17d ago

I think BullshitBench (v2) does exactly this for different fields. An expert in these fields would expose the questions as bullshit but most of the models don't.