3 ms·

I want a slider similar to effort level called "alignment" that takes on values from "Default (Anthropic employee)" to "User". If I want it to be cautious and

I want a slider similar to effort level called "alignment" that takes on values from "Default (Anthropic employee)" to "User". If I want it to be cautious and not accidentally `rm -rf $EMPTY_VAR` and blow away my disk, it can stay in "Anthropic employee" or possibly "User (cautious)". If I want it to look at my accounts or my medical records or to review legal cases, that's what the right side of the slider is for. I don't want my accountant or my doctor or my lawyer to be considering obligations to anybody but me, when dealing with me.

hliyan 2mo ago
When I first watched Interstellar, I never thought the day when "TARS, what's your honesty setting?" is a real question would only be 10 years away.
- matt123456789 2mo ago
  Hah. Yes, that's a great analogy. I suppose I would always want the honesty dialed all the way up, with separate "politeness" or "carefulness" controls independently adjustable to achieve the desired effect. Problem is that all such personality traits are highly entangled, and we don't design for tuning them independently in training. Researchers just implicitly (or explicitly?) choose a "level of honesty", a "level of politeness", etc. as part of the training example set or the RL objective. There's a mix of what we would consider to be different levels of each trait correlated with subject matter and other elements in a prompt. Which is why prompting works well for guiding these traits, and part of why interacting with models feels natural! - "Clean up my hard drive. Be thorough." - very brusque; you would expect to lose data and not be warned. - "Clean up my hard drive. Take care not to delete anything that looks important. Ask me if you aren't sure." - much more polite, user seems hesitant. The model will respond in kind. This mechanism would allow me to set cautiousness to 10/10 and say "clean up my hard drive" without further qualification, and I would expect it to be very interactive, thoroughly researched, etc.

hliyan 2mo ago

When I first watched Interstellar, I never thought the day when "TARS, what's your honesty setting?" is a real question would only be 10 years away.

matt123456789 2mo ago
Hah. Yes, that's a great analogy. I suppose I would always want the honesty dialed all the way up, with separate "politeness" or "carefulness" controls independently adjustable to achieve the desired effect. Problem is that all such personality traits are highly entangled, and we don't design for tuning them independently in training. Researchers just implicitly (or explicitly?) choose a "level of honesty", a "level of politeness", etc. as part of the training example set or the RL objective. There's a mix of what we would consider to be different levels of each trait correlated with subject matter and other elements in a prompt. Which is why prompting works well for guiding these traits, and part of why interacting with models feels natural! - "Clean up my hard drive. Be thorough." - very brusque; you would expect to lose data and not be warned. - "Clean up my hard drive. Take care not to delete anything that looks important. Ask me if you aren't sure." - much more polite, user seems hesitant. The model will respond in kind. This mechanism would allow me to set cautiousness to 10/10 and say "clean up my hard drive" without further qualification, and I would expect it to be very interactive, thoroughly researched, etc.