3 ms·

> AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave.

> AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. AI2027 predicts a future in which LLM performance will increase once we find alternatives to thinking in "human language". At least the video gave me that impression and I think this is what "neuralese" is referring to. Is that a credible prediction?

supriyo-biswas 1y ago
There’s the coconut paper by meta, which kind of does a similar thing by reasoning in the latent space instead of doing it in tokens: https://arxiv.org/abs/2412.06769 https://arxiv.org/abs/2412.06769 Given that anthropic’s interpretability work finds that CoT does not reliably predict the model’s internal reasoning process, I think approaches like the one above are more likely to succeed.
TeMPOraL 1y ago
We also have a lot of past data to inform us on what to expect - our history is full of attempts to suppress or eliminate all kinds of "wrongthink", and all they ever did was to make people communicate their "wrongthink" indirectly while under surveillance.
rtkwe 1y ago
Maybe, but I don't know how we get there from LLMs. How do you go from the token prediction models they represent to something that doesn't just use the language it was trained on?
ben_w 1y ago
> Is that a credible prediction? That they will use it? Yes, models already do. That it will increase performance? Unclear, I have no reason to expect that, not impossible. That it will make it harder to tell what they're up to? Definitely. Even between humans, terms of art often confuse, e.g. "Mike's Nature trick" and "to hide the decline" being colossally misunderstood in https://en.wikipedia.org/wiki/Climatic_Research_Unit_email_controversy https://en.wikipedia.org/wiki/Climatic_Research_Unit_email_c...

supriyo-biswas 1y ago

There’s the coconut paper by meta, which kind of does a similar thing by reasoning in the latent space instead of doing it in tokens: https://arxiv.org/abs/2412.06769 https://arxiv.org/abs/2412.06769 Given that anthropic’s interpretability work finds that CoT does not reliably predict the model’s internal reasoning process, I think approaches like the one above are more likely to succeed.

TeMPOraL 1y ago

We also have a lot of past data to inform us on what to expect - our history is full of attempts to suppress or eliminate all kinds of "wrongthink", and all they ever did was to make people communicate their "wrongthink" indirectly while under surveillance.

rtkwe 1y ago

Maybe, but I don't know how we get there from LLMs. How do you go from the token prediction models they represent to something that doesn't just use the language it was trained on?

ben_w 1y ago

> Is that a credible prediction? That they will use it? Yes, models already do. That it will increase performance? Unclear, I have no reason to expect that, not impossible. That it will make it harder to tell what they're up to? Definitely. Even between humans, terms of art often confuse, e.g. "Mike's Nature trick" and "to hide the decline" being colossally misunderstood in https://en.wikipedia.org/wiki/Climatic_Research_Unit_email_controversy https://en.wikipedia.org/wiki/Climatic_Research_Unit_email_c...