2 ms·

True, but a problem is that self-improving AI leads to a somewhat troubling mode of thinking. AIs switch to an internal babbling type language that makes no sen

by spwa4 2y ago

True, but a problem is that self-improving AI leads to a somewhat troubling mode of thinking. AIs switch to an internal babbling type language that makes no sense but clearly still conveys meaning to the AIs, then think in that language (if it's a language, though not sure what else it could be) and then produce correct results. Worse, when you use multiple agents to get AI LLMs talking to one another, all AI agents switch to this internal language and they make progress despite no human understanding what hell is happening. This seems very bad. Illustration: > How many r in strawberry? I'm asked how many r in strawberry. I can just spell the word and a;dklsjaw; a;ewjraqwpeouypaads;lq qepwiouryaqeopw qewrpoiuyoiauysdqw145124rfa.nkjlwh ;45a8345a894ya4a q4p58q45jaq;lkjas;dlfkja;j <answer>There are 3 (three) r's in strawberry</answer>

hnuser123456 2y ago
They are not going to invent a new language by themselves, they by definition can't even "think" in terms of languages they haven't seen. It does not occur to them that the language they use may be suboptimal. And surely, any better ways of thinking can still be described in English. It seems more likely there will be a gradual transition from us teaching LLMs how to reason, to LLMs being able to actually gobble and process enough data to learn more effective ways to reason, which it can then "teach" us. But that's just the LLM reflecting the way it was trained and aligned.
theptip 2y ago
I’ve heard this described as talking in “Neuralese”. It seems plausible that this will be the most dense language for model-internal dialog (or presumably inter-LLM dialog assuming they share the same weights). You will penalize this inasmuch as your alignment strategy depends on Deliberative Alignment. But at some point I assume that will come with a real capability cost as Neuralese can be more conceptually dense.

by spwa4 2y ago

hnuser123456 2y ago

They are not going to invent a new language by themselves, they by definition can't even "think" in terms of languages they haven't seen. It does not occur to them that the language they use may be suboptimal. And surely, any better ways of thinking can still be described in English. It seems more likely there will be a gradual transition from us teaching LLMs how to reason, to LLMs being able to actually gobble and process enough data to learn more effective ways to reason, which it can then "teach" us. But that's just the LLM reflecting the way it was trained and aligned.

theptip 2y ago

I’ve heard this described as talking in “Neuralese”. It seems plausible that this will be the most dense language for model-internal dialog (or presumably inter-LLM dialog assuming they share the same weights). You will penalize this inasmuch as your alignment strategy depends on Deliberative Alignment. But at some point I assume that will come with a real capability cost as Neuralese can be more conceptually dense.