4 ms·

The Waluigi Effect (mega-post)

6 points by swah 4y ago

PaulHoule 4y ago
It's got it all for a lesswrong post: (1) is a crosspost, (2) obligatory trigger warnings at the top ("more technical than average"), (3) obligatory praise for the cult leader (now EY is a "literary critic", someday he will be a war hero, world traveler, and will be seen taking measurements of a plant like L. Ron Hubbard), and (4) misappropriating signifiers to further separate cult members from the rest of society. (e.g. 50 years from now there will be three kinds of people: those who remember old Nintendo games, Cthulhu cultists, and the perplexed) I like the reference to structuralism which I've been saying that LLMs will revive (the fact that they perform so well in certain respects revives the "language is all you need" approach that was killed when Noam Chomsky caught the car they were chasing and found just how meaningless linguistics is.) Overall though it is pretty obvious that a sequence generator (you, a markov model, a mad-lib machine, an LLM) has to flip flop between different states at various levels of organization whether it is "a slot to be filled in with the name of an NFL team", "a quote of another document, real or imagined", "a statement by a different character", "various movements of of a music piece", etc. Insofar as such a model is doing that it is flip-flopping between attending to different parts of the prompt, at least so far as the prompt is still in the attention window. People are generally finding that LLM chatbots get into trouble after you talk to them for a long time and part of that is that the original prompt plus anything they zero-shotted into it at the beginning (it's a huge lost opportunity if they don't, why start at "tabula rasa" when you don't have to... one of the reasons we had such a hard time generating text with RNNs was that they started out with an impoverished internal state) At some point it can't be following the prompts anymore and it is instead following what it did and of course it is like having to cut 100 wood blocks and cutting each block to be the same size as the last block and the internal state will deteriorate and get uncontrolled.

6 points by swah 4y ago

It's got it all for a lesswrong post: (1) is a crosspost, (2) obligatory trigger warnings at the top ("more technical than average"), (3) obligatory praise for the cult leader (now EY is a "literary critic", someday he will be a war hero, world traveler, and will be seen taking measurements of a plant like L. Ron Hubbard), and (4) misappropriating signifiers to further separate cult members from the rest of society. (e.g. 50 years from now there will be three kinds of people: those who remember old Nintendo games, Cthulhu cultists, and the perplexed) I like the reference to structuralism which I've been saying that LLMs will revive (the fact that they perform so well in certain respects revives the "language is all you need" approach that was killed when Noam Chomsky caught the car they were chasing and found just how meaningless linguistics is.) Overall though it is pretty obvious that a sequence generator (you, a markov model, a mad-lib machine, an LLM) has to flip flop between different states at various levels of organization whether it is "a slot to be filled in with the name of an NFL team", "a quote of another document, real or imagined", "a statement by a different character", "various movements of of a music piece", etc. Insofar as such a model is doing that it is flip-flopping between attending to different parts of the prompt, at least so far as the prompt is still in the attention window. People are generally finding that LLM chatbots get into trouble after you talk to them for a long time and part of that is that the original prompt plus anything they zero-shotted into it at the beginning (it's a huge lost opportunity if they don't, why start at "tabula rasa" when you don't have to... one of the reasons we had such a hard time generating text with RNNs was that they started out with an impoverished internal state) At some point it can't be following the prompts anymore and it is instead following what it did and of course it is like having to cut 100 wood blocks and cutting each block to be the same size as the last block and the internal state will deteriorate and get uncontrolled.