35 ms·

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen commu

As many know and point out, this idea is now very old (40+ years?). The big problem is that it creates "word salad" [0]. In the generative art and procgen community, the term "10,000 bowls of oatmeal" problem has been used [1]. Taking a step back, this is a perennial problem, even with AI old and new. I've heard that the early (good) chess bots, after Deep Blue, had a problem of only being locally context sensitive and being easily fooled by longer range attack planning. We even see it, to a certain extent, in LLMs where they forget or contradict themselves with what was just said. I have no doubt that LLMs will just keep getting better but, as a snapshot of right now, you can see shades of this "word salad" problem, just a few steps removed to become a "logic salad" problem. [0] https://en.wikipedia.org/wiki/Word_salad https://en.wikipedia.org/wiki/Word_salad [1] https://galaxykate0.tumblr.com/post/139774965871/so-you-want-to-build-a-generator https://galaxykate0.tumblr.com/post/139774965871/so-you-want...

tgv 3y ago
Also: it works much better in English than in other languages I know. English grammar is simple, and many of the function words carry a strong prediction for the next tokens.
- echelon 3y ago
  Which languages are harder for LLMs? Is there any writeup or analysis of this?
  - tahnyall 3y ago
    If I had to guess, from my time working with ASR (Advanced Speech Recognition) languages and its difficulty with the Indian (country of) accent; I'd guess Indian languages.
    - java_beyb 3y ago
      what's advanced speech recognition?
  - justsomehnguy 3y ago
    Any with inflections and particularly with grammatical gender.
  - tgv 3y ago
    Idk. First, the obvious candidates: those with small, low quality text corpora. Second, I expect LLMs to perform less on long distance dependencies. English has a pretty strict word order, and the constituents are close to each other. There's also a lot of input available, so there are more large gaps to learn from. But (some) Germanic languages have some weird rules which allows constituents to move far away, so it's likely LLMs will make more errors there. Also, free word order languages might be a problem, although I've always suspected that those languages aren't that free in reality: speakers probably have strong preferences.
3abiton 3y ago
Any suggestiob on the background of the method?
- tempodox 3y ago
  Do a web search for “Markov chain”.
tempodox 3y ago
And the logic salad problem will not be solved by LLMs because there is nothing in their construction that makes them capable of understanding logic.
- deleted 3y ago
  [deleted]
- moon_moon 3y ago
  [dead]
- brookst 3y ago
  Can you elaborate? I would argue that logic has even stronger token prediction characteristics than language. As a total handwave, I would expect an LLM trained on formal logic and a huge corpus of proofs to produce pretty strong logic output.
YeGoblynQueenne 3y ago
The output is "word salad" because the probabilities of the model are uniform, albeit implicitly- eyballing the python, it selects the next n-gram uniformly at random. A better n-gram model would calculate the probability of an n-gram following another n-gram according to their frequency in the training corpus. With a larger corpus and better training techniques (some smoothing for one thing) you'd get much more coherent output. And of course, if you trained a gigantic model on the entire web, then you'd get a very smooth approximation of the text in the corpus and very fluent, grammatical output. So basically, an LLM. >> As many know and point out, this idea is now very old (40+ years?). More like 110 years. Markov Chains were proposed in 1913 (by Markov) and popularised by Shannon in 1948 (in "A Mathematical Theory of Communication", the work that introduced information theory): The underlying mathematics of the n-gram was first proposed by Markov (1913), who used what are now called Markov chains (bigrams and trigrams) to predict whether an upcoming letter in Pushkin’s Eugene Onegin would be a vowel or a con- sonant. Markov classified 20,000 letters as V or C and computed the bigram and trigram probability that a given letter would be a vowel given the previous one or two letters. Shannon (1948) applied n-grams to compute approximations to English word sequences. Based on Shannon’s work, Markov models were commonly used in engineering, linguistic, and psychological work on modeling word sequences by the 1950s. In a series of extremely influential papers starting with Chomsky (1956) and including Chomsky (1957) and Miller and Chomsky (1963), Noam Chomsky argued that “finite-state Markov processes”, while a possibly useful engineering heuristic, were incapable of being a complete cognitive model of human grammatical knowl- edge. These arguments led many linguists and computational linguists to ignore work in statistical modeling for decades. https://web.archive.org/web/20220522005827/https://web.stanford.edu/~jurafsky/slp3/3.pdf https://web.archive.org/web/20220522005827/https://web.stanf... (Note the usual shaking of angry fists at Chomsky. The NLP community continued on its path anyway, and built smooth, fluent generators of total bullshit and absolutely no progress towards a "cognitive model of human grammatical knowledge").
- shawnz 3y ago
  > A better n-gram model would calculate the probability of an n-gram following another n-gram according to their frequency in the training corpus. With a larger corpus and better training techniques (some smoothing for one thing) you'd get much more coherent output. I don't understand what you mean here, isn't that exactly what an n-gram model is already designed to achieve where n > 1? Since this is a 2-gram model isn't it already doing this?
doubloon 3y ago
i put Alexis De Tocqueville through this and the results were legible but hard to read. Then I put a demagogue politician speech through this and the results were almost like a speech that politician would have given. It was hard to tell the difference between the original and the generated. in other words, if the public admires word salad, and someone speaks in word salad, then word salad output would not be considered as problematic by a large number of people.
- chuckadams 3y ago
  I recall one site way back that used a Markov chain generator to mash up Karl Marx and Ayn Rand. Was fairly plausible reading, actually.
  - nurettin 3y ago
    I imagine that's like mixing different kinds of salad.
    - chuckadams 3y ago
      It definitely gives you some whiplash, but most of the time it's just righteous ranting about ... whatever. The whole site was a trove of comedy, featuring quizzes like "Porn Star or My Little Pony?". I couldn't remember its name until I started typing this reply: lileks.com.
- hermitcrab 3y ago
  I did a markov reworking of some of Trump's speeches and got: "Hillary brought death and disaster to Iraq, Syria and Libya, she empowered Iran, and she unleashed ISIS. Now she wants to raise your taxes very substantially. Highest taxed nation in the world is a tenant of mine in Manhattan, so many great people. These are people that have been stolen, stolen by either very stupid politicians ask me the question, how are you going to get rid of all the emails?” “Yes, ma’am, they’re gonna stay in this country blind. My contract with the American voter begins with a plan to end government that will not protect its people is a government corruption at the State Department of Justice is trying as hard as they can to protect religious liberty" Details at: https://successfulsoftware.net/2019/04/02/bloviate/ https://successfulsoftware.net/2019/04/02/bloviate/

tgv 3y ago

Also: it works much better in English than in other languages I know. English grammar is simple, and many of the function words carry a strong prediction for the next tokens.

echelon 3y ago
Which languages are harder for LLMs? Is there any writeup or analysis of this?
- tahnyall 3y ago
  If I had to guess, from my time working with ASR (Advanced Speech Recognition) languages and its difficulty with the Indian (country of) accent; I'd guess Indian languages.
  - java_beyb 3y ago
    what's advanced speech recognition?
- justsomehnguy 3y ago
  Any with inflections and particularly with grammatical gender.
- tgv 3y ago
  Idk. First, the obvious candidates: those with small, low quality text corpora. Second, I expect LLMs to perform less on long distance dependencies. English has a pretty strict word order, and the constituents are close to each other. There's also a lot of input available, so there are more large gaps to learn from. But (some) Germanic languages have some weird rules which allows constituents to move far away, so it's likely LLMs will make more errors there. Also, free word order languages might be a problem, although I've always suspected that those languages aren't that free in reality: speakers probably have strong preferences.

3abiton 3y ago

Any suggestiob on the background of the method?

tempodox 3y ago
Do a web search for “Markov chain”.

tempodox 3y ago

And the logic salad problem will not be solved by LLMs because there is nothing in their construction that makes them capable of understanding logic.

deleted 3y ago
[deleted]
moon_moon 3y ago
[dead]
brookst 3y ago
Can you elaborate? I would argue that logic has even stronger token prediction characteristics than language. As a total handwave, I would expect an LLM trained on formal logic and a huge corpus of proofs to produce pretty strong logic output.

YeGoblynQueenne 3y ago

The output is "word salad" because the probabilities of the model are uniform, albeit implicitly- eyballing the python, it selects the next n-gram uniformly at random. A better n-gram model would calculate the probability of an n-gram following another n-gram according to their frequency in the training corpus. With a larger corpus and better training techniques (some smoothing for one thing) you'd get much more coherent output. And of course, if you trained a gigantic model on the entire web, then you'd get a very smooth approximation of the text in the corpus and very fluent, grammatical output. So basically, an LLM. >> As many know and point out, this idea is now very old (40+ years?). More like 110 years. Markov Chains were proposed in 1913 (by Markov) and popularised by Shannon in 1948 (in "A Mathematical Theory of Communication", the work that introduced information theory): The underlying mathematics of the n-gram was first proposed by Markov (1913), who used what are now called Markov chains (bigrams and trigrams) to predict whether an upcoming letter in Pushkin’s Eugene Onegin would be a vowel or a con- sonant. Markov classified 20,000 letters as V or C and computed the bigram and trigram probability that a given letter would be a vowel given the previous one or two letters. Shannon (1948) applied n-grams to compute approximations to English word sequences. Based on Shannon’s work, Markov models were commonly used in engineering, linguistic, and psychological work on modeling word sequences by the 1950s. In a series of extremely influential papers starting with Chomsky (1956) and including Chomsky (1957) and Miller and Chomsky (1963), Noam Chomsky argued that “finite-state Markov processes”, while a possibly useful engineering heuristic, were incapable of being a complete cognitive model of human grammatical knowl- edge. These arguments led many linguists and computational linguists to ignore work in statistical modeling for decades. https://web.archive.org/web/20220522005827/https://web.stanford.edu/~jurafsky/slp3/3.pdf https://web.archive.org/web/20220522005827/https://web.stanf... (Note the usual shaking of angry fists at Chomsky. The NLP community continued on its path anyway, and built smooth, fluent generators of total bullshit and absolutely no progress towards a "cognitive model of human grammatical knowledge").

shawnz 3y ago
> A better n-gram model would calculate the probability of an n-gram following another n-gram according to their frequency in the training corpus. With a larger corpus and better training techniques (some smoothing for one thing) you'd get much more coherent output. I don't understand what you mean here, isn't that exactly what an n-gram model is already designed to achieve where n > 1? Since this is a 2-gram model isn't it already doing this?

doubloon 3y ago

i put Alexis De Tocqueville through this and the results were legible but hard to read. Then I put a demagogue politician speech through this and the results were almost like a speech that politician would have given. It was hard to tell the difference between the original and the generated. in other words, if the public admires word salad, and someone speaks in word salad, then word salad output would not be considered as problematic by a large number of people.

chuckadams 3y ago
I recall one site way back that used a Markov chain generator to mash up Karl Marx and Ayn Rand. Was fairly plausible reading, actually.
- nurettin 3y ago
  I imagine that's like mixing different kinds of salad.
  - chuckadams 3y ago
    It definitely gives you some whiplash, but most of the time it's just righteous ranting about ... whatever. The whole site was a trove of comedy, featuring quizzes like "Porn Star or My Little Pony?". I couldn't remember its name until I started typing this reply: lileks.com.
hermitcrab 3y ago
I did a markov reworking of some of Trump's speeches and got: "Hillary brought death and disaster to Iraq, Syria and Libya, she empowered Iran, and she unleashed ISIS. Now she wants to raise your taxes very substantially. Highest taxed nation in the world is a tenant of mine in Manhattan, so many great people. These are people that have been stolen, stolen by either very stupid politicians ask me the question, how are you going to get rid of all the emails?” “Yes, ma’am, they’re gonna stay in this country blind. My contract with the American voter begins with a plan to end government that will not protect its people is a government corruption at the State Department of Justice is trying as hard as they can to protect religious liberty" Details at: https://successfulsoftware.net/2019/04/02/bloviate/ https://successfulsoftware.net/2019/04/02/bloviate/