2 ms·
> Has someone tried to generate random words whose length follow the distribution of natural languages? Yes, this is an exercise in chapter one of Chris Mannin
by wodenokoto 4y ago
> Has someone tried to generate random words whose length follow the distribution of natural languages?
Yes, this is an exercise in chapter one of Chris Manning et. al. "Foundations of Statistical NLP" (MIT-1999).
You can do it in almost 2 lines of very code golfed python. 5 if you are pedantic and count the imports (and a million if you count the lines in matplotlib).
import random
from collections import Counter
import matplotlib.pyplot as plt
corpus = Counter(''.join(["qwertyuiopasdfghjklzxcvbnm "[random.randint(0,26)] for i in range(100_00_000)]).split(' ')[:-1])
plt.loglog(range(len(corpus)-1), [el[1] for el in corpus.most_common()], '.')
I remove the last generated word as it is not terminated by a space (I'm a bit unsure what the statistical implication is if the last generated character is a space)
I do not get anywhere near the power distribution that the text book suggest I will get.