5 ms·
The article mentions looking at randomly generated words and suggests that the power law being verified in that case may have to do with the generated lengths.
by lgeorget 4y ago
The article mentions looking at randomly generated words and suggests that the power law being verified in that case may have to do with the generated lengths.
Has someone tried to generate random words whose length follow the distribution of natural languages?
- wodenokoto 4y ago> Has someone tried to generate random words whose length follow the distribution of natural languages? Yes, this is an exercise in chapter one of Chris Manning et. al. "Foundations of Statistical NLP" (MIT-1999). You can do it in almost 2 lines of very code golfed python. 5 if you are pedantic and count the imports (and a million if you count the lines in matplotlib). import random from collections import Counter import matplotlib.pyplot as plt corpus = Counter(''.join(["qwertyuiopasdfghjklzxcvbnm "[random.randint(0,26)] for i in range(100_00_000)]).split(' ')[:-1]) plt.loglog(range(len(corpus)-1), [el[1] for el in corpus.most_common()], '.') I remove the last generated word as it is not terminated by a space (I'm a bit unsure what the statistical implication is if the last generated character is a space) I do not get anywhere near the power distribution that the text book suggest I will get.
- flir 4y agoYou can do that with Markov chains. Just letter-by-letter instead of word-by-word.
- deleted 4y ago[deleted]