3 ms·
Here's a Python program that builds a markov chain model of a corpus (with laplace smoothing) and then calculates the probability of the domain name being gener
by jules 14y ago
Here's a Python program that builds a markov chain model of a corpus (with laplace smoothing) and then calculates the probability of the domain name being generated by that markov chain.
from collections import Counter
from math import log
corpus = """training corpus here"""
n = 1 # token size
def trans(w):
return ((w[i:i+n], w[i+n]) for i in range(len(w)-n))
tokens = Counter(t for t,l in trans(corpus))
transitions = Counter(trans(corpus))
def score(w):
return sum(log(transitions[t,l]+1)-log(tokens[t]+26**n) for t,l in trans(w))
for w in [" llmyw "," domyh "," tretz "," qenis "," debts "]:
print w, ': ', score(w)
When I use the European sovereign-debt crisis Wikipedia article as the corpus, I get
llmyw : -25.9316238289
domyh : -23.311005645
tretz : -21.4220068707
qenis : -20.7421233042
debts : -15.3287127006
Higher numbers are considered more pronounceable, so that's in order of pronounceability. Note that scores of words of different length are incomparable.
The variable `n` sets the number of letters of memory the markov chain has. If you set `n` higher then you need a bigger corpus. If you set n=2 and get a larger corpus and preprocess it to filter out anything that isn't [a-z]+ it would probably work fairly well (I just copied the article in as is).