3 ms·

It's possible he's randomly generated the docs or something. Which, I agree, is a bit lame. It's nice to get optical confirmation that what you're doing is corr

• bravura 16y ago
  No. Grandparent has picked out a few pathological examples out of many good ones. Here is a uniform, random sample of documents: faststretch details honourable judge oriental museum http london wc 1n 3 bg pdsg seminar 1857-8 indian mutiny vera bell katherine r.b rare books collection novy arbat street universitys case The data set is unique terms that occur in a crawl of .uk. Here is how I constructed the data set: I took the UKWAC web-as-corpus crawl (2 bil­lion words, crawled in 2008), ran it through the splitta sen­tence split­ter, removed all funny char­ac­ters, ran the Penn tree­bank word tok­enizer, and per­form term extrac­tion with topia.termextract, dis­card­ing terms that will sin­gle words. I then low­er­cased the terms, sorted them, and uniqued them, to give the dataset.