I never got around to writing a job for that. Now you've got me wondering. I counted each distinct word as a concept, but only from the sentences that matched t

I never got around to writing a job for that. Now you've got me wondering. I counted each distinct word as a concept, but only from the sentences that matched the regex patterns. The 4.4MM concepts, however, includes multi-word concepts (as in 'be part of'). The number of distinct concepts in the corpus is much higher. Edit: I just went home and started a map reduce job to find the total distinct tokens. I should be able to post the results within a few hours. Unsupervised and language-independent, now that's something worth working towards. What data (or how much data) are you working with right now? I plan on selling n-gram datasets from the Web, so I would like to have another set of eyes look over them. If you'd like, shoot me an email at james[at]webscaled[dot]com and I'll send you a copy to experiment with.