The advantage of using well known corpora is that they are also 'tagged' so later on you can extend your text analysis. Text analysis is addictive, the more you

The advantage of using well known corpora is that they are also 'tagged' so later on you can extend your text analysis. Text analysis is addictive, the more you do the more you want. Another option is the British National Corpus at probably around twenty pounds for the cd, with over a million words. The latter is also available in XML. I had a similar situation about two years ago. I did some research on blogs. As a sideline the word 'blogs' does not even appears once in the British National Corpus! This put an end to me comparing against known corpora. Quick and dirty solution. Write your own routines. At the time I used PHP! Python is a better option but my Python skills were limited. I spidered blogs (with cURL). If the word 'blogger' 'blogroll' or 'comments' appeared I spidered further. Once I had 20,000 pages I started calculations. As you add to the corpus your statistics change, so you need to find an efficient method of calculating all these. With PHP I was able to download on average one page per 20 seconds. Your stop words need to also evolve as you go on, things like RSS, will crop up all the time, you would also think that 'wordpress' is one of the most common words in English! This led me to the next stage polishing up some form of pre-processor for cleaning up HTML. (Most Classes out there are not adequate, they tend to leave items in such as CSS style etc..). I used tidy believe it or not, very effectively for quick pre-processing. (Helped a lot to just identify only the <body>). Other adventures along the way was downloading books from the Gutenberg Organization and using them as corpora.