You could try latent semantic analysis of new pages to determine similarity to other documents in your domain. I've been doing a lot of web crawling recently,

You could try latent semantic analysis of new pages to determine similarity to other documents in your domain. I've been doing a lot of web crawling recently, feel free to get in touch if you'd like to discuss further.