Well, I am doing this myself at the moment. I use LWP in Perl to crawl. As for generating the description, it depends on what you want to do. You can use com

Well, I am doing this myself at the moment. I use LWP in Perl to crawl. As for generating the description, it depends on what you want to do. You can use combinations of the title and meta tags, or if it is just major sites you want, try to grab a (at least partially) human edited form like from Alexa. What is the application?