I did a large-scale crawl of the web some years ago and we put together a billion pages. The bulk of the content ended up being index pages - i.e. large list o

I did a large-scale crawl of the web some years ago and we put together a billion pages. The bulk of the content ended up being index pages - i.e. large list of links taking you to the content - pagination, other breadcrumbs etc. You can exhaust a lot of resources without getting anything useful. It is no longer a point and go kind of thing unfortunately, you need a good understanding of page structure, estimate what kinds of links are vital, etc. else there's a ton of crap you'll pick up. Or maybe I was doing something wrong.