If you have a URL and want to find all the pages you can there's only so much you can do (without considering external links). 1. Crawl all the links 2. Check

If you have a URL and want to find all the pages you can there's only so much you can do (without considering external links). 1. Crawl all the links 2. Check for directory listings on all path combinations (those discovered in href's and src's) 3. Check robots.txt for any other discoverable pages 4. Brute force expected or possible URLs 5. Try and parse more links from any javascript (probably hard) That's all I can think of.