>There are a LOT more file-extensions that you'll want to ignore. It'd probably be a good idea just to whitelist filetypes that you actually want to crawl, rat

>There are a LOT more file-extensions that you'll want to ignore. It'd probably be a good idea just to whitelist filetypes that you actually want to crawl, rather than trying to blacklist all the ones you don't.