It's a bit rude to assume I am incompetent. I am sorry, but you vastly underestimate the problem. Just read the other comments and see how many corner cases ar

It's a bit rude to assume I am incompetent. I am sorry, but you vastly underestimate the problem. Just read the other comments and see how many corner cases are there. It took me 3 days to implement a crawler than says OK on 200, and ERROR on 404, and 6 months to deal with exceptions to the rule, and apparently thanks to the feedback Ive received, there are many other exceptions to deal with. Sorry, this is a bit like that old Dropbox comment. The thing is, if your website is easily covered by a 50-line Python script, you do not need Bernard.