A lot of these problem apply to crawling in general. I run adstxt.com which crawls ads.txt files. Some issues we have seen: - Pages that say they are one enco

A lot of these problem apply to crawling in general. I run adstxt.com which crawls ads.txt files. Some issues we have seen: - Pages that say they are one encoding but are actually another - Redirects to other sites - invalid contents or malformed content. How hard should we try to interpret what they meant? For example, an html page that looks like a valid CSV. - Page too long or extremely slow - Invalid certs