Honestly, some sites are so ridiculously malconfigured in their anti-bot zeal that it becomes a Heisenberg like dilemma. E.g. I want to pull in the rss. It is

Honestly, some sites are so ridiculously malconfigured in their anti-bot zeal that it becomes a Heisenberg like dilemma. E.g. I want to pull in the rss. It is there specifically for m2m. If I dare get the robots.txt, i'm flagged as a bot, and denied the whole site. including not just the rss but even the parts that are not denied per the robots.txt