4 ms·
News websites are asking if I'm a robot instead of showing the headlines.Is it common? https://postimg.cc/47xNhT1g https://postimg.cc/47xNhT1g
by harish5p 5y ago
News websites are asking if I'm a robot instead of showing the headlines.Is it common? https://postimg.cc/47xNhT1g https://postimg.cc/47xNhT1g
- deleted 5y ago[deleted]
- lonesword 5y agoUnfortunately yes. They see too many requests from paperdelivery.co and block requests originating from that domain - they only want the google bot to crawl their site :) I think I have two options: 1. Respect their decision to not allow PaperDelivery to fetch that page. 2. Use a set of proxies and headless browsers or outright impersonate the Google bot to trick the news websites into allowing PaperDelivery to fetch that page.