Unfortunately, from what i've learnt from HN that this is not really possible. Most websites ban crawlers\bots with the exception of those run by google and may

Unfortunately, from what i've learnt from HN that this is not really possible. Most websites ban crawlers\bots with the exception of those run by google and maybe bing. It seems that this market is hermetically sealed.