It is not crawling but indexing is the problem. Google has over years learned the patterns and authority of different articles. It will be hard for others to re

It is not crawling but indexing is the problem. Google has over years learned the patterns and authority of different articles. It will be hard for others to replicate but not impossible. What Google should do is offer API based access to these providers but a lot of these providers might no adhere by contracts. So there is that.