You might want to look at the YC-backed company Semantics3, which has indexed 60 million unique products and over 4 billion URLs... all their data is available

You might want to look at the YC-backed company Semantics3, which has indexed 60 million unique products and over 4 billion URLs... all their data is available as APIs with pricing proportional to the number of API calls: https://www.semantics3.com/ https://www.semantics3.com/