That is too much data to parse for a simple website scrape. I do think Common Crawl has a lot of potential for people to use instead of scraping, but I think i

That is too much data to parse for a simple website scrape. I do think Common Crawl has a lot of potential for people to use instead of scraping, but I think its for larger projects. It gave me the idea to look at the links to ID if they are a business or non-business website