Isn't this by problem solved by using commoncrawl data. I wonder what changed to AI companies to do mass crawling individually. https://commoncrawl.org/ https:

Isn't this by problem solved by using commoncrawl data. I wonder what changed to AI companies to do mass crawling individually. https://commoncrawl.org/ https://commoncrawl.org/