> There must a ton of new full-web datasets out there, right? Sadly, no. There's CommonCrawl (https://commoncrawl.org/ https://commoncrawl.org/) which still, s

> There must a ton of new full-web datasets out there, right? Sadly, no. There's CommonCrawl (https://commoncrawl.org/ https://commoncrawl.org/) which still, sadly, far removed from "full-web dataset." So everyone runs their own search instead, hammering the sites, going into gray areas (you either ignore robots.txt or your results suck), etc. It's a tragedy of the commons that keeps Google entrenched: https://senkorasic.com/articles/ai-scraper-tragedy-commons https://senkorasic.com/articles/ai-scraper-tragedy-commons