Kind of sad that CommonCrawl, or something like it, has not removed the need for tons of different companies hitting all the servers in the world. I guess part

Kind of sad that CommonCrawl, or something like it, has not removed the need for tons of different companies hitting all the servers in the world. I guess part of it is wanting more control (more frequent visits, etc) and part is simply having lots of VC money and doing something they can do to try and impress more investors - "We have proprietary 5 PB dataset!" (literally adds nothing to commoncrawl).