I think it was wikipedia recently that was pissed because bots were crawling the site instead of using the already available datasets. AI bots don't care about

I think it was wikipedia recently that was pissed because bots were crawling the site instead of using the already available datasets. AI bots don't care about caches. That's one of the big issues