Surprised at how crawling as a whole seems to have taken a sustained step backwards with old best practices that are solved being new to devs today? I can't he

Surprised at how crawling as a whole seems to have taken a sustained step backwards with old best practices that are solved being new to devs today? I can't help but wonder if big AI crawlers belonging to the LLMs wouldn't be doing some amount of local caching with Squid or something. Maybe it's beneficial somehow to let the websites tarpit them or slow down requests to use more tokens.