4 ms·
I suspect that the growth in LLM "crawlers" is part traditional crawlers and a growing part WebFetch tools agents use directly on behalf of users. Personally,
by verdverm 26d ago
I suspect that the growth in LLM "crawlers" is part traditional crawlers and a growing part WebFetch tools agents use directly on behalf of users.
Personally, my page fetch count is way up because I have a custom deep research agent, I've almost weaned myself off traditional search, but that agent is fetching dozens of pages, like I used to click through a bunch of search results to figure out which are relevant, adjust my query, etc...
That now happens at a higher frequency and volume through my agent.
- jruohonen 26d ago> agents use directly on behalf of users Could well be, who knows? But, well, then, in a sense you're a part of the problem, causing a DoS for infrastructures and other, human users.
- verdverm 26d agoMore people can serve up static markdown for the agents with `Content-Type: application/markdown`, CDNs work really well. I think both sides of this dynamic would find it beneficial. I suspect much of the infra stress is on all the "enhancements" found in the modern web application and the targeted advertising ecosystem