3 ms·
There is no moderation, so, yes, anything (legal) goes, I suppose. But to the point: it is interesting that even CERN is taking a hit from the LLM crawlers.
by jruohonen 9d ago
There is no moderation, so, yes, anything (legal) goes, I suppose. But to the point: it is interesting that even CERN is taking a hit from the LLM crawlers.
- verdverm 9d agoI suspect that the growth in LLM "crawlers" is part traditional crawlers and a growing part WebFetch tools agents use directly on behalf of users. Personally, my page fetch count is way up because I have a custom deep research agent, I've almost weaned myself off traditional search, but that agent is fetching dozens of pages, like I used to click through a bunch of search results to figure out which are relevant, adjust my query, etc... That now happens at a higher frequency and volume through my agent.
- jruohonen 9d ago> agents use directly on behalf of users Could well be, who knows? But, well, then, in a sense you're a part of the problem, causing a DoS for infrastructures and other, human users.
- verdverm 9d agoMore people can serve up static markdown for the agents with `Content-Type: application/markdown`, CDNs work really well. I think both sides of this dynamic would find it beneficial. I suspect much of the infra stress is on all the "enhancements" found in the modern web application and the targeted advertising ecosystem