6 ms·

Has anyone done a talk/blog/whatever on how llm crawlers are different than classical crawlers? I'm not up on the difference.

klodolph 11mo ago
The only real difference that LLM crawlers tend to not respect /robots.txt and some of them hammer sites with some pretty heavy traffic. The trap in the article has a link. Bots are instructed not to follow the link. The link is normally invisible to humans. A client that visits the link is probably therefore a poorly behaved bot.
phantomathkg 11mo ago
Yes https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-th... https://www.usebox.net/jjm/blog/the-problem-of-the-llm-crawlers/ https://www.usebox.net/jjm/blog/the-problem-of-the-llm-crawl...
- DeepYogurt 11mo ago
  Thanks!
superkuh 11mo ago
Recently there have been more crawlers coming from tens to hundreds of IP netblocks from dozens (or more!) of ASN in highly time and URL correlated fashion with spoofed user-agent(s) and no regard for rate or request limiting or robots.txt. These attempt to visit every possible permutation of URLs on the domain and have a lot of bandwidth and established tcp connections available to them. It's not that this didn't happen pre-2023 but it's noticably more common now. If you have a public webserver you've probably experienced it at least once. Actual LLM involvement as the requesting user-agent is vanishingly small. It's the same problem as ever: corporations, their profit motive during $hypecycle coupled with access to capital for IT resources, and the protection of the abusers via the company's abstraction away of legal liability for their behavior.
btown 11mo ago
IMO there was something of a de facto contract, pre-LLMs, that the set of things one would publicly mirror/excerpt/index and the set of things one would scrape were one and the same. Back then, legitimate search engines wouldn’t want to scrape things that would just make their search results less relevant with garbage data anyways, so by and large they would honor robots.txt and not overwhelm upstream servers. Bad actors existed, of course, but were very rarely backed by companies valued in the billions of dollars. People training foundation models now have no such constraints or qualms - they need as many human-written sentences as possible, regardless of the context in which they are extracted. That’s coupled with a broader familiarity with ubiquitous residential proxy providers that can tunnel traffic through consumer connections worldwide. That’s an entirely different social contract, one we are still navigating.
- stephenitis 11mo ago
  Text, images, video, all of it I can’t think of any form of data they don’t want to scoop up, other than noise and poisoned data
- cwbriscoe 11mo ago
  I am not well versed in this problem but can't the web servers rate limit by known IP addresses of these crawler/scrapers?
  - ninja3925 11mo ago
    Large cloud providers could offer that solution but then, crawlers can also change cycle IPs
  - strogonoff 11mo ago
    You cannot block LLM crawlers by IP address, because some of them use residential proxies. Source: 1) a friend admins a slightly popular site and has decent bot detection heuristics, 2) just Google “residential proxy LLM”, they are not exactly hiding. Strip-mining original intellectual property for commercial usage is big business.
dspillett 11mo ago
The crawlers themselves are not that different: it is their number, how the information is used once scraped (including referencing or lack thereof), and if they obey the rules: 1. Their number: every other company and the mangy mutt that is its mascot is scraping for LLMs at the moment, so you get hit by them far more than you get hit by search engine bots and similar. This makes them harder to block too, because even ignoring tricks like using botnets to spread requests over many source addresses (potentially the residential connections of unwitting users infected by malware) the share number coming from so many places, new places all the time, means you can not maintain a practical blocklist of source addresses. The number of scrapers out there means that small sites can be easily swamped, much like when HN, slashdot, or a popular reddit subsection, links to a site, and it gets “hugged to death” by a sudden glut of individual people who are interested. 2. Use of the information: Search engines actually provide something back: sending people to your site. Useful if that is desirable which in many cases it is. LLMs don't tend to do that though: by the very nature of LLMs very few results from them come with any indication of the source of the data they use for their guesswork. They scrape, they take, they give nothing back. Search engines had a vested interest in your site surviving as they don't want to hand out dead links, those scraping for LLMs have no such requirement because they can still summarise your work from what is effectively cached within their model. This isn't unique to LLMs, go back a few years to the pre-LLM days and you will find several significant legal cases about search engines offering summaries of the information found instead of just sending people to the site where the information is. 3. Ignoring rules: Because so many sites are attempting to block scrapers now, usually at a minimum using accepted methods to discourage it (robots.txt, nofollow attributes, etc.), these signals are just ignored. Sometimes this is malicious with people running the scrapers simply not caring despite knowing the problem they could create, sometimes it is like the spam problem in mail: each scraper thinks it'll be fine because it is only them, with each of the many also thinking the same thing… With people as big as Meta openly defending piracy as just fine for the purposes of LLM training, others see that as a declaration of open season. Those that are malicious or at least amoral (most of them) don't care. Once they have scraped your data they have, as mentioned above, no vested interest in whether your site lives or dies (either by withing away from lack of attention or falling over under their load to never be brought back up), in fact they might have incentive to want your site dead: it would no longer compete with the LLM as a source of information. No one of these is the problem, but together they are a significant problem.

klodolph 11mo ago

The only real difference that LLM crawlers tend to not respect /robots.txt and some of them hammer sites with some pretty heavy traffic. The trap in the article has a link. Bots are instructed not to follow the link. The link is normally invisible to humans. A client that visits the link is probably therefore a poorly behaved bot.

phantomathkg 11mo ago

Yes https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-th... https://www.usebox.net/jjm/blog/the-problem-of-the-llm-crawlers/ https://www.usebox.net/jjm/blog/the-problem-of-the-llm-crawl...

DeepYogurt 11mo ago
Thanks!

superkuh 11mo ago

Recently there have been more crawlers coming from tens to hundreds of IP netblocks from dozens (or more!) of ASN in highly time and URL correlated fashion with spoofed user-agent(s) and no regard for rate or request limiting or robots.txt. These attempt to visit every possible permutation of URLs on the domain and have a lot of bandwidth and established tcp connections available to them. It's not that this didn't happen pre-2023 but it's noticably more common now. If you have a public webserver you've probably experienced it at least once. Actual LLM involvement as the requesting user-agent is vanishingly small. It's the same problem as ever: corporations, their profit motive during $hypecycle coupled with access to capital for IT resources, and the protection of the abusers via the company's abstraction away of legal liability for their behavior.

btown 11mo ago

IMO there was something of a de facto contract, pre-LLMs, that the set of things one would publicly mirror/excerpt/index and the set of things one would scrape were one and the same. Back then, legitimate search engines wouldn’t want to scrape things that would just make their search results less relevant with garbage data anyways, so by and large they would honor robots.txt and not overwhelm upstream servers. Bad actors existed, of course, but were very rarely backed by companies valued in the billions of dollars. People training foundation models now have no such constraints or qualms - they need as many human-written sentences as possible, regardless of the context in which they are extracted. That’s coupled with a broader familiarity with ubiquitous residential proxy providers that can tunnel traffic through consumer connections worldwide. That’s an entirely different social contract, one we are still navigating.

stephenitis 11mo ago
Text, images, video, all of it I can’t think of any form of data they don’t want to scoop up, other than noise and poisoned data
cwbriscoe 11mo ago
I am not well versed in this problem but can't the web servers rate limit by known IP addresses of these crawler/scrapers?
- ninja3925 11mo ago
  Large cloud providers could offer that solution but then, crawlers can also change cycle IPs
- strogonoff 11mo ago
  You cannot block LLM crawlers by IP address, because some of them use residential proxies. Source: 1) a friend admins a slightly popular site and has decent bot detection heuristics, 2) just Google “residential proxy LLM”, they are not exactly hiding. Strip-mining original intellectual property for commercial usage is big business.

dspillett 11mo ago

The crawlers themselves are not that different: it is their number, how the information is used once scraped (including referencing or lack thereof), and if they obey the rules: 1. Their number: every other company and the mangy mutt that is its mascot is scraping for LLMs at the moment, so you get hit by them far more than you get hit by search engine bots and similar. This makes them harder to block too, because even ignoring tricks like using botnets to spread requests over many source addresses (potentially the residential connections of unwitting users infected by malware) the share number coming from so many places, new places all the time, means you can not maintain a practical blocklist of source addresses. The number of scrapers out there means that small sites can be easily swamped, much like when HN, slashdot, or a popular reddit subsection, links to a site, and it gets “hugged to death” by a sudden glut of individual people who are interested. 2. Use of the information: Search engines actually provide something back: sending people to your site. Useful if that is desirable which in many cases it is. LLMs don't tend to do that though: by the very nature of LLMs very few results from them come with any indication of the source of the data they use for their guesswork. They scrape, they take, they give nothing back. Search engines had a vested interest in your site surviving as they don't want to hand out dead links, those scraping for LLMs have no such requirement because they can still summarise your work from what is effectively cached within their model. This isn't unique to LLMs, go back a few years to the pre-LLM days and you will find several significant legal cases about search engines offering summaries of the information found instead of just sending people to the site where the information is. 3. Ignoring rules: Because so many sites are attempting to block scrapers now, usually at a minimum using accepted methods to discourage it (robots.txt, nofollow attributes, etc.), these signals are just ignored. Sometimes this is malicious with people running the scrapers simply not caring despite knowing the problem they could create, sometimes it is like the spam problem in mail: each scraper thinks it'll be fine because it is only them, with each of the many also thinking the same thing… With people as big as Meta openly defending piracy as just fine for the purposes of LLM training, others see that as a declaration of open season. Those that are malicious or at least amoral (most of them) don't care. Once they have scraped your data they have, as mentioned above, no vested interest in whether your site lives or dies (either by withing away from lack of attention or falling over under their load to never be brought back up), in fact they might have incentive to want your site dead: it would no longer compete with the LLM as a source of information. No one of these is the problem, but together they are a significant problem.