6 ms·
As a site operator who has been battling with the influx of extremely aggressive AI crawlers, I’m now wondering if my tactics have accidentally blocked internet
by VladVladikoff 7mo ago
As a site operator who has been battling with the influx of extremely aggressive AI crawlers, I’m now wondering if my tactics have accidentally blocked internet archive. I am totally ok with them scraping my site, they would likely obey robots.txt, but these days even Facebook ignores it, and exceeds my stipulated crawl delay by distributing their traffic across many IPs. (I even have a special nginx rule just for Facebook.)
Blocking certain JA3 hashes has so far been the most effective counter measures. However I wish there was an nginx wrapper around hugin-net that could help me do TCP fingerprinting as well. As I do not know rust and feel terrified of asking an LLM to make it. There is also a race condition issue with that approach, as it is passive fingerprinting even the JA4 hashes won’t be available for the first connection, and the AI crawlers I’ve seen do one request per IP so you don’t get a chance to block the second request (never happens).
- mycall 7mo agoEvasion techniques like JA3 randomization or impersonation can bypass detection.
- noads2000 7mo ago[dead]
- VladVladikoff 7mo agoI am aware, fortunately I haven't seen much of this... yet. Also JA4 is supposed to be a bit less vulnerable to this. Also this is why I really want TCP and HTTP fingerprinting. But the best i've found so far is https://github.com/biandratti/huginn-net https://github.com/biandratti/huginn-net and is only available as rust library, I really need it as an nginx module. I've been tempted to try to vibe code an nginx module that wraps this library.
- andrepd 7mo agoI wonder if it would be practical to have bot-blocking measures that can be bypassed with a signature from a set of whitelisted keys... In this case the server would be happy to allow Internet Archive crawlers.
- freedomben 7mo agoThat's an interesting idea. Mtls could probably be used for this pretty easily. It would require IA to support it if course, but could be a nice solution. I wonder, do they already support it? I might throw up a test...
- danrl 7mo ago> they would likely obey robots.txt If only... Despite providing a useful service, they are not as nice towards site owners as one would hope. Internet Archive says: > We see the future of web archiving relying less on robots.txt file declarations geared toward search engines https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... They are not alone in that. The "Archiveteam", a different organization, not to be confused with archive.org, also doesn't respect robots.txt according to their wiki: https://wiki.archiveteam.org/index.php?title=Robots.txt https://wiki.archiveteam.org/index.php?title=Robots.txt I think it is safe to say that there is little consideration for site owners from the largest archiving organizations today. Whether there should be is a different debate.
- sunaookami 7mo agoWhat an absolutely insufferable explanation from ArchiveTeam. What else do you expect from an organization aggressively crawling websites and bringing them down to their knees because they couldn't care less?
- rossng 7mo agoI'm curious to hear about examples of where this has happened. Because ArchiveTeam also has an important role in rescuing cultural artefacts that have been taken into private hands and then negligently destroyed.
- tredre3 7mo agoHaving a laudable goal doesn't absolve them from bad behavior.
- Dylan16807 7mo agoIt's a good reason to not worry about hypothetical bad behavior and wait for evidence of real bad behavior.
- 7mo ago