5 ms·
OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv https://github.com/greyhat-academy/
by walterbell 2y ago
OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv https://github.com/greyhat-academy/lists.d/blob/main/scraper...
For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ https://kevinfreitas.net/tools-experiments/
> The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the content on blog post and pages on your site when one of these bots slithers by.
- brookst 2y agoThe latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.
- walterbell 2y agoObfuscators can evolve alongside other LLM arms races.
- ben_w 2y agoYes, but with an attacker having advantage because it directly improves their own product even in the absence of this specific motivation for obfuscation: any Completely Automated Public Turing test to tell Computers and Humans Apart can be used to improve the output of an AI by requiring the AI to pass that test. And indeed, this has been part of the training process for at least some of OpenAI models before most people had heard of them.
- mrbungie 2y ago1. It is a moral victory: at least they won't use your own text. 2. As a sibling proposes, this is probably going to become an perpetual arms race (even if a very small one in volume) between tech-savvy content creators of many kinds and AI companies scrapers.
- rickyhatespeas 2y agoIt will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.
- walterbell 2y ago> now every time someone searches for your site they will be taken straight to competitors There are non-LLM forms of distribution, including traditional web search and human word of mouth. For some niche websites, a reduction in LLM-search users could be considered a positive community filter. If LLM scraper bots agree to follow longstanding robots.txt protocols, they can join the community of civilized internet participants.
- knuppar 2y agoExactly. Not every website needs to be at the top of SEO (or LLM-O?). Increasingly the niche web feels nicer and nicer as centralized platforms expand.
- scrollaway 2y agoIndeed, it's like dumping rotting trash all over your garden and saying "Ha! Now Jehovah's witnesses won't come here anymore".
- jonnycomputer 2y agoNo, its like building a fence because your neighbors' dogs keep shitting in your yard and never clean it up.
- luckylion 2y agoYou can still fine-tune though. I often run User-Agent: *, Disallow: / with User-Agent: Googlebot, Allow: / because I just don't care for Yandex or baidu to crawl me for the 1 user/year they'll send (of course this depends on the region you're offering things to). That other thing is only a more extreme form of the same thing for those who don't behave. And when there's a clear value proposition in letting OpenAI ingest your content you can just allow them to.
- nerdponx 2y agoSeems like an effective technique for preventing your content from being included in the training data then!
- wood_spirit 2y agoRather than garbage, perhaps just serve up something irrelevant and banal? Or splice sentences from various random project Gutenberg books? And add in a tarpit for good measure. At least in the end it gives the programmer one last hoorah before the AI makes us irrelevant :)
- smt88 2y agoI have zero faith that OpenAI respects attempts to block their scrapers
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- tylerchilds 2y agothat’s what makes this clever. they aren’t blocking them. they’re giving them different content instead.
- GaggiX 2y agoI imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything.
- walterbell 2y agoAttackers don't have a monopoly on LLM expertise, defenders can also use LLMs for obfuscation. Technology arms races are well understood.
- GaggiX 2y agoI hate LLM companies, I guess I'm going to use OpenAI API to "obfuscate" the content or maybe I will buy an NVIDIA GPU to run a llama model, mhm maybe on GPU cloud.
- walterbell 2y agoWith tiny amounts of forum text, obfuscation can be done locally with open models and local inference hardware (NPU on Arm SoC). Zero dollars sent to OpenAI, NVIDIA, AMD or GPU clouds.
- GaggiX 2y ago>local inference hardware (NPU on Arm SoC). Okay the battle is already lost from the beginning.
- walterbell 2y agoThere are alternatives to NVIDIAmaxing with brute force. See the Chinese paper on DeepSeek V3, comparable to recent GPT and Claude, trained with 90% fewer resources. Research on efficient inference continues. https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSee...
- pogue 2y agoWhat specifically are you suggesting? Is this a project that already exists or a theory of yours?
- ceejayoz 2y ago> OpenAI publishes IP ranges for their bots... If blocking them becomes standard practice, how long do you think it'd be before they started employing third-party crawling contractors to get data sets?
- bonestamp2 2y agoMaybe they want sites to block them that don't want to be crawled since it probably saves them a lawsuit down the road.
- deleted 2y ago[deleted]
- pmontra 2y agoInstead of nonsense you can serve a page explaining how you can ride a bicycle to the moon. I think we had a story about that attack to LLMs a few months ago but I can't find it quickly enough.
- sangnoir 2y agoiFixIt has detailed fruit-repair instructions. IIRC, they are community-authored.
- aorth 2y agoNote that the official docs from OpenAI listing their user agents and IP ranges is here: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots