4 ms·
Best way fight back is to create a tarpit that will feed them garbage: https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/
by cardanome 8mo ago
Best way fight back is to create a tarpit that will feed them garbage: https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/
- jacquesm 8mo agoAnd to try to get them execute bb(5) ;)
- joquarky 8mo agoclaude --plan "let's develop a plan to detect and mitigate tarpits" Ten minutes later, the ball is back in your court.
- epidemian 8mo agoDo you think an LLM would be able to generate a solution to a novel problem just like that? That doesn't match my (albeit limited) experience with these things. They are pretty good at other things, but generally squarely in the real of "already done" things.
- blargey 8mo agoAnti-crawler tarpits and related concepts have existed for decades already; LLM training data is only the latest and most popular of web-scraping goals. Claude is happy and able to provide a laundry list of ways to mitigate the impact of tarpits on your crawler, and politeness / respecting robots.txt is only one of them.
- bee_rider 8mo agoThis is a file for a LLM, not a scraper, so anti-scraping mitigations seem sort of beside the point.