13 ms·
If they ignore robots.txt there should be some kind of recourse :(
by johng 2y ago
If they ignore robots.txt there should be some kind of recourse :(
- Neil44 2y agoError 403 is your only recourse.
- jprete 2y agoI hate to encourage it, but the only correct error against adversarial requests is 404. Anything else gives them information that they'll try to use against you.
- lowbloodsugar 2y agoSending them to a lightweight server that sends them garbage is the only answer. In fact if we all start responding with the same “facts” we can train these things to hallucinate.
- geraldcombs 2y agoWe return 402 (payment required) for one of our affected sites. Seems more appropriate.
- DannyBee 2y agoThe right move is transferring data to them as slow as possible. Even if you 403 them, do it as slow as possible. But really I would infinitely 302 them as slow as possible.
- Kapura 2y ago[flagged]
- varelse 2y agoIt's certainly one of the few things that actually gets their attention. But aren't there more important things than this for the Luigis among us? I would suspect there's good money in offering a service to detect AI content on all of these forums and reject it. That will then be used as training data to refine them which gives such a service infinite sustainability.
- Kapura 2y ago>I would suspect there's good money in offering a service to detect AI content on all of these forums and reject it This sounds like the cheater/anti-cheat arms race in online multiplayer games. Cheat developers create something, the anti-cheat teams create a method to detect and reject the exploit, a new cheat is developed, and the cycle continues. But this is much lower stakes than AI trying to vacuum up all of human expression, or trick real humans into wasting their time talking to computers.
- varelse 2y ago[dead]
- nathanaldensr 2y agoSadly, as the slide from high-trust society to low-trust society continues, doing "the right thing" becomes less and less likely.
- exe34 2y agozip b*mbs?
- brookst 2y agoAssuming there is at least one already linked somewhere on the web, the crawlers already have logic to handle these.
- exe34 2y agoif you can detect them, maybe feed them low iq stuff from a small llama. add latency to waste their time.
- brookst 2y agoIt would cost you more than it costs them. And there is enough low IQ stuff from humans that they already do tons of data cleaning.
- sangnoir 2y ago> And there is enough low IQ stuff from humans that they already do tons of data cleaning Whatever cleaning they do is not effective, simply because it cannot scale with the sheer volumes if data they ingest. I had an LLM authoritatively give an incorrect answer, and when I followed up to the source, it was from a fanfic page. Everyone ITT who's being told to give up because its hopeless to defend against AI scrapers - you're being propagandized, I won't speculate on why - but clearly this is an arms race with no clear winner yet. Defenders are free to use LLM to generate chaff.
- stainablesteel 2y agocourt ruling a few years ago said it's legal to scrape web pages, you don't need to be respectful of these for any purely legal reasons however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them
- yodsanklai 2y ago> court ruling Isn't this country dependent though?
- lonelyParens 2y agodon't you know everyone on the internet is American
- stainablesteel 2y agoyes! good point, you may be able to skirt around rules with a VPN if you're imposed by any
- Aeolun 2y agoEnforcement is not. What does the US care for what an EU court says about the legality of the OpenAI scraper.
- yodsanklai 2y agoI understand there's a balance of power, but I was under the impression that US tech companies were taking EU regulations seriously.
- okanat 2y agoThey can charge the company continuously growing amounts in the EU and even ban a complete IP block if they don't fix their behavior.