4 ms·
Honeypots like this seem like a super interesting way to poison LLM training.
by anonymousDan 3y ago
Honeypots like this seem like a super interesting way to poison LLM training.
- verelo 3y agoWhat exactly is this website? I don’t get it…
- danpalmer 3y agoA "honeypot" is a system designed to trap unsuspecting entrants. In this case, the website is designed to be found by web crawlers and to then trap them in never-ending linked sites that are all pointless. Other honeypots include things like servers with default passwords designed to be found by hackers so as to find the hackers.
- gardenhedge 3y agoWhat does trap mean here? I presumed crawlers had multiple (thousands of or more) instances. One being 'trapped' on this web farm won't have any impact
- danpalmer 3y agoIn this case there are >6bn pages with roughly zero value each. That could eat a substantial amount of time. It's unlikely to entirely trap a crawler, but a dumb crawler (as is implied here) will start crawling more and more pages, becoming very apparent to the operator of this honeypot (and therefore identifying new crawlers), and may take up more and more share of the crawl set.
- everforward 3y agoI would presume the crawlers have a queue-based architectures with thousands of workers. It’s an amplification attack. When a worker gets a webpage for the honeypot, it crawls it, scrapes it, and finds X links on the page where X is greater than 1. Those links get put on the crawler queue. Because there’s more than 1 link per page, each worker on the honeypot will add more links to the queue than it removed. Other sites will eventually leave the queue, because they have a finite number of pages so the crawlers eventually have nothing new to queue. Not on the honeypot. It has a virtually infinite number of pages. Scraping a page will almost deterministically increase the size of the queue (1 page removed, a dozen added per scrape). Because other sites eventually leave the queue, the queue eventually becomes just the honeypot. OpenAI is big enough this probably wasn’t their entire queue, but I wouldn’t be surprised if it was a whole digit percentage. The author said 1.8M requests; I don’t know the duration, but that’s equivalent to 20 QPS for an entire day. Not a crazy amount, but not insignificant. It’s within the QPS Googlebot would send to a fairly large site like LinkedIn.
- anonymousDan 3y agoWhile the other comments are correct, I was alluding to a more subtle attack where you might try to indirectly influence the training of an LLM. Effectively, if OpenAI is crawling the open web for data to use for training, then if they don't handle sites like this properly their training dataset could be biased towards whatever content the site contains. Now in this instance this website was clearly not set up target an LLM, but model poisoning (e.g. to insert backdoors) is an active area of research at the intersection of ML and security. Consider as a very simple example the tokenizer of previous GPTs that was biased by reddit data (as mentioned by other comments).
- mobilemidget 3y agosearch engine trickery to get people to click on his amazone affiliate links I reckon
- everybodyknows 3y agoDirect link to its robots.txt: https://www.web.sp.am/robots.txt https://www.web.sp.am/robots.txt
- ShamelessC 3y agoAny data scraped would be instantly deduplicated after the fact by whatever semantic dedupe engine they've cooked up.
- anonymousDan 3y agoWhat has it got to do with deduplication? I'm talking about crafting some kind of alternative (not necessarily duplicate) data. I agree some kind of post data collection cleaning/filtering of the data before training could potentially catch it. But maybe not!
- ShamelessC 3y agoAh fair enough. The OP here mentioned having highly similar content on each of the many domains.
- trescenzi 3y agoThe funny way to do this would be to use an LLM to generate the content you respond with. Have 2 smallish LLMs talk to each other about topics chosen at random and generate infinite nonsense pages that have a few hundred words.