19 ms·
AI scrapers request commented scripts
- rokkamokka 1y agoI'm not overly surprised, it's probably faster to search the text for http/https than parse the DOM
- embedding-shape 1y agoNot probably, searching through plaintext (which they seem to be doing) VS iterating on the DOM have vastly different amount of work behind them in terms of resources used and performance that "probably" is way underselling the difference :)
- franktankbank 1y agoReminds me of the shortcut that works for the happy path but is utterly fucked by real data. This is an interesting trap, can it easily be avoided without walking the dom?
- embedding-shape 1y agoYes, parse out HTML comments which is also kind of trivial if you've ever done any sort of parsing, listen for "<!--", whenever you come across it, ignore everything until the next "-->". But then again, these people are using AI to build scrapers, so I wouldn't put too much pressure on them to produce high-quality software.
- stevage 1y agoLots of other ways to include URLs in an HTML document that wouldn't be visible to a real user, though.
- jcheng 1y agoIt's not quite as trivial as that; one could start the page with a <script> tag that contains "<!--" without matching "-->", and that would hide all the content from your scraper but not from real browsers. But I think it's moot, parsing HTML is not very expensive if you don't have to actually render it.
- marginalia_nu 1y agoThe regex approach is certainly easier to implement, but honestly static DOM parsing is pretty cheap, but quite fiddly to get right. You're probably gonna be limited by network congestion (or ephemeral ports) before you run out of CPU time doing this type of crawling.
- Noumenon72 1y agoIt doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
- michael1999 1y agoCrawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.
- tveyben 1y agoHuman behavior is interesting - me, me, me…
- mostlysimilar 1y agoThe article mentions using this as a means of detecting bots, not as a complaint that it's abusive. EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.
- pseudalopex 1y agoPlease don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".[1] [1] https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- woodrowbarlow 1y agothe first few words of the article are: > Last Sunday I discovered some abusive bot behaviour [...]
- mostlysimilar 1y ago> The robots.txt for the site in question forbids all crawlers, so they were either failing to check the policies expressed in that file, or ignoring them if they had.
- deleted 1y ago[deleted]
- latenightcoding 1y agowhen I used to crawl the web, battle tested Perl regexes were more reliable than anything else, commented urls would have been added to my queue.
- rightbyte 1y agoDOM navigation for fetching some data is for tryhards. Using a regex to grab the correct paragraph or div or whatever is fine and is more robust versus things moving around on the page.
- horseradish7k 1y agobut not when crawling. you don't know the page format in advance - you don't even know what the page contains!
- chaps 1y agoDoing both is fine! Just, once you've figured out your regex and such, hardening/generalizing demands DOM iteration. It sucks but it is what is is.
- OhMeadhbh 1y agoI blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references. Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.
- tuwtuwtuwtuw 1y agoDo you think that if some CS programs taught parsing, the authors of the bot would parse the HTML to properly extract links, instead of just doing plain text search? I doubt it.
- OhMeadhbh 11mo agoSure. Ideally CS programs would be more than the trade schools they are today. When I was in school, the "CS Program" was a specialization in the math department. I was already in a Physics program when they created a "Computer Science & Engineering" degree. And I remember both the Math and CSE degrees being pretty decent. But that was back in the day when people took CS related courses because they were interested in computers, not because they wanted to get a super high paying job for a FANG company. Maybe you're right. You can lead a horse to water, but you can't make it think deep thoughts about parsing.
- ericmcer 1y agoHow would recommend doing it? If I was just trying to pull <a/> tag links out I feel like treating it like text and using regex would be way more efficient than a full on HTML parser like JSDom or something.
- singron 1y agoYou don't need javascript to parse HTML. Just use an HTML parser. They are very fast. HTML isn't a regular language, so you can't parse it with regular expressions. Obligatory: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
- sharkjacobs 1y agoFun to see practical applications of interesting research[1] [1]https://news.ycombinator.com/item?id=45529587 https://news.ycombinator.com/item?id=45529587
- bakql 1y ago>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.
- XenophileJKO 1y agoWhat about people using an LLM as their web client? Are you now saying the website owner should be able to dictate what client I use and how it must behave?
- aDyslecticCrow 1y ago> Are you now saying the website owner should be able to dictate what client I use and how it must behave? Already pretty well established with Ad-block actually. It's a pretty similar case even. AI's don't click ads, so why should we accept their traffic? If it's un-proportionally loading the server without contributing to the funding of the site, get blocked. The server can set whatever rules it wants. If the maintainer hates google and wants to block all chrome users, it can do so.
- XenophileJKO 1y agoThat was kind of what I was really hinting at, as the HN community tends to embrace things like ad blockers and archive links on stories, but god forbid someone read a site using an LLM.
- 1gn15 1y agoHumans are usually hypocritical. They support whatever they personally use while opposing whatever inconveniences them, even though they're basically the same thing. This whole thing has made me hate humans, so so much. Robots are much better.
- aDyslecticCrow 1y agoI use adblock myself, and don't feel bad for using it (it's a security and privacy tool). But i don't blame websites that kick me out for it; hosting costs money. Server owners should have all the right to set the terms of their server access. Better tools to control LLMs and scrapers are all good in my book. I really wish ad platforms were better at managing malware, trackers and fraud through. It is rather difficult to fully argue for website owner authority with how bad ads actually are for the user.
- vale11_amo2 1y ago[flagged]
- edm0nd 1y agoone of the best movies, yes.
- bigbuppo 1y agoSounds like you should give the bots exactly what they want... a 512MB file of random data.
- kelnos 1y agoMost people have to pay for their bandwidth, though. That's a lot of data to send out over and over.
- jcheng 1y ago512MB file of incredibly compressible data, then?
- QuadmasterXLII 1y agoCould I recommend https://cubes.hgreer.com/ssg/output.html https://cubes.hgreer.com/ssg/output.html ? 50:1 compression ratio, but it's legitimately an implementation of a rubiks cube, that I wasn't actually making as any sort of trap, just wasn't thinking about file size, so any rule that filters it out is going to have a nasty false positive rate
- aDyslecticCrow 1y agoScraper sinkhole of randomly generated inter-linked files filled with AI poison could work. No human would click that link, so it leads to the "exclusive club".
- winddude 1y ago[flagged]
- stevage 1y agoThe title is confusing, should be "commented-out".
- hexage1814 1y ago[flagged]
- ang_cire 1y agoYep. Robots.txt is a framework intended for performance, not a legal or ethical imperative. If you want to control how someone accesses something, the onus is on you to put access controls in place. The people who put things on a public, un-restricted server and then complain that the public accessed it in an un-restricted way might be excusable if it's some geocities-esque Mom and Pop site that has no reason to know better, but 'cryptography dog' ain't that.
- ordu 1y agoIt is an asocial state of mind. We have locks and security systems that prevent people from stealing. But if all people agreed to not steal, then we could save that efforts for something better. The ideal approach doesn't work with the stealing, and now it doesn't work with HTTP either. It just raises costs for a society with no lasting benefit for anyone: site owners just figure out ways to restrict access and no more scraping of pages that they do not want to be scraped. A healthy society relies on a cooperation between members. It relies on them accepting some rules that limits their behavior. Like we agreed not to kill others, and now I can go outside without weapons and anti-bullet defenses.
- ang_cire 1y agoVisiting websites isn't theft. Scraping isn't theft. Copyright infringement isn't theft. Turning a digital asset like web pages that have no intrinsic limits on how many times they can be accessed into a limited good because you don't want someone to have something is the actual asocial mindset.
- Anamon 1y agoWhat controls do you suggest? Saying that a handful of mass copyright infringers with billion dollar investors are simply part of the "public" like every regular visitor is seriously distorting the issue here. Sites with a robots.txt banning bots are only "unrestricted" in a strictly technical sense. They are clearly setting terms of use that these rogue bots are violating. Besides, robots.txt is legally binding in certain jurisdictions, it's not just a polite plea. And if we decide that anything not technically prevented is legal, then we're also legitimising botnets, DDoS attacks, and a lot more. Hacking into a corporate system through a malconfiguration or vulnerability is also illegal, despite the fact that the defenses failed. Finally, we all know that the only purpose these bots are scraping for is mass copyright infringement. That's another layer where the "if it's accessible, it's fair game" logic falls apart. I can download a lot of publicly accessible art, music, or software, but that doesn't mean I can do with those files whatever I want. The only reason these AI companies haven't been sued out of existence yet, like they should've been, is that it's trickier to prove provenance than if they straight up served the unmodified files.
- mikeiz404 1y agoTwo thoughts here when it comes to poisoning unwanted LLM training data traffic 1) A coordinated effort among different sites will have a much greater chance of poisoning the data of a model so long as they can avoid any post scraping deduplication or filtering. 2) I wonder if copyright law can be used to amplify the cost of poisoning here. Perhaps if the poisoned content is something which has already been shown to be aggressively litigated against then the copyright owner will go after them when the model can be shown to contain that banned data. This may open up site owners to the legal risk of distributing this content though… not sure. A cooperative effort with a copyright holder may sidestep this risk but they would have to have the means and want to litigate.
- Anamon 1y agoAs for 1, it would be great to have this as a plugin for WordPress etc. that anyone could simply install and enable. Pre-processing images to dynamically poison them on each request should be fun, and also protect against a deduplication defense. I'd certainly install that.
- renegat0x0 1y agoMost web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities. I know a thing or two about web scraping. There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi). Zip bombs are also not for me. Reading header content length is enough to not read the page/file. I provide byte limit to check if response is not too big for me. For other cases reading timeout is enough. Oh, and did you know that requests timeout is not really timeout a timeout for page read? So server can spoonfeed you bytes, one after another, and there will be no timeout. That is why I created my own crawling system to mitigate these problems, and have one consistent mean of running selenium. https://github.com/rumca-js/crawler-buddy https://github.com/rumca-js/crawler-buddy Based on library https://github.com/rumca-js/webtoolkit https://github.com/rumca-js/webtoolkit
- hnav 1y agocontent-length is computed after content-encoding
- ahoka 1y agoIf it’s present at all.
- Mars008 1y agoLooks like it's time for in-browser scrappers. They will be indistinguishable from the servers side. With AI driver can pass even human tests.
- overfeed 1y ago> Looks like it's time for in-browser scrappers. If scrapers were as well-behaved as humans, website operators wouldn't bother to block them[1]. It's the abuse that motivates the animus and action. As the fine articles spelt out, scrapers are greedy in many ways, one of which is trying to slurp down as many URLs as possible without wasting bytes. Not enough people know about common crawl, or know how to write multithreaded scrapers with high utilization across domains without suffocating any single one. If your scraper is URL FIFO or stack in a loop, you're just DOSing one domain at a time. 1. The most successful scrapers avoid standing out in any way
- sokoloff 1y agoWell, if they’re going to request commented out scripts, serve them up some very large scripts…
- lrpe 1y ago[flagged]
- throw_me_uwu 1y ago> most likely trying to non-consensually collect content for training LLMs No, it's just background internet scanning noise
- lucasluitjes 1y agoThis. If you were writing a script to mass-scan the web for vulnerabilities, you would want to collect as many http endpoints as possible. JS files, regardless of whether they're commented out or not, are a great way to find endpoints in modern web applications. If you were writing a scraper to collect source code to train LLMs on, I doubt you would care as much about a commented-out JS file. I'm not sure you'd even want to train on random low-quality JS served by websites. Anyone familiar with LLM training data collection who can comment on this?
- babra1 1y ago[flagged]
- huflungdung 1y ago[dead]