4 ms·
The AI-Scraping Free-for-All Is Coming to an End
- WaltPurvis 1y agohttp://archive.today/SqPCL http://archive.today/SqPCL
- jmkni 1y agoIt is a bit ironic that a paywalled article like this will have a top level comment with the archive link, which can then be easily scraped by AI (along with the comments)
- tenuousemphasis 1y agoIt's not ironic at all. The only reason the anti-paywall sites work is that the news companies in fact want some scrapers reading the full article.
- mschuster91 1y agoActually, the team behind archive dot today in at least spiegel.de has premium accounts, I presume bought with anonymous credit cards. You can see artifacts when their servers are at queue load and you see the URLs, a few resources have the JWT with the account details in the URL. IIRC the clearname of the account in the token is Masha Rabinovich, with an email account masha@dns.li, an identity that has cropped up in various investigations [1][2]. [1] https://gyrovague.com/2023/08/05/archive-today-on-the-trail-of-the-mysterious-guerrilla-archivist-of-the-internet/ https://gyrovague.com/2023/08/05/archive-today-on-the-trail-... [2] https://webapps.stackexchange.com/questions/145817/who-owns-archive-today-and-archive-is-and-where-is-it-hosted https://webapps.stackexchange.com/questions/145817/who-owns-...
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- ec109685 1y agoAlso interesting how sites like this are mainstream whereas a link to a site hosting an mp3 of pirated music wouldn’t be tolerated in discussion forums like this. I think a big difference is that there’s no micro transactions or compulsory licensing for content, so it always feels patently unfair to buy a subscription to read one article.
- yencabulator 1y agoI'd argue it's more that RIAA has historically been much more aggressive at suing than newspapers or magazines.
- ec109685 1y agoTrue. I think it has ended up a net good. People make a living on music, and licensed music is everywhere.
- orbisvicis 1y agoKinda hard to discuss the news when your members can't read the news.
- JacobKfromIRC 1y agoIn this case, it also seems like the paywall doesn't show up if you have JavaScript disabled, which I find strange, but lots of news sites are like that I think.
- euroderf 1y agoRelated: Has anyone trained an LLM strictly on HN comments and linked-to articles ? I for one would get a kick out of interrogating it.
- 1gn15 1y agoBiased TL;DR: Reddit (notable for having a high stock value from their "selling data" business [1]), Medium, Quora, and Cloudflare competitor Fastly created a standard to restrict what the reader can do with the data users created, called Really Simple Licensing (RSL). Basically robots.txt but with more details, notably with details on how much you should pay Reddit/Medium/Quora. While this likely has no legal weight (except for EU TDM for commercial use, where the law does take into account opt-outs), they are betting on using services like CloudFlare and Fastly to enforce this. [1] https://www.investors.com/research/the-new-america/reddit-stock-rddt-ai-ipo-google-search/ https://www.investors.com/research/the-new-america/reddit-st...
- isodev 1y agoIn other words, a lightweight form of DRM. Here come the reasons why we shouldn’t all deploy CloudFlare and similar as gatekeepers to the web. Is there even one example of a “tech mega corp” that has grown to control more than 1/5 of its market without this circling back to hurt people in some way? A single example?
- PhantomHour 1y ago> While this likely has no legal weight I wouldn't be quite so sure about that. The AI industry has entirely relied on 'move fast and break things' and 'old fart judges who don't understand the tech' as their legal strategy. The idea that AI training is fair use isn't so obvious, and quite frankly is entirely ridiculous in a world where AI companies pay for the data. If it's not fair use to take reddit's data, it's not fair use to take mine either. On a technological level the difference to prior ML is straightforward: A classical classifier system is simply incapable of emitting any copyrighted work it was trained on. The very architecture of the system guarantees it to produce new information derived from the training data rather than the training data itself. LLMs and similar generative AI do not have that safeguard. To be practically useful they have to be capable of emitting facts from training data, but have no architectural mechanism to separate facts from expressions. For them to be capable of emitting facts they must also be capable of emitting expressions, and thus, copyright violation. Add in how GenAI tends to directly compete with the market of the works used as training data in ways that prior "fair use" systems did not and things become sketchy quickly. Every major AI company knows this, as they have rushed to implement copyright filtering systems once people started pointing out instances of copyrighted expressions being reproduced by AI systems. (There are technical reasons why this isn't a very good solution to curtail copyright infringement by AI) Observe how all the major copyright victories amount to judges dismissing cases on grounds of "Well you don't have an example specific to your work" rather than addressing whether such uses are acceptable as a collective whole.
- deadbabe 1y agoJust ladder kicking at this point.
- jsnell 1y agoThe headline seems pretty aspirational. The licensing standard they're talking about will achieve nothing. Anti-bot companies selling scraping protections will run out of runway: there's a limited set of signals, and none of them are robust. As the signals get used, they're also getting burned. And it's politically impossible to expand the web platform to have robust counter-abuse capabilities. Putting the content behind a login wall can work for large sites, but not small ones. The free-for-all will not end until adversarial scraping becomes illegal.
- carlosjobim 1y ago> Putting the content behind a login wall can work for large sites, but not small ones. Syndication is the answer. Small artists are on Spotify, small video makers are on YouTube.
- salawat 1y agoYes. Conglomeration and centralization. More, more, more! See the problem?
- carlosjobim 1y agoYou don't have to syndicate a million small creators to have a product worthwhile for consumers, it could be a thousand, a hundred, ten thousand creators in a syndicate. You can have a huge number of syndicates, which benefits creators and consumers.
- orbisvicis 1y agoBut in such an environment syndicates will have an incentive to centralize.
- carlosjobim 1y agoI don't see why. In general, there are competing syndicates and businesses of every size in most sectors of the economy.
- aaaggg 1y agoL - wish they'd stop posting articles that are paywalled...
- janalsncm 1y ago> There was for years an experimental period, when ethical and legal considerations about where and how to acquire training data for hungry experimental models were treated as afterthoughts. Those things were afterthoughts because for the most part the experimental methods sucked compared to the real thing. If we were in mid 2016 and your LSTM was barely stringing together coherent sentences, it was a curiosity but not a serious competitor to StackOverflow. I say this not because I don’t think law/ethics are important in the abstract, but because they only became relevant after significant technological improvement.
- Zigurd 1y agoSites containing original content will adopt active measures against LLM scraper bots. Unlike search indexing bots, there's much less upside to allowing scraping for LLM training material. Openly adversarial actions like serving up poisoned text that would induce LLMs to hallucinate is much more defensible.
- ath3nd 1y agoNext: the AI bubble is coming to an end. Also fingers crossed that the career and employment of Mark Zuckerberg also follow suit soon.
- throwawayqqq11 1y agoAI companies struggle to convert profits, meta does not. VC will eventually run out, then comes the burst.
- xarope 1y agoI can see how the AI companies would work around this though: user queries "static" training data in LLM; LLM guesses something, then searches internet in real-time for data to support the guesses. This would be classified as "browsing" rather than trawling. (the searched data then get added back into the corpus, thus sadly sidestepping all the anti-AI trawling mechanisms) Kind of like the way a normal user would. The problem is, as others have already mentioned, how would the LLMs know what is a good answer versus a bad, when a "normal" user also has this issue?
- ericdotlee 1y agoWhat a lot of these journalists don't realize is ai tools are the internet funnels of the future. People use ChatGPT not Google to source info. The way you get results is begging these tools to search for specific bits of info in order to get visibility.
- rkozik1989 1y agoI've got a hard time believing most folks are clicking on the sources that LLM chatbots sometimes provide with their answers. Especially because they've been hard sold on the idea that AI chatbots are as smart as the smartest people on the planet. They're likely just going to do what normal folks do when smart people answer questions i.e. assume they're right and move on.
- skarz 1y agoI do. I treat ChatGPT output the same as Wikipedia content. I find what I am looking for then immediately open the source to confirm. I would never in a million years take ChatGPT output or Wikipedia content and blindly reference it without doing my due diligence.
- koolhead17 1y agoHow has Linkedin succeeded why rest failed?
- PolicyPhantom 1y agoFree-for-All was a natural assumption in the early internet, but in the age of AI, alignment with contracts and governance becomes essential. Technical capability alone is not enough — without mechanisms like licensing or audits to ensure legitimacy, such practices may prove socially unsustainable.