11 ms·

News publishers limit Internet Archive access due to AI scraping concerns

• blell 9mo ago
  That’s good. I don’t like archival sites. Let things disappear.
  • braebo 9mo ago
    Yea.. I’ve noticed data hoarding largely resembles yet-another form of death denialism.
• ninjagoo 9mo ago
  Publishers like The Guardian and NYT are blocking the IA/Wayback Machine. 20% of news websites are blocking both IA and Common Crawl. As an example, https://www.realtor.com/news/celebrity-real-estate/james-van-der-beek-texas-spicewood-ranch-sale-kids-wife/ https://www.realtor.com/news/celebrity-real-estate/james-van... is unarchivable, with IA being 429ed while the site is accessible otherwise.
  • trollbridge 9mo ago
    And whilst the IA will honour requests not to archive/index, more aggressive scrapers won't, and will disguise their traffic as normal human browser traffic. So we're basically decided we only want bad actors to be able to scrape, archive, and index.
    • JumpCrisscross 9mo ago
      > we're basically decided we only want bad actors to be able to scrape, archive, and index AI training will be hard to police. But a lot of these sites inject ads in exchange for paywall circumvention. Just scanning Reddit for the newest archive.is or whatever should cut off most of the traffic.
    • mananaysiempre 8mo ago
      > If you find yourself wondering, or just feeling, "Why is everyone I wind up dealing with an asshole?" you might want to consider the possibility that you have set up an asshole filter. https://siderea.dreamwidth.org/1209794.html https://siderea.dreamwidth.org/1209794.html
      • trollbridge 8mo ago
        It’s hard to filter out legit looking traffic tho.
  • fc417fc802 9mo ago
    Presumably someone has already built this and I'm just unaware of it, but I've long thought some sort of crowd sourced archival effort via browser extension should exist. I'm not sure how such an extension would avoid archiving privileged data though.
    • ajb 9mo ago
      That exists for court documents (RECAP) but I think they didn't have to solve the issue of privilege as PACER publishes unprivileged docs.
      • nxobject 9mo ago
        In particular, habeas petitions against DHS, and SSA appeals aren’t available online for public inspection: you have to go to a clerk’s office and pay for physical copies. (I think this may have been reasonable given the circumstances in past decades… not so now.)
  • nullhole 9mo ago
    Can you give a reference for The Guardian blocking IA? I just checked with an article from today - already archived, and a manual re-archive worked.
  • ccgreg 9mo ago
    That 20% number is for a limited list of relatively large news websites. If you include the long tail of news, the % of blocking is much smaller.
    • username223 9mo ago
      I'm part of that small but (hopefully) growing percentage, because Common Crawl is a deeply dishonest front for AI data scraping. Quoting Wikipedia: """ In November 2025, an investigation by technology journalist Alex Reisner for The Atlantic revealed that Common Crawl lied when it claimed it respected paywalls in its scraping and requests from publishers to have their content removed from its databases. It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies. """ My site is CC-BY-NC-SA, i.e. non-commercial and with attribution, and Common Crawl took a dubious position on whether fair use makes that irrelevant. They can burn.
      • ccgreg 9mo ago
        Did you see our reply? https://commoncrawl.org/blog/setting-the-record-straight-common-crawls-commitment-to-transparency-fair-use-and-the-public-good https://commoncrawl.org/blog/setting-the-record-straight-com... Also, if your site has CC-BY-NC-SA markings, we have preserved them.
        • username223 9mo ago
          Hopefully my site is no longer part of Common Crawl. I'm not interested in participating in your project, block CCBot in robots.txt, and have requested deletion of my data via your form.
          • ccgreg 9mo ago
            Did you see our reply? Edit: by which I mean, we sent you an email that explains what we did and how to verify it. Did you not receive an email reply? If not, please contact us again. Also, if your site has CC-BY-NC-SA markings, we have preserved them.
• OGEnthusiast 9mo ago
  If most of the Internet is AI-generated slop (as is already the case), is there really any value in expensing so much bandwidth and storage to preserve it? And on the flip side, I'd imagine the value of a pre-2022 (ChatGPT launch) Internet snapshot on physical media will probably increase astronomically.
  • ninjagoo 9mo ago
    Perhaps the AI slop isn't worth preserving, but the unarchivability of news and other useful content has implications for future public discourse, historians, legal matters and who knows what else. In the past libraries used to preserve copies of various newspapers, including on microfiche, so it was not quite feasible to make history vanish. With print no longer out there, the modern historical record becomes spotty if websites cannot be archived. Perhaps there needs to be a fair-use exception or even a (god forbid!) legal requirement to allow archivability? If a website is open to the public, shouldn't it be archivable?
    • phatfish 9mo ago
      Erm, there is still a newspaper stand in the supermarket I go to (Wallmart for the Americans). Not sure if the British library keeps a copy of the print news I see, but they should!
  • nicole_express 9mo ago
    The sites that are most valuable to preserve are likely the same ones that are most likely to put up barriers to archiving
• macinjosh 9mo ago
  We need something like SETI@home/Folding@home but for crawling and archiving the web or maybe something as simple as a browser extension that can (with permission) archive pages you view.
  • ryoshu 9mo ago
    This is a good idea. Not sure what ToS it would violate. But a good idea.
  • pclmulqdq 9mo ago
    Your TV probably does that, and you definitely gave it permission when you clicked "accept" on the terms.
    • macinjosh 9mo ago
      good thing I don't have a TV!
  • ninjagoo 9mo ago
    In the US at least, there is no expectation of privacy in public. Why should these websites that are public-facing get an exemption from that? Serving up content to the public should imply archivability. Sometimes it feels like ai-use concerns are a guise to diminish the public record. While on the other hand services like Ring or Flock are archiving the public forever.
    • sejje 9mo ago
      Ring and Flock are not a standard we should be striving towards. Their massive databases tracking citizens need to go.
  • deleted 9mo ago
    [deleted]
  • dunder_cat 9mo ago
    This exists although not in the traditional BOINC space, it's Archiveteam^1. I run two of their warrior^2 instances in my home k3s instance via the docker images. One of them is set to the "Team's choice" where it spends most of its time downloading Telegram chats. However, when they need the firepower for sites with imminent risk of closure, it will switch itself to those. The other one is set to their URL shortener project, "Terror of Tiny Town"^3. Their big requirement is you need to not be doing any DNS filtering or blocking of access to what it wants, so I've got the pod DNS pointed to the unfiltered quad9 endpoint and rules in my router to allow the machine it's running on to bypass my PiHole enforcement+outside DNS blocks. ^1 https://wiki.archiveteam.org/ https://wiki.archiveteam.org/ ^2 https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior ^3 https://wiki.archiveteam.org/index.php/URLTeam https://wiki.archiveteam.org/index.php/URLTeam
• derefr 9mo ago
  I wonder if these publishers would be more amenable to a private archiver that only serves registered academic / journalistic research projects (the way most physical private archives do), with a specific provision to never provide data to companies that would resell it or use it for training of generative models.
  • ninjagoo 9mo ago
    They probably have internal archives if they're smart; but that isn't accessible to the public. I think the issue isn't whether the data is archived, but whether that information is available to the public for the foreseeable future.
    • g-b-r 9mo ago
      They sure have archives of the newspapers, they're much less likely to have archives of what they publish online. And a local archive is one fire, business decision, poor technical choice etc away from getting permanently lost
  • eternauta3k 9mo ago
    They already have archives with online and printed articles which they license to libraries, because the libraries take care of rate limiting and limiting abuse.
  • coffeefirst 9mo ago
    Yes. Most publishers already do syndication deals. This is a fine idea. The problem with the LLMs is they capture the value chain and give back nothing. It didn’t have to be this way. It still doesn’t.
• Havoc 9mo ago
  Yup. Recently built something that needs to do low volume scraping. About 40% success rate - rest hits bot detection even on first try
  • ninjagoo 9mo ago
    Did you have rate limits built in? Ultimately scraping tools will need to mimic humans. Ironic. I wonder if bots/ai will need to build their own specialized internet for faster sharing of data, with human centered interfaces to human spaces.
    • fc417fc802 9mo ago
      IPFS and IPNS already exist.
    • Havoc 8mo ago
      Yeah it gets blocked literally on first fetch so not a rate limit issue.
• zeagle 9mo ago
  I mean why wouldn’t they? All their IP was scraped for at their own cost of hosting it for AI training. It further pulls away from their own business models as people ask the AI models the questions instead of reading primary sources. Plus it doesn’t seem likely they’ll ever be compensated for that loss given the economy is all in on AI. At least search engines would link back.
  • ninjagoo 9mo ago
    That is a good question. However, copyright exists (for a limited time) to allow for them to be compensated. AI doesn't change that. It feels like blocking AI-use is a ploy to extract additional revenue. If their content is regurgitated within copyright terms, yes, they should be compensated.
    • fc417fc802 9mo ago
      The problem is that producing a mix of personalized content that doesn't appear (at least on its face) to violate copyright still completely destroys their business model. So either copyright law needs to be updated or their business model does. Either way I'm fairly certain that blocking AI agent access isn't a viable long term solution.
      • ninjagoo 9mo ago
        > Either way I'm fairly certain that blocking AI agent access isn't a viable long term solution. Great point. If my personal AI assistant cannot find your product/website/content, it effectively may no longer exist! For me. Ain't nobody got the time to go searching that stuff up and sifting through the AI slop. The pendulum may even swing the other way and the publishers may need to start paying me (or whoever my gatekeeper is) for access to my space...
        • zeagle 8mo ago
          That’s a perspective I hadn’t considered. Although the whole thing stinks of middlemen extracting all the profit between producers and consumers e.g. ag sector by the laws won’t catch up or even force integration. Thanks!
• sejje 9mo ago
  This is a good thing, IMO. I am sad about link rot and old content disappearing, but it's better than everything be saved for all time, to be used against folks in the future.
  • otterley 9mo ago
    Consider the impact, though, on our ability to learn and benefit from history. If the records of people’s activities cannot be preserved, are we doomed to live in ignorance?
    • ninjagoo 9mo ago
      What's that famous quote - those who do not learn from history ... BUT, it's hard to learn from history if there's no history to learn...
    • sejje 9mo ago
      I don't think so. Most of my original creations were before the archiving started, and those things are lost. But they weren't the kind of history you learn and benefit from--nor is most of the internet. The truly important stuff exists in many forms, not just online/digital. Or will be archived with increased effort, because it's worth it.
      • nine_k 9mo ago
        Think about the stuff archeologists get to work with.
      • otterley 9mo ago
        Like it or not, the Internet is today’s store of record for a significant proportion—if not the majority—of the world’s activities. If you don’t want your bad behavior preserved for the historical record, perhaps a better answer is to not engage in bad behavior instead of relying on some sort of historical eraser.
        • sejje 9mo ago
• jruohonen 9mo ago
  It affects science too (and there you'd want solid archiving as much as possible). Increasingly, meta-data is full of errors and general purpose search engines for science are breaking down, including even things like Google Scholar. I suppose some big science publishers are blocking AI bots too.
  • ninjagoo 9mo ago
    > I suppose some big science publishers are blocking AI bots too. That's a travesty, considering that a huge chunk of science is public-funded; the public is being denied the benefits of what they're paying for, essentially.
    • galleywest200 9mo ago
      The public can still access the sites themselves.
      • ninjagoo 9mo ago
        > The public can still access the sites themselves. Indefinitely? Probably not. What about when a regime wants to make the science disappear?
        • pa7ch 9mo ago
          What has that got to do with blocking AI crawlers?
          • ninjagoo 9mo ago
            If it's publicly funded, why shouldn't AI crawlers have access to that data? Presumably those creating the AI crawlers paid taxes that paid for the science.
            • JumpCrisscross 9mo ago
• g-b-r 9mo ago
  This is awful, they need to at the very least allow private archivals. Maybe the Internet Archive might be ok to keeping some things private until x time passes; or they could require an account to access them
• _slih 9mo ago
  There's a compliance angle to this that nobody's talking about. Regulatory frameworks like SOC 2 and HIPAA require audit trails and evidence retention. A lot of that evidence lives at URLs. When a vendor's security documentation, a published incident response, or a compliance attestation disappears from the web and can't be archived, you've got a gap in your audit trail that no auditor is going to be happy about. I've seen companies fail compliance reviews because a third-party vendor's published security policy that they referenced in their own controls no longer exists at the URL they cited. The web being unarchivable isn't just a cultural loss. It's becoming a real operational problem for anyone who has to prove to an auditor that something was true at a specific point in time.
  • ninjagoo 9mo ago
    At some point Insurance is going to require companies to obtain paper copies of any documentation/policies, precisely to avoid this kind of situation. It may take a while to get there though. It'll probably take a couple of big insurance losses before that happens.
    • _slih 9mo ago
      Insurance is already moving that direction for cyber policies. Some underwriters now require screenshots or PDF exports of third-party vendor security attestations as part of the application process, not just URLs. The carriers learned the hard way that 'we linked to their SOC 2 landing page' doesn't hold up when that page disappears after an acquisition or rebrand.
      • pwg 9mo ago
        > when that page disappears after an acquisition or rebrand. Sadly, it does not even have to be an acquisition or rebrand. For most companies, a simple "website redo", even if the brand remains unchanged, will change up all the URL's such that any prior recorded ones return "not found". Granted, if the identical attestation is simply at a new url, someone could potentially find that new url and update the "policy" -- but that's also an extra effort that the insurance company can avoid by requiring screen shots or PDF exports.
• shevy-java 9mo ago
  > The Financial Times, for example, blocks any bot that tries to scrape its paywalled content, including bots from OpenAI, Anthropic, Perplexity, and the Internet Archive But then it was not really open content anyway. > When asked about The Guardian’s decision, Internet Archive founder Brewster Kahle said that “if publishers limit libraries, like the Internet Archive, then the public will have less access to the historical record.” Well - we need something like wikipedia for news content. Perhaps not 100% wikipedia; instead, wikipedia to store the hard facts, with tons of verification; and a news editorial that focuses on free content but in a newspaper-style, e. g. with professional (or good) writers. I don't know how the model could work, but IF we could come up with this then newspapers who have gatewalls to information would become less relevant automatically. That way we win long-term, as the paid gatewalls aren't really part of the open web anyway.
  • JumpCrisscross 9mo ago
    > it was not really open content anyway Practically no quality journalism is. > we need something like wikipedia for news Wikipedia editors aren’t flying into war zones.
    • fc417fc802 9mo ago
      Statistically, at least a few of them live in war zones. And I'm sure some of them would fly in to collect data if you paid them for it.
      • JumpCrisscross 9mo ago
        > at least a few of them live in war zones Which is a valuable perspective. But it's not a subsitute for a seasoned war journalist who can draw on global experience. (And relating that perspective to a particular home market.) > I'm sure some of them would fly in to collect data if you paid them for it Sure. That isn't "a news editorial that focuses on free content but in a newspaper-style, e. g. with professional (or good) writers." One part of the population imagines journalists as writers. They're fine on free, ad-supported content. The other part understands that investigation is not only resource intensive, but also requires rare talent and courage. That part generally pays for its news. Between the two, a Wikipedia-style journalistic resource is not entertaining enough for the former and not informative enough for the latter. (Importantly, compiling an encyclopedia is principally the work of research and writing. You can be a fine Wikipedia–or scientific journal or newspaper–editor without leaving your room.)
• nananana9 9mo ago
  The silver lining is that it's increasingly not worth being archived as well.
  • Flavius 9mo ago
    Agreed. It’s mostly just disposable clickbait masquerading as journalism at this point. Outside of feeding people's FOMO, there's little content worth preserving for history.
  • idiotsecant 9mo ago
    We really lucked out existing at a time when the internet was a place for weirdos and enthusiasts. I think those days are well and done.
    • JuniperMesos 9mo ago
      The internet can't simultaneously be a place for weirdos and enthusiasts, and a vital part of the economy that everyone uses for a huge number of disparate things in daily life. Parts of the internet can be places for weirdos and enthusiasts, but spaces that cater to weirdos and enthusiasts are by necessity not popular or viral spaces.
• RajT88 9mo ago
  Proposed solution: Sell a "truck full of DAT tapes" type service to AI scrapers with snapshots of the IA. Sort of like the cloud providers have with "Data Boxes". It will fund IA, be cheaper than building and maintaining so many scrapers, and may relieve the pressure on these news sites.
  • atrus 9mo ago
    Even sites with that option already (like wikipedia) still report being hammered by scrapers. It's the full-funded aligned with the incompetent at work here.
  • digiown 9mo ago
    IA has always been in legal jeopardy without offering paid access. For that to work we need to get rid of copyright first.
    • RajT88 9mo ago
      Or offer it in countries with lax copyright. The industry will find ways to work around it. But - as another poster pointed out - Wikipedia offers this, and still gets hammered by scrapers. Why buy when free, I guess?
• JumpCrisscross 9mo ago
  Let’s be honest, one of the most-common uses of these archive sites has been paywall circumvention. An academics-only archive might make sense, or one that is mutually-owned and charges a fee for lookup. But a public archive for content that costs money to make obviously doesn’t work.
  • lurking_swe 9mo ago
    if that’s the real motive, why don’t they allow access to scrape content after some period? when that news is not as relevant. For example after 6 months.
    • JumpCrisscross 9mo ago
      > why don’t they allow access to scrape content after some period? when that news is not as relevant. For example after 6 months I belive many publications used to do this. The novel threat is AI training. It doesn't make sense to make your back catalog de facto public for free like that. There used to be an element of goodwill in permitting your content to be archived. But if the main uses are circumventing compensation and circumventing licensing requirements, that goodwill isn't worth much.
    • otterley 9mo ago
      Enabling research is a business model for many publications. Libraries pay money for access to the publishers’ historical archives. They don’t want to cannibalize any more revenue streams; they’re already barely still operating as it is.
      • lurking_swe 9mo ago
        i see, i didn’t consider this angle. thanks for pointing that out.
• zachlatta 9mo ago
  The death of trust on the cloud.
• f33d5173 9mo ago
  So instead of scraping IA once, the AI companies will use residential proxies and each scrape the site themselves, costing the news sites even more money. The only real loser is the common man who doesn't have the resources to scrape the entire web himself. I've sometimes dreamed of a web where every resource is tied to a hash, which can be rehosted by third parties, making archival transparent. This would also make it trivial to stand up a small website without worrying about it get hug-of-deathed, since others would rehost your content for you. Shame IPFS never went anywhere.
  • terminalshort 9mo ago
    But don't you have to sign a license agreement that prohibits scraping in order to purchase a subscription that allows you to bypass the paywall?
  • fartfeatures 9mo ago
    IPFS was an attempt at this: https://en.wikipedia.org/wiki/InterPlanetary_File_System https://en.wikipedia.org/wiki/InterPlanetary_File_System
    • lukeasch21 9mo ago
      Coincidentally most of the funding towards IPFS development dried up because the VC money moved onto the very technology enabling these problems...
    • Seattle3503 9mo ago
      Is there a good post-mortem of IPFS out there?
      • iririririr 9mo ago
        What do you mean? It is alive and "well". Just extremely slow now that interest waned.
        • __MatrixMan__ 9mo ago
• WesBrownSQL 9mo ago
  As someone who has been dealing with SOC 2, HIPAA, ISO 9001, etc., for years, I have always maintained copies of the third-party agreements for all of our downstream providers for compliance purposes. This documentation is collected at the time of certification, and our policies always include a provision for its retrieval on schedule. The problem is when you certify their policy said X and were in compliance, they quietly change that and don't send proper notification downstream to us, and captain lawsuit comes by, we have to be able to prove that they did claim they were in compliance and the time we certified. We don't want to rely on their ability to produce that documentation. We can't prove that it wasn't tampered with, or that there is a chain of custody for their documentation and policies. If I wanted to use a vendor that wouldn't provide that information, then I didn't use them. Welcome to the world of highly regulated industries.
  • tadfisher 9mo ago
    What does this have to do with news sites blocking AI scrapers and the Internet Archive? Are you a bot?
• Brian_K_White 9mo ago
  Time for a crowd source plugin that relays copies of what individuals view right from the browser. Users control what sites they want to allow it to record so no privacy worries, especially assuming the plugin is open source. No automated crawling. The plugin does not drive the users browser to fetch things. Just whatever a user happens to actually view on their own, some percentage of those views from the activated domains gets submitted up to some archive. Not every view, just like maybe 100 people each submit 1% of views, and maybe it's a random selection or maybe it's weighted by some feedback mechanism where the archive destination can say "Hey if the user views this particular url, I still don't have that one yet so definitely send that one if you see it rather than just applying the normal random chance" Not sure how to protect the archive itself or it's operators.
  • digiown 9mo ago
    SingleFile does the archiving fairly well. > no privacy worries This is harder than you might expect. Publishing these files is always risky because sites can serve you fingerprinting data, like some hidden HTML tag containing your IP and other identifiers.
    • Brian_K_White 9mo ago
      oof good point
    • 8bitsrule 9mo ago
      >SingleFile does the archiving fairly well. As does Tranquility Reader, if you're interested only in the primary content of the page ... and, usually, in a much smaller footprint ... with a PDF option.
    • devsda 8mo ago
      collecting from multiple sources to find and scrub out the differences may make fingerprinting somewhat ineffective ?
  • nerdsniper 9mo ago
    For a historical archive, the issue with this is that it could be difficult to ensure that the data being sent from users' devices wasn't modified in some way, leading to an inaccurate archival copy.
• jackfranklyn 9mo ago
  [flagged]
• cdrnsf 9mo ago
  This is a natural response to AI companies plundering the web to enrich themselves and provide no benefit to the sites being scraped.
  • CivBase 9mo ago
    Seems more like an easy excuse to shut down a means for people to bypass their paywalls. It would be trivial for AI companies to continue getting this data without using the Internet Archive.
    • cdrnsf 9mo ago
      I imagine that's a consideration, but there's plenty of pushback against AI companies scraping outside of this.
• upboundspiral 9mo ago
  I feel like a government funded search engine would resolve a lot of the issues with the monetized web. The purpose of a search engine is to display links to web pages, not the entire content. As such, it can be argued it falls under fair use. It provides value to the people searching for content and those providing it. However we left such a crucially important public utility in the hands of private companies, that changed their algorythms many times in order to maximize their profits and not the public good. I think there needs to be real competition, and I am increasingly becoming certain that the government should be part of that competition. Both "private" companies and "public" governement are biased, but are biased in different ways, and I think there is real value to be created in this clash. It makes it easier for individuals to pick and choose the best option for themselves, and for third independent options to be developed. The current cycle of knowledge generation is academia doing foundational research -> private companies expanding this research and monetizing it -> nothing. If the last step was expanded to the government providing a barebones but useable service to commodotize it, years after private companies have been able to reap immense profits, then the capabilities of the entire society are increased. If the last step is prevented, then the ruling companies turn to rentseeking and sitting on their lawrels, turn from innovating to extracting.
  • LPisGood 9mo ago
    The government having the power to curate access to information seems bad. You could try to separate it as an independent agency, but as the current US administration is showing, that’s not really a thing.
    • upboundspiral 9mo ago
      The idea is that the government is biased towards hiding certain information and private companies are biased towards hiding a different set. While unlikely, the ideal would be for the government to provide a foundational open search infrastructure that would allow people to build on it and expand it to fit their needs in a way that is hard to do when a private companies eschews competition and hides its techniques. Perhaps it would be better for there to be a sanctioned crawler funded by the government, that then sells the unfiltered information to third parties like google. This would ensure IP rights are protected while ensuring open access to information.
• mellosouls 9mo ago
  editorialised. Original title (submitted previously a few times correctly by others): News publishers limit Internet Archive access due to AI scraping concerns
• notepad0x90 9mo ago
  The internet isn't so simple anymore. I think it's important to separate commercial websites from non-commercial ones. Commercial sites shouldn't be expected to be achievable to begin with, unless it's part of their business model. A lot of sites (like reddit), started of as ad-supported sites, but now they're commercial (not just post-IPO, but accept payments and sell things to/from consumers). Even for ad-supported sites, there is a difference between ad-supported non-profit, and sites that exist to generate revenue from ads. As in, the primary purpose of the site is to generate ad-revenue, the content is just a means to that end. I've said it before, and I'll say it again: The main issue is not design patterns, but lack of acceptable payment systems. The EU with their dismantling of visa and mastercard now have the perfect opportunity to solve this, but I doubt they will. They'll probably just create a european wechat.
• yellowapple 9mo ago
  Framing this as some anti-AI thing is wild. The simpler, more obvious, and more evidenced reason for this is that these sites want to make money with ads and paywalls that an archived copy tends to omit by design. Scapegoating AI lets them pretend that they're not the greedy bad guys here — just like how the agricultural sector is hell-bent on scapegoating AI (and lawns, and golf courses, and long showers, and free water at restaurants) for excess water consumption when even the worst-offending datacenters consume infinitesimally-tiny fractions of the water farms in their areas consume.
  • JuniperMesos 9mo ago
    Yeah I assume what the news publishers actually care about is the thing where, when someone posts a paywalled news article on hacker news, one of the first comments is invariably a link to an archive site that bypasses the paywall so people can read it without paying for it. > just like how the agricultural sector is hell-bent on scapegoating AI (and lawns, and golf courses, and long showers, and free water at restaurants) for excess water consumption when even the worst-offending datacenters consume infinitesimally-tiny fractions of the water farms in their areas consume. When I learned about how much water agriculture and industry uses in the state of California where I live, I basically entirely stopped caring about household water conservation in my daily life (I might not go this far if I had a yard or garden that I watered, but I don't where I currently live). If water is so scarce in an urban area that an individual human taking a long shower or running the dishwasher a lot is at all meaningful, then either the municipal water supply has been badly mismanaged, or that area is too dry to support human settlement; and in either case it would be wise to live somewhere else.
• bmiekre 9mo ago
  Explain it to me like I’m 5, why is ai scraping the way back machine bad?
• colesantiago 9mo ago
  I fear that these news publishers would come after RSS next as I see hundreds of AI companies misusing the terms of the news publishers's RSS feed for profit on mass scraping. They do not care and we will be all worse off for it if these AI companies keep continuing to bombard news publishers RSS feeds. It is a shame that the open web as we know it is closing down because of these AI companies.
• daniel31x13 9mo ago
  I maintain an open-source project called Linkwarden and this exact discussion is one of the reasons why it exists, teams needed a way to preserve referenced URLs reliably without having to depend on external services. It stores webpages in multiple formats (HTML snapshot, screenshot, PDF snapshot, and a fully dedicated reader view) so you’re not relying on a single fragile archive method. There’s both a hosted cloud plan [1] which directly supports the project, and a fully self-hosted option [2], depending on how much control you need over storage and retention. [1]: https://linkwarden.app https://linkwarden.app [2]: https://github.com/linkwarden/linkwarden https://github.com/linkwarden/linkwarden
  • iririririr 9mo ago
    Neat. How does the archive.org integration works? Does it just POST the url to them for them to fetch? Or is there any integration/trust to store what you already fetched on the client directly on their archives?
    • daniel31x13 9mo ago
      > Does it just POST the url to them for them to fetch? Correct.
  • raybb 9mo ago
    Linkwarden is awesome and with the singlefile extension it's pretty easy to store things you can see but the scraper gets blocked on. One question, what's your stance on adding a way to mark articles as read or "archive" them like other apps that are branded a bit more as storing things to read later. You can technically do something similar with tags but it's a bit clunky of a UX.
    • daniel31x13 9mo ago
      Thanks! At the moment we’re focused on archiving rather than read-later workflows, but this is great feedback. I’ve already added it to the feature requests list.
• holoduke 9mo ago
  The end of traditional news sites is coming. At least for the newspaper websites. Future mcp like systems will generate on the fly newstites in your desired style and content. Journalists will have some kind of paid per view model provided by these gpt like platforms which of course take a too big of a chunk. I can't imagine a WSJ is able to survive.
• gosub100 9mo ago
  But wait, I thought AI was so great for all industries? Publishers can have AI-generated articles, and instantly fix grammar problems, And translate it seamelessly to every language, and even use AI-generated images where appropriate to enrich the article. It was going to make us all so productive? What happened? Why would you want to _block_ AI from ingesting the material?
• xannabxlle 9mo ago
  My first impression is that news companies don't want their content scraped for copyright reasons, and roundaboutly scapegoating AI
  • spiderfarmer 9mo ago
    As a website owner I hate the fact that more than 90% of my traffic is now bots, fake bots, bots masquerading as real visitors and real visitors who try try to use my platform to spam others. Now AI companies are using residential proxies to get around the obvious countermeasures, I have resorted to blocking all countries that are not my target audience. It really sucks. The internet is terminally ill.
    • wildrhythms 8mo ago
      Every day we inch closer to a 'digital passport' gatekeeping system.
• sunaookami 9mo ago
  Yeah sure, "AI scraping concerns". No, they don't want to get caught secretly editing and deleting articles.
  • IshKebab 9mo ago
    It's obviously not that, or they would have done this years ago. It very clearly is AI scraping concerns. Their content has new value because it's high quality text that AI scrapers want, and they don't want to give it away for free via the internet archive. They will announce official paid AI access plans soon. Bookmark my works.
• tl2do 9mo ago
  The issue of digital decay and publishers blocking archiving efforts is indeed concerning. It's especially striking given that news publishers, perhaps more than any other entity, have profoundly benefited from the vast accumulation of human language and cultural heritage throughout history. Their very existence and influence are built upon this foundation. To then, in an age where information preservation is more critical than ever (and their content is frequently used for AI training), actively resist archiving or demand compensation for their contributions to the collective digital record feels disingenuous, if not outright shameless. This stance ultimately harms the public good and undermines the long-term accessibility of our shared knowledge and historical narrative.
• zmmmmm 9mo ago
  Dear news publications - if you aren't willing to accept an independent record of what you published, I can't accept your news. It's a critical piece of the framework that keeps you honest. I don't care if you allow AI scraping either way, but you have to facilitate archival of your content - independently, not under your own control.
  • germandiago 9mo ago
    First thing that came to my mind went along the same reasoning.
    • zeta0134 9mo ago
      The second thing that came to mind was paywall evasion. Any time a news article behind a paywall gets posted here, someone in the comments has the archive link ready to go, because of course they do. The incentives for online news are really wacky just to begin with. A coin at the convenience store for the whole dang paper used to be the simplest thing in the world.
      • saint_fiasco 9mo ago
        I suppose that could be solved with a delay. Limit internet archive for articles that are less than a week old.
        • MonkeyClub 9mo ago
          That would diminish archival accuracy, an outlet could amend the text without third party proof.
          • jasonfarnon 9mo ago
            I don't see the connection to adding the delay. I think the suggestion was to have a snapshot at time of publication but wait a week to make it public.
• humanji 9mo ago
  I’m coming at this from a founder/product angle, not a technical one, so excuse the naive framing. What worries me isn’t scraping itself, but the second-order effects. If large parts of the web become intentionally unarchivable, we’re slowly losing a shared memory layer. Short-term protection makes sense, but long-term it feels like knowledge erosion. Genuinely curious how people here think about preserving public knowledge without turning everything into open season for mass scraping.
  • bsimpson 9mo ago
    This partially feels like an intentional pendulum swing from Twitter/Facebook cancel culture and other forms of policing. I'm thinking in particular about the rise of platforms like Discord where being opaque to search/archiving is seen as a feature. Being gatekept and ephemeral makes people more comfortable sharing things that might get a takedown notice on other platforms, and it's hard for people who don't like you in the future to try to find jokes/quotes they don't like to damage your future reputation. Clearly very different than news articles going offline, but I do think there's been a vibe shift around the internet. People feel overly surveilled in daily life, and take respite in places that make surveillance harder.
• GeoAtreides 9mo ago
  <richevans>How does it feel to live long enough to see all your favorite sites go down in flames?</richevans>
• nxobject 9mo ago
  Brewster’s concerns about the historical record are real and will eventually affect news orgs: their journalism may as well be ephemeral now without separate archiving. If a Wikipedia contributor, for example has to jump through extra hoops to get a stable link of a Times article, why wouldn’t they end up choosing an equally reliable WaPo article instead? Tragedy of the commons.
  • Mindwipe 9mo ago
    Given the Times and the Guardian are British they will be archived by the British Library, as it's a legal obligation. That doesn't mean anything American library that doesn't pay authors Public Lending Right fees gets to.
• ccgreg 9mo ago
  Prof. Jeff Jarvis speaking about copyright for news in front of Congress: https://www.youtube.com/watch?v=tX26ijBQs2k https://www.youtube.com/watch?v=tX26ijBQs2k
• fn-mote 9mo ago
  Too little, too late. AI scrapers are better and better at acting human. AI scrapers already have a massive corpus; the marginal value of today’s need is low and will remain so long after access is cut off. When they manage to block archive.is too then I will believe they are at least a little serious.
  • visarga 9mo ago
    I think people forget one thing - LLMs don't even need to scrape, we copy paste and put articles and documents right into their mouth, they only need to keep the mouth open. Copy pasted content is also preselected manually, might filter out some garbage as well. A subscriber opens the FT, reads an article about semiconductor export controls, pastes it into Claude to ask "what does this mean for my portfolio?" - the FT's content just entered a model's reasoning process, got synthesized with other knowledge, and produced derivative value. No scraper was involved. The paywall was respected. The subscriber paid. And yet the publisher's content was "consumed" by an AI in exactly the way they're trying to prevent.
• RsAaNtDoYsIhSi 9mo ago
  BitCoin fixes this.
• cmiles74 9mo ago
  Isn't the real problem here the unscrupulous AI scrapers? These sites want to be paid for their content to be used for AI training, if this same content is scraped by the Internet Archive the AI companies can get the content for free. It's unfortunate that this undermines the usefulness of the Internet Archive, I don't see an alternative. IMHO, we'll soon see these AI scrapers cease to advertise themselves leading to sites like the NY Times trying to blacklist IP ranges as this battle continues. Fun times ahead!
• anjel 9mo ago
  Punishing archive.org for archive.today's sins
• lofaszvanitt 9mo ago
  The problem with the Internet Archive is that site owners are not presented the data by the archive who, when and what did they access from their archive. This must be an absolute must. So if you own monkey.com, then after showing site ownership you must have access to every access data related to your site. Problem solved.
• a2128 9mo ago
  This is terrible for transparency and record keeping. X has also blocked internet archive access under similar concerns, but the end result was that now it's very difficult to tell who said what and when, posts can be deleted or edited, and no public figure can be held accountable for something wrong they said, or making contradictory statements over time, via a trustworthy archive. You just have to rely on screenshots that may or may not have been fabricated, and maybe nobody's even captured a screenshot. If it's a public figure you normally trust, versus some random people's screenshots, of course you're gonna dismiss the screenshots as fake. It feels almost intentional to bring the platform into the dark ages.
• nurettin 8mo ago
  The current scraping schemes look inane. IA and other data vendors would probably much prefer if slop companies simply paid them to divert logs to on-site machines instead. Less harmful than scraping, IA gets paid and slop gets what it wants.
• 1vuio0pswjnm7 8mo ago
  "Currently, however, the Internet Archive does not disallow any specific crawlers through its robots.txt file, including those of major AI companies. As of January 12, [2026,] the robots.txt file for archive.org read: "Welcome to the Archive! Please crawl our files. We appreciate it if you can crawl responsibly. Stay open!" Shortly after we inquired about this language, it was changed. The file now reads, simply, "Welcome to the Internet Archive!""
• throwaway_20357 8mo ago
  If news sites opt out of being archived by the Internet Archive, are their archives available anywhere or just lost? Will there be no way to access the headlines of a certain day or the reporting about a certain topic in the past even for research or scientific purposes?
• gymbeaux 8mo ago
  I’m seeing more and more (re)CAPTCHAs as time goes on. It’s a shame because programmatic access to web content is very nice to have. There must have been a point, many moons ago now, when the internet had as much freely accessible data as it’s probably ever going to have. Probably the day before Zillow blocked all web scraping and API access without forking up $$$. Reddit is a more recent example of how arbitrarily restricting data that was once freely available purely for $$$ has made the world and the internet worse.