4 ms·

Publishers like The Guardian and NYT are blocking the IA/Wayback Machine. 20% of news websites are blocking both IA and Common Crawl. As an example, https://www

Publishers like The Guardian and NYT are blocking the IA/Wayback Machine. 20% of news websites are blocking both IA and Common Crawl. As an example, https://www.realtor.com/news/celebrity-real-estate/james-van-der-beek-texas-spicewood-ranch-sale-kids-wife/ https://www.realtor.com/news/celebrity-real-estate/james-van... is unarchivable, with IA being 429ed while the site is accessible otherwise.

trollbridge 9mo ago
And whilst the IA will honour requests not to archive/index, more aggressive scrapers won't, and will disguise their traffic as normal human browser traffic. So we're basically decided we only want bad actors to be able to scrape, archive, and index.
- JumpCrisscross 9mo ago
  > we're basically decided we only want bad actors to be able to scrape, archive, and index AI training will be hard to police. But a lot of these sites inject ads in exchange for paywall circumvention. Just scanning Reddit for the newest archive.is or whatever should cut off most of the traffic.
- mananaysiempre 8mo ago
  > If you find yourself wondering, or just feeling, "Why is everyone I wind up dealing with an asshole?" you might want to consider the possibility that you have set up an asshole filter. https://siderea.dreamwidth.org/1209794.html https://siderea.dreamwidth.org/1209794.html
  - trollbridge 8mo ago
    It’s hard to filter out legit looking traffic tho.
fc417fc802 9mo ago
Presumably someone has already built this and I'm just unaware of it, but I've long thought some sort of crowd sourced archival effort via browser extension should exist. I'm not sure how such an extension would avoid archiving privileged data though.
- ajb 9mo ago
  That exists for court documents (RECAP) but I think they didn't have to solve the issue of privilege as PACER publishes unprivileged docs.
  - nxobject 9mo ago
    In particular, habeas petitions against DHS, and SSA appeals aren’t available online for public inspection: you have to go to a clerk’s office and pay for physical copies. (I think this may have been reasonable given the circumstances in past decades… not so now.)
nullhole 9mo ago
Can you give a reference for The Guardian blocking IA? I just checked with an article from today - already archived, and a manual re-archive worked.
ccgreg 9mo ago
That 20% number is for a limited list of relatively large news websites. If you include the long tail of news, the % of blocking is much smaller.
- username223 9mo ago
  I'm part of that small but (hopefully) growing percentage, because Common Crawl is a deeply dishonest front for AI data scraping. Quoting Wikipedia: """ In November 2025, an investigation by technology journalist Alex Reisner for The Atlantic revealed that Common Crawl lied when it claimed it respected paywalls in its scraping and requests from publishers to have their content removed from its databases. It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies. """ My site is CC-BY-NC-SA, i.e. non-commercial and with attribution, and Common Crawl took a dubious position on whether fair use makes that irrelevant. They can burn.
  - ccgreg 9mo ago
    Did you see our reply? https://commoncrawl.org/blog/setting-the-record-straight-common-crawls-commitment-to-transparency-fair-use-and-the-public-good https://commoncrawl.org/blog/setting-the-record-straight-com... Also, if your site has CC-BY-NC-SA markings, we have preserved them.
    - username223 9mo ago
      Hopefully my site is no longer part of Common Crawl. I'm not interested in participating in your project, block CCBot in robots.txt, and have requested deletion of my data via your form.
      - ccgreg 9mo ago
        Did you see our reply? Edit: by which I mean, we sent you an email that explains what we did and how to verify it. Did you not receive an email reply? If not, please contact us again. Also, if your site has CC-BY-NC-SA markings, we have preserved them.

trollbridge 9mo ago

And whilst the IA will honour requests not to archive/index, more aggressive scrapers won't, and will disguise their traffic as normal human browser traffic. So we're basically decided we only want bad actors to be able to scrape, archive, and index.

JumpCrisscross 9mo ago
> we're basically decided we only want bad actors to be able to scrape, archive, and index AI training will be hard to police. But a lot of these sites inject ads in exchange for paywall circumvention. Just scanning Reddit for the newest archive.is or whatever should cut off most of the traffic.
mananaysiempre 8mo ago
> If you find yourself wondering, or just feeling, "Why is everyone I wind up dealing with an asshole?" you might want to consider the possibility that you have set up an asshole filter. https://siderea.dreamwidth.org/1209794.html https://siderea.dreamwidth.org/1209794.html
- trollbridge 8mo ago
  It’s hard to filter out legit looking traffic tho.

fc417fc802 9mo ago

Presumably someone has already built this and I'm just unaware of it, but I've long thought some sort of crowd sourced archival effort via browser extension should exist. I'm not sure how such an extension would avoid archiving privileged data though.

ajb 9mo ago
That exists for court documents (RECAP) but I think they didn't have to solve the issue of privilege as PACER publishes unprivileged docs.
- nxobject 9mo ago
  In particular, habeas petitions against DHS, and SSA appeals aren’t available online for public inspection: you have to go to a clerk’s office and pay for physical copies. (I think this may have been reasonable given the circumstances in past decades… not so now.)

nullhole 9mo ago

Can you give a reference for The Guardian blocking IA? I just checked with an article from today - already archived, and a manual re-archive worked.

ccgreg 9mo ago

That 20% number is for a limited list of relatively large news websites. If you include the long tail of news, the % of blocking is much smaller.

username223 9mo ago
I'm part of that small but (hopefully) growing percentage, because Common Crawl is a deeply dishonest front for AI data scraping. Quoting Wikipedia: """ In November 2025, an investigation by technology journalist Alex Reisner for The Atlantic revealed that Common Crawl lied when it claimed it respected paywalls in its scraping and requests from publishers to have their content removed from its databases. It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies. """ My site is CC-BY-NC-SA, i.e. non-commercial and with attribution, and Common Crawl took a dubious position on whether fair use makes that irrelevant. They can burn.
- ccgreg 9mo ago
  Did you see our reply? https://commoncrawl.org/blog/setting-the-record-straight-common-crawls-commitment-to-transparency-fair-use-and-the-public-good https://commoncrawl.org/blog/setting-the-record-straight-com... Also, if your site has CC-BY-NC-SA markings, we have preserved them.
  - username223 9mo ago
    Hopefully my site is no longer part of Common Crawl. I'm not interested in participating in your project, block CCBot in robots.txt, and have requested deletion of my data via your form.
    - ccgreg 9mo ago
      Did you see our reply? Edit: by which I mean, we sent you an email that explains what we did and how to verify it. Did you not receive an email reply? If not, please contact us again. Also, if your site has CC-BY-NC-SA markings, we have preserved them.