3 ms·

Am I correct that this has come about because archive.org respects robots.txt and these sites have blocked their crawler from indexing their sites? I'm not sur

Am I correct that this has come about because archive.org respects robots.txt and these sites have blocked their crawler from indexing their sites? I'm not sure how to articulate my thoughts on this exactly, other than to say it's disappointing that doing the right thing (i.e. respecting robots.txt) is rewarded with the burden of soliciting responses to a petition while at the same time others are rewarded with profit for ignoring those same directives.

cmeacham98 5mo ago
Correct. Example snippet from the nytimes.com robots.txt: User-agent: archive.org_bot Disallow: /
- joecool1029 5mo ago
  Which they don’t respect. I’ve had it for my blog for years and they still added it to wayback machine, see my last comment for their official announcement of the ignore robots.txt policy, it is not new.
  - socalgal2 5mo ago
    robots.txt means they shouldn't auto-scan your site. Any user though can go to the wayback machine and type in a URL and the wayback machine will read that URL. That was the intent of robots.txt (don't scan) not (don't read period). It's spelled out in the spec for robots.txt
    - keane 5mo ago
      The <meta name="robots"> tag and robots.txt serve different roles: robots.txt controls crawling, while the robots meta tag influences indexing and other behavior. https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/meta/name/robots https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... I wonder how archive.org_bot behaves when <meta name="robots" content="noindex, noarchive, nocache" /> is present.
      - socalgal2 5mo ago
        The person above those is complaining about entries in their logs from bots. A robot can't read a tag without first reading the document. So sure, if they're a good bot they might not store the results but the server's logs will still show the bot's GET request.
  - ninjagoo 5mo ago
    > I’ve had it for my blog for years Just out of curiosity, why don't you want your public blog archived? not questioning, just trying to understand the logic/motivations? Also, I think you're being unfairly downvoted.
- mjmas 5mo ago
  Is there a difference between that and User-agent: ia_archiver ?
  - keane 5mo ago
    That was operated by Alexa Internet and the results powered the Internet Archive (same founder) presumably until their acquisition: https://web.archive.org/web/20140528103516/https://alexa.zendesk.com/hc/en-us/articles/200450194-Alexa-s-Web-and-Site-Audit-Crawlers https://web.archive.org/web/20140528103516/https://alexa.zen... https://en.wikipedia.org/wiki/Alexa_Internet https://en.wikipedia.org/wiki/Alexa_Internet
Paracompact 5mo ago
Don't know if it helps your musings at all, but there's a good chance that if a high-profile crawler like archive.org disrespected their robots.txt, that archive.org would be faced with lawsuits (or some other form of pressure). This is not merely the most moral move; rather it is the only sensible move. The only reason "others are rewarded with profit" in cases like these are because pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating.
- GolfPopper 5mo ago
  >pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating I think you're looking at the wrong end of the spectrum there. It's some of the biggest players who flaunt the rules. "Several AI companies said to be ignoring robots dot txt exclusion, scraping content without permission: report" (2024) https://www.tomshardware.com/tech-industry/artificial-intelligence/several-ai-companies-said-to-be-ignoring-robots-dot-txt-exclusion-scraping-content-without-permission-report https://www.tomshardware.com/tech-industry/artificial-intell...
  - Paracompact 5mo ago
    Fair point. Being small and shadowy is a sufficient condition to avoid litigation, but not a necessary one. Another sufficient condition is having billions of dollars to throw around. Unfortunately, archive.org is well known, well loved, and fundamentally harmless.
    - fragmede 5mo ago
      > fundamentally harmless. This is going to go in a boring direction with an argument thread that's been made since Internet time immemorial, and before. The argument goes: Pirating articles off nyt.com leads to lost sales of subscriptions, so it's not harmless. The response is, inevitably, no it doesn't, it leads to more sales. Or, people who weren't going to pay weren't going to pay anyway, so might as well give it to them for free, and be happy (as the NYT) for free advertising. And then the follow up, "No, it's a lost sale and journalism needs the money." HN is for thoughtful and substantive discussion, not for rehashing the same boring argument we've all read a thousand times. So my question isn't which camp is right. Both camps are firm in their beliefs. Copyright infringement is fine, copyright infringement is not. My question is in today's AI-fueled digital hellscape, how do we support journalists and the arts? If journalism only exists because eg Jeff Bezos pays for the Washington Post, we're going to get biased reporting (which has existed since long before the Internet); If art only exists because the artists come from rich families or have patrons like the Renaissance era, is society better off?
Gigachad 5mo ago
It's because they want to restrict AI companies from stealing content, but they can't do it if internet archive proxies it all for them. All of the LLMs would be massively less useful if it wasn't for scraping the latest news.
- stephen_g 5mo ago
  LLMs have other ways of accessing the content, they don’t need the Web Archive. Every LLM company can afford to spin up a new subscriber account every day, proxying to appear different IPs from all sorts of ASNs, do some crawling until the account gets banned, and then do it again, and again, and again.
  - overfeed 5mo ago
    > LLMs have other ways of accessing the content, they don’t need the Web Archive. What's the conclusion from this train if thought? Just because some burglars can pick locks doesn't mean you should leave your front door unlocked. Locking a door (or robots.txt) is how one can establish mens rea for those who bypass the barrier.
    - AnthonyMouse 5mo ago
      This is like arguing that services can't provide access to libraries that provide public WiFi because it would give the public legal permission to pirate TV shows. They're two unrelated things. And then some members of the public argue that they're making fair use rather than pirating anything, but that still has nothing to do with the library.
    - stephen_g 5mo ago
      But as I understand it, the Web Archive does respect robots.txt, while LLM scrapers absolutely do not and use all sorts of dodgy methods to get around it already... The actual root cause is that we're allowing LLM companies to completely disregard copyright laws for their profit. Whether the LLM companies scrape the Web Archive or the original source doesn't change the copyright infringement implications in any way, and cutting off the web archive doesn't practically change anything (because as I understand, LLM scraping is already prolific all over the web).
joecool1029 5mo ago
No, archive.org does NOT respect robots.txt. You need to reach out to them directly and ask your site not be included: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- input_sh 5mo ago
  Aren't you choosing to ignore something very specific specified in that article? Why do you make it seem that article implies it's their overall policy? > A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org).
  - joecool1029 5mo ago
    > Aren't you choosing to ignore something very specific specified in that article? Of course not, did you ignore the lines right after? “As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.” The announcement is from 9 years ago. I already mentioned they ignored the robots.txt for my own blog.
- LocalH 5mo ago
  I'd rather they disregard robots.txt than the opposite situation, where someone does not use robots.txt on a domain to allow IA to archive it, then for whatever reason the domain lapsed and got swooped up by a parker who then subsequently adds a robots.txt blocking IA from the whole site, which would have then caused IA to remove all historical archives of that domain from public view.
  - pseudalopex 5mo ago
    Hiding old archives when robots.txt changed was a problem Internet Archive created and could have fixed any time.
userbinator 5mo ago
It's the same idiocy that DRM created. Be a pirate, because a pirate is free...
deleted 5mo ago
[deleted]

cmeacham98 5mo ago

Correct. Example snippet from the nytimes.com robots.txt: User-agent: archive.org_bot Disallow: /

joecool1029 5mo ago
Which they don’t respect. I’ve had it for my blog for years and they still added it to wayback machine, see my last comment for their official announcement of the ignore robots.txt policy, it is not new.
- socalgal2 5mo ago
  robots.txt means they shouldn't auto-scan your site. Any user though can go to the wayback machine and type in a URL and the wayback machine will read that URL. That was the intent of robots.txt (don't scan) not (don't read period). It's spelled out in the spec for robots.txt
  - keane 5mo ago
    The <meta name="robots"> tag and robots.txt serve different roles: robots.txt controls crawling, while the robots meta tag influences indexing and other behavior. https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/meta/name/robots https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... I wonder how archive.org_bot behaves when <meta name="robots" content="noindex, noarchive, nocache" /> is present.
    - socalgal2 5mo ago
      The person above those is complaining about entries in their logs from bots. A robot can't read a tag without first reading the document. So sure, if they're a good bot they might not store the results but the server's logs will still show the bot's GET request.
- ninjagoo 5mo ago
  > I’ve had it for my blog for years Just out of curiosity, why don't you want your public blog archived? not questioning, just trying to understand the logic/motivations? Also, I think you're being unfairly downvoted.
mjmas 5mo ago
Is there a difference between that and User-agent: ia_archiver ?
- keane 5mo ago
  That was operated by Alexa Internet and the results powered the Internet Archive (same founder) presumably until their acquisition: https://web.archive.org/web/20140528103516/https://alexa.zendesk.com/hc/en-us/articles/200450194-Alexa-s-Web-and-Site-Audit-Crawlers https://web.archive.org/web/20140528103516/https://alexa.zen... https://en.wikipedia.org/wiki/Alexa_Internet https://en.wikipedia.org/wiki/Alexa_Internet

Paracompact 5mo ago

Don't know if it helps your musings at all, but there's a good chance that if a high-profile crawler like archive.org disrespected their robots.txt, that archive.org would be faced with lawsuits (or some other form of pressure). This is not merely the most moral move; rather it is the only sensible move. The only reason "others are rewarded with profit" in cases like these are because pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating.

GolfPopper 5mo ago
>pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating I think you're looking at the wrong end of the spectrum there. It's some of the biggest players who flaunt the rules. "Several AI companies said to be ignoring robots dot txt exclusion, scraping content without permission: report" (2024) https://www.tomshardware.com/tech-industry/artificial-intelligence/several-ai-companies-said-to-be-ignoring-robots-dot-txt-exclusion-scraping-content-without-permission-report https://www.tomshardware.com/tech-industry/artificial-intell...
- Paracompact 5mo ago
  Fair point. Being small and shadowy is a sufficient condition to avoid litigation, but not a necessary one. Another sufficient condition is having billions of dollars to throw around. Unfortunately, archive.org is well known, well loved, and fundamentally harmless.
  - fragmede 5mo ago
    > fundamentally harmless. This is going to go in a boring direction with an argument thread that's been made since Internet time immemorial, and before. The argument goes: Pirating articles off nyt.com leads to lost sales of subscriptions, so it's not harmless. The response is, inevitably, no it doesn't, it leads to more sales. Or, people who weren't going to pay weren't going to pay anyway, so might as well give it to them for free, and be happy (as the NYT) for free advertising. And then the follow up, "No, it's a lost sale and journalism needs the money." HN is for thoughtful and substantive discussion, not for rehashing the same boring argument we've all read a thousand times. So my question isn't which camp is right. Both camps are firm in their beliefs. Copyright infringement is fine, copyright infringement is not. My question is in today's AI-fueled digital hellscape, how do we support journalists and the arts? If journalism only exists because eg Jeff Bezos pays for the Washington Post, we're going to get biased reporting (which has existed since long before the Internet); If art only exists because the artists come from rich families or have patrons like the Renaissance era, is society better off?

Gigachad 5mo ago

It's because they want to restrict AI companies from stealing content, but they can't do it if internet archive proxies it all for them. All of the LLMs would be massively less useful if it wasn't for scraping the latest news.

stephen_g 5mo ago
LLMs have other ways of accessing the content, they don’t need the Web Archive. Every LLM company can afford to spin up a new subscriber account every day, proxying to appear different IPs from all sorts of ASNs, do some crawling until the account gets banned, and then do it again, and again, and again.
- overfeed 5mo ago
  > LLMs have other ways of accessing the content, they don’t need the Web Archive. What's the conclusion from this train if thought? Just because some burglars can pick locks doesn't mean you should leave your front door unlocked. Locking a door (or robots.txt) is how one can establish mens rea for those who bypass the barrier.
  - AnthonyMouse 5mo ago
    This is like arguing that services can't provide access to libraries that provide public WiFi because it would give the public legal permission to pirate TV shows. They're two unrelated things. And then some members of the public argue that they're making fair use rather than pirating anything, but that still has nothing to do with the library.
  - stephen_g 5mo ago
    But as I understand it, the Web Archive does respect robots.txt, while LLM scrapers absolutely do not and use all sorts of dodgy methods to get around it already... The actual root cause is that we're allowing LLM companies to completely disregard copyright laws for their profit. Whether the LLM companies scrape the Web Archive or the original source doesn't change the copyright infringement implications in any way, and cutting off the web archive doesn't practically change anything (because as I understand, LLM scraping is already prolific all over the web).

joecool1029 5mo ago

No, archive.org does NOT respect robots.txt. You need to reach out to them directly and ask your site not be included: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

input_sh 5mo ago
Aren't you choosing to ignore something very specific specified in that article? Why do you make it seem that article implies it's their overall policy? > A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org).
- joecool1029 5mo ago
  > Aren't you choosing to ignore something very specific specified in that article? Of course not, did you ignore the lines right after? “As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.” The announcement is from 9 years ago. I already mentioned they ignored the robots.txt for my own blog.
LocalH 5mo ago
I'd rather they disregard robots.txt than the opposite situation, where someone does not use robots.txt on a domain to allow IA to archive it, then for whatever reason the domain lapsed and got swooped up by a parker who then subsequently adds a robots.txt blocking IA from the whole site, which would have then caused IA to remove all historical archives of that domain from public view.
- pseudalopex 5mo ago
  Hiding old archives when robots.txt changed was a problem Internet Archive created and could have fixed any time.

userbinator 5mo ago

It's the same idiocy that DRM created. Be a pirate, because a pirate is free...

deleted 5mo ago

[deleted]