5 ms·
I think a case could be made that the Archive should be the reference by default for citations -- that is, when a URL is cited, the Archive's snapshot API is tr
by danso 10y ago
I think a case could be made that the Archive should be the reference by default for citations -- that is, when a URL is cited, the Archive's snapshot API is triggered, and the archive URL is used as the direct link with the original URL being the "backup". Sites/pages change so much that it's more helpful to point the average user to what a page looked like when the reference was accessed and included. It may rob the original URL some traffic, but Wikipedia isn't meant to be a link-aggregator for external content.
The obvious roadblocks are:
- Websites, even content-focused ones, can be so complicated in terms of JS that the Archive might not capture it accurately.
- The Archive's policy is to honor robots.txt and other no-archive directives.
- nothrabannosir 10y agoIt may rob the original URL some traffic, That is an advantage. It reduces extraneous incentives to post links, which should be only about the information they provide and not about page views. At wikipedia's scale, and sensitivity to information purity, that's relevant. See also the rel="nofollow" decision from a few years back.
- eli 10y agoIt'd be a real shame if publishers decide to opt out of the archive to reclaim "lost" pageviews.
- iotscale 10y agoIt's probably good that it's possible to opt-out of being archived but what annoys me is a useful domain changing hands and the new owners deciding to wipe its history.
- thomasahle 10y agoSurely that's not possible?? And there is no way to find the old archive, even under an alias?
- lorenzhs 10y agoIt's not wiped (they don't delete anything) but if robots.txt forbids indexing then access to old archives is blocked. It can be quite aggravating, but at least the information is not lost---if robots.txt or archive.org's policy changes, it can be retrieved again.
- RandomInteger4 10y agoWait, so every time someone makes a citation on wikipedia it causes Archive.org to archive that URL? This seems like it could be abused by a malicious agent, no?
- voxic11 10y agoYou can already just tell archive.org to archive a URL. How would this be different?
- smitherfield 10y agoYou could have a "reverse DDoS" where archive.org blocks requests from Wikipedia, removing the ability to add Wikipedia citations.
- klodolph 10y agoThere's no such thing as a "reverse DDoS", that's just a regular denial of service attack.
- cmdrfred 10y agoA theoretical reverse DDoS: You somehow prevent traffic from reaching their load balancer so it automatically scales their AWS instances to zero.
- toomuchtodo 10y agoYour minimum instances would always be 1 in the autoscaling group associated with your ELBs (or now, ALBs). Regardless, neither the Internet Archive nor Wikimedia use AWS or other cloud providers, as it would be prohibitively expensive. They both run their own infrastructure/ops.
- cmdrfred 10y agoMostly a joke, just trying to think of what that term would mean.
- JoshTriplett 10y ago> - The Archive's policy is to honor robots.txt and other no-archive directives. What bothers me is that they do so retroactively based on the current robots.txt, not the one contemporary with the archived content. So if a domain parker takes over a domain, and their robots.txt excludes everyone from every page (or everyone but Google), then archive.org no longer provides its archive of the old content.
- lucb1e 10y agoThat seems like an awful bug, is this by design?!
- simcop2387 10y agoBy design from the last time I saw it discussed. The idea is that a change there could indicate that there was a mistake and the data shouldn't have been crawled for one reason or another. There's just no way to know in an automated fashion.
- lucb1e 10y agoI'm inclined to say that robots.txt should be ignored altogether and removal requests should be handled per individual case. But I guess that goes too far... Still, if the domain owner changes, they should not be able to remove content from old archives. That's like being able to remove stuff from encyclopedias about a palace somewhere, just because you live in the place where the palace once stood. They are The Internet Archive after all, it's logical to archive contents like the domain owner and robots.txt for a given point in time. A change of owner can be easily detected.
- mhurron 10y ago> removal requests should be handled per individual case Are you going to fund the internet archive to handle that workload? > if the domain owner changes, they should not be able to remove content from old archives. Why not? If that work belongs to anyone, it is the current domain owner. Why does the fact that the internet archive happened to crawl it mean that suddenly they lose control of their information? This is elevating the Internet Archive from 'hey it's cool someone made a copy of that while it was up and no one cared' 'because the Internet Archive crawled it the world has absolute rights to that information from now on, wishes of the owner be damned.'