5 ms·
The feature I most want from the Internet Archive is the ability to donate them an old domain name and enough cash to renew it for the next hundred years such t
by simonw 5y ago
The feature I most want from the Internet Archive is the ability to donate them an old domain name and enough cash to renew it for the next hundred years such that they can keep an archived version of a site available (without breaking any incoming links) for a very long time.
They would also need to be able to handle legal administration costs of things like DMCA take-down notices, but I assume they already have to deal with that for the rest of the archive so hopefully that's not an extra complexity for them.
- EGreg 5y agoWhy not use MaidSAFE or IPFS for that?
- hluska 5y agoThat doesn’t solve the ‘everyone you know is dead’ problem.
- frakkingcylons 5y agoWith regards to IPFS, you'd need to find a pinning service that offers long-term contracts. Moreover, I have more confidence in the Internet Archive in being around in 10, 20, 30 years compared to any existing pinning service. That's not meant to be a slight against said pinning services, just that the IA is well established in their role.
- EGreg 5y agoCheck out Freenet They drop stuff that has the least amount of accessing, so if you want to keep something online you have to pay someone to keep accessing it. Makes sense, but I'd argue that it should be a market, meaning the price should go up as more people try to access stuff, but anyone can start to seed popular content and collect revenues also for hosting it (this is better than wasting electricity on accessing stuff or doing proof of work). But isn't FileCoin exactly that for IPFS? MaidSAFE goes a step further and has nodes rebalance autonomously and earn the most safecoin, as something gets more popular it gets seeded more.
- Ericson2314 5y agoA far cheaper solution is a browser extension that looks up DNS differently based on the age of a the link. It wouldn't be hard to maintain a hand-crafted database of when domains are reused for something completely different, or even when the same conceptual website has breakages, and use that to choose between Internet Archive or live web accordingly. When one is browsing from an internet archive page, the date is known, when someone is browsing from a live website, heuristics can be used, along with "bisecting" dates when the link is dead. Ultimately we want more content addressing to avoid this problem entirely (see below), Or DNS -> PubKey, PubKey -> latest content, with some law that the pubkeys shall not be reused for unrelated things. vs DNS which is mere ephemeral Huffman encoding. So see below for the stuff on IPFS. But the trick above is a good stop-gap, and indeed the database itself used to back the extension could be on IPFS.
- ignitionmonkey 5y agoI made something very similar, though instead of age, it enforces page authorship for links (using PGP signatures). https://webverify.jahed.dev/ https://webverify.jahed.dev/
- zamadatix 5y agoTrying to figure out the age of links referenced against a hand crafted database (or trying to figure out if the current version is "too different" based on age automatically) using a browser extension only serves to create an unreliable solution for a few using the extension. Cheaper sure but it's also doing a whole lot less. Alternative content addressed systems may also work better for finding the content than someone hosting your DNS records for a long time but the bulk of the problem space is in guaranteeing active hosting in a way viewable to viewers of the age will be available for many years not addressing the content. On top of hosting and addressing the Internet Archive offers the ability to view old content on modern browsers even if modern browsers have 0 support for such content anymore (or if browsers ever had support at all even). Forward compatibility isn't something solved by a protocol.
- Ericson2314 5y ago> using a browser extension only serves to create an unreliable solution for a few using the extension. Cheaper sure but it's also doing a whole lot less. This is rather pessimistic thinking. The same money that goes into buying up domains could go into lobbying browsers to add this functionality be default. > bulk of the problem space is in guaranteeing active hosting in a way viewable to viewers of the age will be available for many years not addressing the content. You're moving the goal posts. I am not saying "IPFS means we don't need the internet archive". We absolute do need the internet archive. Content addressing helps by making archiving transparent, so the archival copy is not worse than the original. Fundamentally, consumers producers or archivists may be the party most interested in the continued existence of some information at different moments in the lifespan of that information. Location-based addressing forces the producers to shoulder the burden of hosting, but content-based addressing allows the work to be distributed among those 3 however we see fit. Of course the burdened must still be barred! That doesn't mean the flexibility isn't extremely useful. > On top of hosting and addressing the Internet Archive offers the ability to view old content on modern browsers even if modern browsers have 0 support for such content anymore (or if browsers ever had support at all even). Forward compatibility isn't something solved by a protocol. Yeah that's great too, and again not something I am arguing is not good, or not necessary.
- btrettel 5y agoA full-text search for the Wayback Machine would be my top feature request. It's not uncommon to lose the URL of a site and for active webpages to not have the URL of the old website. Plus I'm sure there are many interesting archived webpages I could find with a full-text search. I understand they've tried this or things like it a few times but they haven't ever kept the feature.
- gregsadetsky 5y agoI imagine that the costs to run this would outweight the potential marketing benefits, but it'd be amazing to see Algolia take this project on to benefit everyone. The Wayback Machine's data is ~20PB..? What is approximately the size of the indexable text (i.e. the text content of html pages, sans tags)? And what would the index size be like, approximately? I imagine that creating (and maintaining, of course) the index would be the most time-consuming part? Is it at all possible to imagine hosting this index... somewhere... and doing sqlite http range-like queries on it..? Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? And have separate docid -> archive.org url mapping?
- dreamcompiler 5y agoI'd think you'd want something with at least the features of Lucene, whose index is 20%-30% the size of the data. https://lucene.apache.org/core/features.html https://lucene.apache.org/core/features.html
- wolverine876 5y ago> Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? The problem, as with any search engine, is the ranking algorithm. Without a sufficent one, the search results are useless. What use is a list of every page in the Wayback Machine containing the word "apple"? The Wayback Machine possibly would need a much larger index than any normal search engine: not only the present websites, but all the historical versions (though I don't know what proportion of the web they've indexed).
- lepouet 5y agoI agree, i'm a passionate photographer and i could pay good money to know that my pictures could be seen long time after my death. Maybe startups exists that do this, but they will die, i need something with enough critical mass that i can trust.
- osigurdson 5y agoPerhaps have a look at Arweave (https://www.arweave.org https://www.arweave.org).
- boarnoah 5y agoThat doesn't address the concern re: something needing critical mass to increase its chance of survival over a longer term. Really the way I see it outside a few large banking firms, its kind of hard to be sure any provider of digital services would be around in the 50+ year term for this kind of public archive. I hope the Internet Archive manages it. EDIT: I do worry the IA has a bit of a lightning rod effect with skirting issues re: legality of archiving content. IMO its no guarantee it survives any significant time span either.
- wolverine876 5y ago> Really the way I see it outside a few large banking firms, its kind of hard to be sure any provider of digital services would be around in the 50+ year term for this kind of public archive. A library could do it. Perhaps leading institutions like the British Library or Library of Congress. I've thought that IA should be a Library of Congress project, and may eventually end up under their auspices.
- toomuchtodo 5y agoIt costs the Internet Archive $2/GB to host content in perpetuity. They have a tool, Archive It, that will periodically crawl your site for archival purposes if you are not technical. For my needs, I run a report monthly for the content I’ve archived using my IA account to determine archived GBs, and then donate the amount needed to cover those costs. Consider reaching out to their patron services email address with any questions. Edit: $2/GB citation: https://help.archive.org/hc/en-us/articles/360014755952-Archive-org-Information https://help.archive.org/hc/en-us/articles/360014755952-Arch...
- ineedasername 5y agoIs that $/GB per year?
- toomuchtodo 5y agoLast I read, that was one time [1]. Of course if you have the means, consider a bit more per GB. [1] https://help.archive.org/hc/en-us/articles/360014755952-Archive-org-Information https://help.archive.org/hc/en-us/articles/360014755952-Arch...
- tablespoon 5y ago> It costs the Internet Archive $2/GB to host content in perpetuity. Do you have source/more info than that? Lets say the internet archive is 100 PB [1], that's 100,000,000 GB [2], and at that rate it comes out to $200 million [3] for the whole thing forever. That's a lot of money, but also a lot less than I was expecting for something like that. [1] https://www.protocol.com/internet-archive-preserving-future https://www.protocol.com/internet-archive-preserving-future: "The web archive alone is about 45 petabytes — 4,500 terabytes — and the Internet Archive itself is about double that size (the group has other collections, like a huge database of educational films, music and even long-gone software programs)." [2] https://www.google.com/search?q=100+petabytes+to+gb https://www.google.com/search?q=100+petabytes+to+gb: "100 petabyte = 1e+8 gigabytes" [3] https://www.google.com/search?q=1e%2B8+*+%242 https://www.google.com/search?q=1e%2B8+*+%242: "1e+8 * (US$ 2) = 200 million US$"
- adventured 5y ago> The feature I most want from the Internet Archive The feature I most want from IA is a streamlined system to delete content they have archived on domains that I own, including a proper privacy law compliance effort on their part. They have intentionally made it a difficult, manual process to get content removed. They operate as a de facto malicious crawler. They massively violate GDPR with how they operate and few seem to care about that fact, including all the commenters on HN (which universally give them a free pass on being malicious and violating GDPR very aggressively). When IA has to comply with laws like GDPR, that's the end of IA.
- gojomo 5y agoShouldn't it be hard to delete things from a library and historical-archive? If you had to choose between the GDPR, & an accurate historical record, which would you prefer?
- Mezzie 5y agoWe won't. Pretty much the only way things leave archives of record (which is what IA is trying to be) are through Acts of God (if the archive burns down/all of IA's servers are taken out in a mass alien EMP attack). An author suggesting that the LoC remove their copy of a book/other work (including digital works) because they want to unpublish it would not fly. The parent comment has an issue with anything on the Web not hidden in some way being considered 'public' and 'published', but that would be something that would require international cooperation to hash out.
- pjc50 5y ago> When IA has to comply with laws like GDPR, that's the end of IA. Will you be happy when you've burned down that library?
- Mezzie 5y agoI can't speak to GDPR specifically because I'm not European, but a fair amount of laws have leeway for preservation purposes. (For example, section 108 of the Copyright Act in the US functionally exempts archives from being punished for copying provided they are doing so for preservation purposes). There are very good reasons that archives will not destroy or alter information outside of very clear difficult and manual processes. And actually, looking at it, I don't think they're necessarily in violation of GDPR [0]. Point 3 says: "Where personal data are processed for archiving purposes in the public interest, Union or Member State law may provide for derogations from the rights referred to in Articles 15, 16, 18, 19, 20 and 21 subject to the conditions and safeguards referred to in paragraph 1 of this Article in so far as such rights are likely to render impossible or seriously impair the achievement of the specific purposes, and such derogations are necessary for the fulfilment of those purposes." According to GDPR, national law of EU parties overrules GDPR when it comes to personal data being used in archival context. I don't know every EU country's stance, but most of the bigger economies would allow for this. There is also a difference between deleting the data and rendering it inaccessible to the public. Keeping something under wraps is generally more 'acceptable', but active destruction of the item (digital or not) and its providence is much more limited. Also there's a difference between personally identifying data (covered by GDPR), your content (which would be covered under copyright and not GDPR), and connections people can make if that content is available (not covered at all because it's not anybody else's issue if you write something terrible and people keep recognizing you over it so long as you did actually write it). [0] https://gdpr-info.eu/art-89-gdpr/ https://gdpr-info.eu/art-89-gdpr/