31 ms·

Internet Archive as a default host-of-record for startups

356 points by bpierre 5y ago

riobard 5y ago
Correct me if I'm wrong, but isn't this problem the ideal use case for projects like IPFS? Anyone interested to preserve the content can join as a node to balance the load, right? And if so, why don't we see widespread adoption?
- jayd16 5y ago
  IPFS does the opposite, right? It doesn't guarantee the archive is available, which is what Carmack is asking for. Incentives to scale bandwidth with need already exist as long as you have he data at all. That is to say, IPFS doesn't help if the desire blooms after the nodes dry up. Things could still be lost.
  - Ericson2314 5y ago
    The point of IPFS is not keep the data archived, but to allows users to no care who does the archiving. Concretely, this would be to skip the "many people on encountering a dead URL don't bother to try the internet archive" problem.
- ricardobayes 5y ago
  There are many a projects who go open source when they fail. The extra step here is IA would become the A record for the project (at least temporarily?)
- Ericson2314 5y ago
  I was involved with planning https://nlnet.nl/project/SoftwareHeritage-P2P/ https://nlnet.nl/project/SoftwareHeritage-P2P/ for just this reason --- hopefully we will finally be able to start work on it sometime too far off. Indeed the real challenge of archival is not loosing the stuff, by making sure that people can still find the stuff. "Orphaned" information that no one knows exists, or is bothering to interact with, isn't that valuable compared to resources that are actively being used and still "live" in the culture. Of course, the archive can never serve the same amount of bandwidth, but the goal is a) interested parties can mirror the stuff they care about in a higher bandwidth / item way after some huge disruption c) random viewers never notice something going down, nor who is serving the info, but just a temporary drop in connection quality. Ultimately, location-based addressing is a stupid way to run society, needlessly fragile by baking in very property claims (IPs, DNS, etc.) that are incidental to the task at hand. Content-based addressing, with location based hints to avoid trying to solve really hard problems all at once, is the only way to make culture more robust.
  - btown 5y ago
    The great thing about location-based addressing is that an archive of the set of known locations is not subject to the same ownership rules as the canonical live version of those addresses. A document listing all Geocities URLs can be placed in content-addressed storage without needing geocities.com to be owned by the party that emplaces that document. And a chain can be maintained such that people are incentivized to remember that document into the far future. Coupled with archival of the actual content, you bypass the exclusivity of domain ownership. Of course, ensuring that there's persistence of attention as well is a tougher problem. But one only needs to look at sites like https://reddit.com/r/tumblr https://reddit.com/r/tumblr to realize that there is immense societal interest in "meme archaeology." Reducing the barriers to entry to would-be archaeologists, giving them a "chain" of breadcrumbs that lead to content, and building communities that will socially reward people for their archaeology work, is the best thing we can possibly do.
    - Ericson2314 5y ago
      > The great thing about location-based addressing is that an archive of the set of known locations is not subject to the same ownership rules as the canonical live version of those addresses. Erm, to me this sounds like putting up with link rot as hack around bad IP law? There are already IP exceptions for preservation. And if content-addressing was the norm, geocities-type sites might bow to market pressure to not "own" the content, but merely have some some sort of license for being the exclusive pinning service and running the ads or whatever. This is like avoiding the problem where your the rent on your current apartment doesn't fall as much as the market writ large because your landlord knows moving is not free.
- deleted 5y ago
  [deleted]
- dannyobrien 5y ago
  It is -- and Brewster Kahle and the Archive have been thinking about this for a long while (see this talk from him five years ago: https://archive.org/details/LockingTheWebOpen_2016 https://archive.org/details/LockingTheWebOpen_2016 ). The model you can think for this would be to have as the Archive as the "node of last resort" of content-addressable storage, making sure there's always one node up with the content you want. The incentive challenges are making sure that the average number of nodes is more than one, because, as Brewster likes to say, "libraries burn; it's what they do", plus all the traditional challenges of maintaining a commons at high levels of resilience. Once you have data on a network like IPFS, we can use a number of incentive models to make sure it stays there, including charitable projects like the Archive, government support (archives are traditionally state projects -- if every country's archive was pinning this content, it would be far more resilient), and decentralized incentive frameworks like Filecoin. (Disclosure: I work for the Filecoin Foundation; in our decentralized preservation work, we've funded the Internet Archive's work in this area, though I should emphasise that IA works with a lot of different decentralizing technologies through their https://getdweb.net/ https://getdweb.net/ community.)
- thesausageking 5y ago
  IPFS only does addressability, it doesn't provide storage. You could use a decentralized storage network like Arweave, Filecoin, or Sia. https://www.arweave.org/ https://www.arweave.org/ https://www.filecoin.com/ https://www.filecoin.com/ http://sia.tech/ http://sia.tech/
  - cle 5y ago
    Or "centralized" ones like Fleek, Textile, Pinata, etc. https://fleek.co/hosting/ https://fleek.co/hosting/ https://docs.textile.io/buckets/ https://docs.textile.io/buckets/ https://www.pinata.cloud/ https://www.pinata.cloud/
kodah 5y ago
Personally, I think eternally archiving everything and infinitely available public data has been not-so-great. If this was an "archive with consent" sort of system, then sure. My response may be better summarized as, "Does IA support robots.txt, and if not why?"
- 1MachineElf 5y ago
  There is some public discussion about why IA does not strictly adhere to robots.txt: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
  - kodah 5y ago
    They're basically saying they're choosing to ignore a web convention that explicitly states that people don't want their websites archived or searchable because they want them to be. Sounds pretty unethical to me.
    - BenjiWiebe 5y ago
      When those people die and quit paying for hosting, the information on their website doesn't magically become useless. Perhaps other people still have a need for it. Thank goodness the IA doesn't blindly obey robots.txt.
- rectang 5y ago
  Throughout human history, records have been forgotten, rewritten, changed, mutated, degraded, eroded away to nothingness. "The internet is forever" has always struck me as inhumane. Make a mistake or expose a weakness on the internet and it will always accompany you. It turns out that the internet is not always forever. I find that comforting.
  - symlinkk 5y ago
    Maybe humans should become more accommodating of past mistakes.
dleslie 5y ago
There's nothing about his suggestion that requires use of a Blockchain. IA could certify authenticity easily without it.
simonw 5y ago
The feature I most want from the Internet Archive is the ability to donate them an old domain name and enough cash to renew it for the next hundred years such that they can keep an archived version of a site available (without breaking any incoming links) for a very long time. They would also need to be able to handle legal administration costs of things like DMCA take-down notices, but I assume they already have to deal with that for the rest of the archive so hopefully that's not an extra complexity for them.
- EGreg 5y ago
  Why not use MaidSAFE or IPFS for that?
  - hluska 5y ago
    That doesn’t solve the ‘everyone you know is dead’ problem.
  - frakkingcylons 5y ago
    With regards to IPFS, you'd need to find a pinning service that offers long-term contracts. Moreover, I have more confidence in the Internet Archive in being around in 10, 20, 30 years compared to any existing pinning service. That's not meant to be a slight against said pinning services, just that the IA is well established in their role.
    - EGreg 5y ago
      Check out Freenet They drop stuff that has the least amount of accessing, so if you want to keep something online you have to pay someone to keep accessing it. Makes sense, but I'd argue that it should be a market, meaning the price should go up as more people try to access stuff, but anyone can start to seed popular content and collect revenues also for hosting it (this is better than wasting electricity on accessing stuff or doing proof of work). But isn't FileCoin exactly that for IPFS? MaidSAFE goes a step further and has nodes rebalance autonomously and earn the most safecoin, as something gets more popular it gets seeded more.
- 5y ago
a-dub 5y ago
it sounds like he wants to be able to archive and respinup/run things like mmoprg game servers as easily as static content can be archived and served today. that would be a huge and expensive paradigm shift for internet service backends that traditionally have been only designed to be run by one entity and typically are a mix of custom, open source and proprietary software that is run in a specific way. i suppose it could be done, but there hasn't been any reason to make that investment. service backends also tend to be more "living software" where part of the system is the team that continuously builds, updates and operates it. basically, it would look something like java applets, but for entire internet service backends. one step and all the services, databases, everything would spin up and start serving. that would be great but is probably a ways out.
- jayd16 5y ago
  Well there's other static things like FAQs, news feeds, even hosted game content that a game might rely on. It's not just dynamic services. I think Carmack is referring to that.
  - a-dub 5y ago
    ..."something like this could combine with a blockchain style technology to make internet applications that could outlive companies. A niche multiuser game that couldn't meet company revenue goals could still be "fed" by anyone that wanted to push resources at it, since the " that's not static assets, that's a multiuser game server. i think he's envisioning entire internet service backends that can be packaged up like java applets and re-run on demand paired with some kind of decentralized serving infrastructure that any user can insert coins and resurrect a sophisticated web service from the past. more likely i suspect we'll see more efforts by hobbyists to resurrect these things and more releases of backends from failed projects into the public domain. with so much physical gear that requires service backends being made today, we may even see regulation that requires release of the source for a service when a service is shut down. crazy to think that if the company who made your car or tractor fails, that your perfectly good car or tractor could cease function when they shut down the service backend.
alberth 5y ago
>>” I wonder if there could be a world where the IA acts as a default host-of-record for startups, with a super-easy CDN relationship such that the content” Doesn’t IA already partner with Cloudflare to do exactly what Carmack is suggesting. https://blog.cloudflare.com/cloudflares-always-online-and-the-internet-archive-team-up-to-fight-origin-errors/ https://blog.cloudflare.com/cloudflares-always-online-and-th...
- magila 5y ago
  You still need to provide a separate origin server (i.e. host-of-record) when using Always Online. AO is designed to use IA as a backstop when your origin server goes down.
kderbyma 5y ago
I love this idea. I am working on a mini project for a decentralized game which can be self hosted or snapshots - so you can share / modify a portion of it and provide a custom experience
ordinaryradical 5y ago
Imagine someone building this for SaaS hosting--a perma-Heroku, or something like it. That's actually a huge value-add. Suddenly tinyStartupA doesn't need to convince largeCorpB that it's going to be around for forever. The service can exist in perpetuity without the company. Complex repercussions obviously around acquisition, IP, and other business dimensions however. Maybe unworkable even. But I think there's a world where this actually exists and lowers the barrier to building business-critical software and selling to companies that need a 50-year commitment to risk you.
- jayd16 5y ago
  I wonder if its worth it for these platforms (like Heroku) to simply add a donation portal. It's not as future proof as something fully open, but it wouldn't need the design, implementation and maintenance of a brand new fully open PAAS ecosystem.
mwattsun 5y ago
I think it's interesting to think about what we have lost because we couldn't keep everything from a 100 years ago and what society 100 years from now will be grateful we preserved. Off the top of my head, we lost a lot of common wisdom in dealing with the flu pandemic of 1918 because personal letters and most newspapers were not preserved. I think 100 years from now they might wish we had preserved more from marginal and/or world communities. What folk wisdom is being lost? Perhaps we need to expand our definition of what is worth saving.
- lelandfe 5y ago
  An excellent place to start is talking to your parents and grandparents and recording their history and stories online.
  - mwattsun 5y ago
    That's a good suggestion I've been following
- bananamerica 5y ago
  Hard to know what will be of interest for future historians. Some things in which we place great value can be considered irrelevant, while some of our junk can become historical gold.
  - mwattsun 5y ago
    > some of our junk can become historical gold That's what interests me. For example, there's a cool repository of 12 step speaker meeting talks hosted in Iceland [1] and frankly, some of the talks are junk, but there's a lot of wisdom. What I find interesting is how it showcases how ordinary citizens talk to each other. The words they use, the accents, the little gems of folk wisdom contained, along with some uncommon stories. This will be valuable 100 years from now if, for example, you want to build a virtual world based in the mid to late 20th century and you want to get the accents correct. What phrases did people use? What were some common misconceptions? Maybe 100 years from now addiction will no longer be a problem. If my virtual world is to be accurate I need to know what it was like for ordinary people when it was a problem. Etc... [1] https://xa-speakers.org/ https://xa-speakers.org/
tonymet 5y ago
Is it really worth archiving ?
- maxbond 5y ago
  It's difficult to know beforehand! A startup might be a total flop that delivers nothing of value to us now, but it might be a valuable datapoint for future historians to understand how startup culture changed and evolved. Or it may be interesting to future founders - I've seen some startups with perfectly fine ideas fail, and then a few years later, someone succeeds doing something very similar. This may not be the best example, but while I'm sure the Rosetta Stone, being a treaty, likely would have seemed like something worth preserving, would anyone have imagined that it would be the pivotal document in understanding ancient Egyptian? That it would be one of the most important documents of all time?
  - tonymet 5y ago
    that's optimistic, and I could see the potential. But I've become a bit suspicious of the content on the internet and the costs in archiving it. Maybe Internet Archive could have a curation threshold . I would rather revert to the internet where quality content was published openly on the web. But presently web content, at least that upranked by Google, is very low quality (seo, clickbait, biased, trivial)
soheil 5y ago
Interesting use cases for IA is to go back to early days of when a startup first launched to see their pitch before they raised real money and added tons of nonsensical bs to their homepage. eg. http://web.archive.org/web/20180901110658/https://www.snowflake.com/ http://web.archive.org/web/20180901110658/https://www.snowfl... http://web.archive.org/web/20140701061721/https://databricks.com/ http://web.archive.org/web/20140701061721/https://databricks...
ineedasername 5y ago
I don't understand why Carmack thinks blockchain should be a component of this. Anyone care to elaborate on how that would make this easier/better?
- Jonovono 5y ago
  .... He says it right after "to make internet applications that could outlive companies". If it's on the blockchain it doesn't matter if the company storing all of the archives shuts down, the content would still exist, forever, until their is a network running the chain. I suppose something like Torrent could be used?
  - Mezzie 5y ago
    He has a point, but it's that private companies shouldn't archives of record. I actually think using blockchain for things like ensuring providence is interesting, since in archives being able to have a clean record of what happened to a piece is VERY useful. It just won't earn a ton of money, so we'll need to wait for the capitalism to burn off to see more not-for-profit uses.
- ricardobayes 5y ago
  Similarly how github is a blockchain. I think he means the ease of version control by this.
  - fs111 5y ago
    Github is a software development tooling provider, not a blockchain
    - rat9988 5y ago
      For his defense, he probably meant Git and typed too quickly. It's still obvious what he meant.
      - 5y ago
nickdothutton 5y ago
I wrote a little about this and the alternatives here, especially for sites with user-generated content which may be impossible to find/regenerate from elsewhere. I call it “Beating the Samson Option”, of pulling the temple down upon your head. https://blog.eutopian.io/beating-the-samson-option/ https://blog.eutopian.io/beating-the-samson-option/
zaps 5y ago
I was with him up until “blockchain”.
midoBB 5y ago
908087 5y ago
polote 5y ago
The idea is more interesting when you think about scrapers. Take any ecommerce website, there are several scrapers that download all pages every hours, it would be more efficient if a provider had a live copy of the website and then serve the requests to the scrapers or could even send webhook. A website could handle tons of scrappers without having high bandwidth, only the provider will need high bandwidth. The issue is that scrapers often play with cookies and dynamic websites and such solution wouldn't work on these cases
- akouri 5y ago
  scraper*
- azalemeth 5y ago
  (in the nicest way and since your post is recent, please edit s/scrappers/scrapers! It very much changed how I read it first time around -- I thought you were referring to a type of failed startup!)
  - polote 5y ago
    Thanks, I need to improve my english haha
- atdrummond 5y ago
  Is scraper one “p” or two? My inclination would be scrapper is “scrap”-er rather than “scrape”-er.
- pjc50 5y ago
  Scrapers. A scrAper is the thing you use to remove ice from windscreens or scrape data from websites. A scraPPer is someone who likes getting into fights or collecting scrap metal.
pjc50 5y ago
I'm old enough to remember when the host-of-record for failed startups was fuckedcompany.com ... I do wonder how many startups actually want to be archived, rather than just ditch everything with unseemly speed as soon as they get acquishutdown.
- monkeydust 5y ago
  There's huge valuable learnings to be had in failure. Through some coordination they could perhaps get compensation for sharing although this goes against it all being open and free.
nur23kg 5y ago
AlexanderTheGr8 5y ago
What compression does IA use to store websites? Using a 2x better compression will allow them to store 2x more websites/content. I am doing some compression research, and would love to help IA in any way I can. There are some amazing SOTA compression algorithms available now. And if IA ignores images/video, and focuses only on text, they can store an insane amount of websites at a very low cost.
- jrwr 5y ago
  right now, most of it is coming in as ZSTD with a central index. older stuff was gzip
  - AlexanderTheGr8 5y ago
    I am working on a ML-based compression which will be slow to compress/decompress but will give much better compression than zstd/gzip (even 2-3x better). Do you think that it's a useful algorithm for archival reasons or for storing huge amounts of data in cloud?
1vuio0pswjnm7 5y ago
Why is there only one IA? Why is IA not globally distributed, like a CDN? I use IA for "problem" websites, e.g., ones that rely on SNI, i.e., ones hosted at certain CDNs. I simply add add these sites to a list and the local proxy does the rest. http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f list } IA "hosts" an enormous number of sites without the need for SNI (plaintext hostnames sent over the wire). EDIT: @sebow the way they (re)format the HTML is less friendly to the text-only browser I use.
- sebow 5y ago
  archive.today/.ph/etc is vastly superior for doing the intended purpose here: archive[given that you have the URL obviously] IA is more of an curated internet archive + explorer(which granted is very good).
- amenghra 5y ago
  What’s your issue with SNI/threat model? If you use a non-SNI site, anyone can tell which site you are visiting since there’s only one domain on that IP.
  - yellowapple 5y ago
    Seconding that curiosity. Pretty much every web server I've built uses SNI (at least if it's hosting sites under multiple domains), and the only "downside" of which I'm aware is the lack of IE6 support.
  - 1vuio0pswjnm7 5y ago
    There is a difference between making something "impossible" and making something "easier". Performing reverse DNS lookups, or otherwise trying to maintain a global table of 1:1 domain:IP mappings and perform lookups in real-time, is nowhere near as easy nor reliable as sniffing SNI. IME, it is neither easy nor reliable, nor worth the effort. SNI is the preferred method. SNI is easier. SNI is 100% reliable for detecting what hostname the user is trying to access. What is the point of so-called "DNS privacy/Private DNS" if "anyone can tell which site you are visiting" simply by observing IP addresses, without any need to see domainnames. If SNI (plaintext hostnames sent over the wire) is a non-issue, then why are people working on encrypted Client Hello in TLS1.3.
msla 5y ago
I'd love it if the Wayback Machiine were less touchy/more reliable. The failure mode I see very often is that the frontend apparently doesn't know what the backend's doing: The part which ingests URLs and tells you what URLs have been archived does not know what archives the backend has, so it will tell you a page has been archived and give you a link to the archive, but when you click the link, it tells you it does not have the page archived, oh, look, it exists online, would you like to archive it now? Archive it again, and it will tell you that you can only archive a page once every 45 minutes. If you're a weird little obsessive like myself, you go through this process a half-dozen times for one page before it acknowledges that, yes, it does have the page archived (once, mind you) and you can actually see it. While I'm filing bitch reports... The Wayback Machine apparently loves setting cookies. It will set cookies until it has exceeded its own ability to accept cookies, at which point it will give you a blank page and you have to look in the developer console to figure out that it sent you a "too many cookies" error in the response header. I've had to force my browsers to not accept any cookies from the Internet Archive to fix this.
markjgraham 5y ago
Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is always open to suggestions about how we might do our jobs better, and improve our service. We also host regular meetups (and have hosted summits and a camp) related to the Decentralized Web. See: https://blog.archive.org/tag/dweb/ https://blog.archive.org/tag/dweb/ The Internet Archive also offers archive-it.org, a subscription service, for those who want a higher level of support and more features. We appreciate any support you can offer, financial and otherwise. Please share any bug reports, feature suggestions and other feedback with us via email to info@archive.org Oh/and… checkout the new PDF Search feature we just launched at the bottom of web.archive.org. More to come like that in 2022. Finally, you might also find some of the things I wrote here of interest: https://gijn.org/2021/05/05/tips-for-using-the-internet-archives-wayback-machine-in-your-next-investigation/ https://gijn.org/2021/05/05/tips-for-using-the-internet-arch...
- sayhar 5y ago
  Thank you for your service!
- chaps 5y ago
  I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals. Cheers.
  - gjs278 5y ago
thrdbndndn 5y ago
I genuinely didn't understand the point he's trying to make. Could someone ELI5?
AtNightWeCode 5y ago
How about a default way of building iterative versioned static web sites where dynamic content is separated and used only for dynamic data. Meaning, most sites can easily be stored offline.
iJohnDoe 5y ago
IA needs to respects robots.txt and they need to make it easier to request data to be removed. Not everyone fully supports the IA mission and they need to respect that view as much as they respect their supporters.
- smarx007 5y ago
  They clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ctrl+S and uploading it to IA. Removing data from the web in 2021? Hmm… https://web.archive.org/web/20211222032633/https://news.ycombinator.com/item?id=29645135 https://web.archive.org/web/20211222032633/https://news.ycom... oops!
  - iJohnDoe 5y ago
    Are you indicating that everything in the IA was added via a person clicking on “archive this page”? I don’t think that is correct. A lot of it was added via automated methods?
    - smarx007 5y ago
      Not necessarily via a button but all of the pages were submitted to them by 3rd parties. So, they don't seem to crawl to discover new URIs eagerly, see https://news.ycombinator.com/item?id=29643506 https://news.ycombinator.com/item?id=29643506 For example, I host my own ArchiveBox at home (you get fulltext search as a bonus) and it is configured to submit every URL I save to IA: https://imgur.com/a/Yhnxo1W https://imgur.com/a/Yhnxo1W IA considers that to be manual submission not subject to robots.txt rules. https://archivebox.io https://archivebox.io

356 points by bpierre 5y ago

Correct me if I'm wrong, but isn't this problem the ideal use case for projects like IPFS? Anyone interested to preserve the content can join as a node to balance the load, right? And if so, why don't we see widespread adoption?

jayd16 5y ago
IPFS does the opposite, right? It doesn't guarantee the archive is available, which is what Carmack is asking for. Incentives to scale bandwidth with need already exist as long as you have he data at all. That is to say, IPFS doesn't help if the desire blooms after the nodes dry up. Things could still be lost.
- Ericson2314 5y ago
  The point of IPFS is not keep the data archived, but to allows users to no care who does the archiving. Concretely, this would be to skip the "many people on encountering a dead URL don't bother to try the internet archive" problem.
ricardobayes 5y ago
There are many a projects who go open source when they fail. The extra step here is IA would become the A record for the project (at least temporarily?)
Ericson2314 5y ago
I was involved with planning https://nlnet.nl/project/SoftwareHeritage-P2P/ https://nlnet.nl/project/SoftwareHeritage-P2P/ for just this reason --- hopefully we will finally be able to start work on it sometime too far off. Indeed the real challenge of archival is not loosing the stuff, by making sure that people can still find the stuff. "Orphaned" information that no one knows exists, or is bothering to interact with, isn't that valuable compared to resources that are actively being used and still "live" in the culture. Of course, the archive can never serve the same amount of bandwidth, but the goal is a) interested parties can mirror the stuff they care about in a higher bandwidth / item way after some huge disruption c) random viewers never notice something going down, nor who is serving the info, but just a temporary drop in connection quality. Ultimately, location-based addressing is a stupid way to run society, needlessly fragile by baking in very property claims (IPs, DNS, etc.) that are incidental to the task at hand. Content-based addressing, with location based hints to avoid trying to solve really hard problems all at once, is the only way to make culture more robust.
- btown 5y ago
  The great thing about location-based addressing is that an archive of the set of known locations is not subject to the same ownership rules as the canonical live version of those addresses. A document listing all Geocities URLs can be placed in content-addressed storage without needing geocities.com to be owned by the party that emplaces that document. And a chain can be maintained such that people are incentivized to remember that document into the far future. Coupled with archival of the actual content, you bypass the exclusivity of domain ownership. Of course, ensuring that there's persistence of attention as well is a tougher problem. But one only needs to look at sites like https://reddit.com/r/tumblr https://reddit.com/r/tumblr to realize that there is immense societal interest in "meme archaeology." Reducing the barriers to entry to would-be archaeologists, giving them a "chain" of breadcrumbs that lead to content, and building communities that will socially reward people for their archaeology work, is the best thing we can possibly do.
  - Ericson2314 5y ago
    > The great thing about location-based addressing is that an archive of the set of known locations is not subject to the same ownership rules as the canonical live version of those addresses. Erm, to me this sounds like putting up with link rot as hack around bad IP law? There are already IP exceptions for preservation. And if content-addressing was the norm, geocities-type sites might bow to market pressure to not "own" the content, but merely have some some sort of license for being the exclusive pinning service and running the ads or whatever. This is like avoiding the problem where your the rent on your current apartment doesn't fall as much as the market writ large because your landlord knows moving is not free.
deleted 5y ago
[deleted]
dannyobrien 5y ago
It is -- and Brewster Kahle and the Archive have been thinking about this for a long while (see this talk from him five years ago: https://archive.org/details/LockingTheWebOpen_2016 https://archive.org/details/LockingTheWebOpen_2016 ). The model you can think for this would be to have as the Archive as the "node of last resort" of content-addressable storage, making sure there's always one node up with the content you want. The incentive challenges are making sure that the average number of nodes is more than one, because, as Brewster likes to say, "libraries burn; it's what they do", plus all the traditional challenges of maintaining a commons at high levels of resilience. Once you have data on a network like IPFS, we can use a number of incentive models to make sure it stays there, including charitable projects like the Archive, government support (archives are traditionally state projects -- if every country's archive was pinning this content, it would be far more resilient), and decentralized incentive frameworks like Filecoin. (Disclosure: I work for the Filecoin Foundation; in our decentralized preservation work, we've funded the Internet Archive's work in this area, though I should emphasise that IA works with a lot of different decentralizing technologies through their https://getdweb.net/ https://getdweb.net/ community.)
thesausageking 5y ago
IPFS only does addressability, it doesn't provide storage. You could use a decentralized storage network like Arweave, Filecoin, or Sia. https://www.arweave.org/ https://www.arweave.org/ https://www.filecoin.com/ https://www.filecoin.com/ http://sia.tech/ http://sia.tech/
- cle 5y ago
  Or "centralized" ones like Fleek, Textile, Pinata, etc. https://fleek.co/hosting/ https://fleek.co/hosting/ https://docs.textile.io/buckets/ https://docs.textile.io/buckets/ https://www.pinata.cloud/ https://www.pinata.cloud/

kodah 5y ago

Personally, I think eternally archiving everything and infinitely available public data has been not-so-great. If this was an "archive with consent" sort of system, then sure. My response may be better summarized as, "Does IA support robots.txt, and if not why?"

1MachineElf 5y ago
There is some public discussion about why IA does not strictly adhere to robots.txt: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- kodah 5y ago
  They're basically saying they're choosing to ignore a web convention that explicitly states that people don't want their websites archived or searchable because they want them to be. Sounds pretty unethical to me.
  - BenjiWiebe 5y ago
    When those people die and quit paying for hosting, the information on their website doesn't magically become useless. Perhaps other people still have a need for it. Thank goodness the IA doesn't blindly obey robots.txt.
rectang 5y ago
Throughout human history, records have been forgotten, rewritten, changed, mutated, degraded, eroded away to nothingness. "The internet is forever" has always struck me as inhumane. Make a mistake or expose a weakness on the internet and it will always accompany you. It turns out that the internet is not always forever. I find that comforting.
- symlinkk 5y ago
  Maybe humans should become more accommodating of past mistakes.

dleslie 5y ago

There's nothing about his suggestion that requires use of a Blockchain. IA could certify authenticity easily without it.

simonw 5y ago

The feature I most want from the Internet Archive is the ability to donate them an old domain name and enough cash to renew it for the next hundred years such that they can keep an archived version of a site available (without breaking any incoming links) for a very long time. They would also need to be able to handle legal administration costs of things like DMCA take-down notices, but I assume they already have to deal with that for the rest of the archive so hopefully that's not an extra complexity for them.

EGreg 5y ago
Why not use MaidSAFE or IPFS for that?
- hluska 5y ago
  That doesn’t solve the ‘everyone you know is dead’ problem.
- frakkingcylons 5y ago
  With regards to IPFS, you'd need to find a pinning service that offers long-term contracts. Moreover, I have more confidence in the Internet Archive in being around in 10, 20, 30 years compared to any existing pinning service. That's not meant to be a slight against said pinning services, just that the IA is well established in their role.
  - EGreg 5y ago
    Check out Freenet They drop stuff that has the least amount of accessing, so if you want to keep something online you have to pay someone to keep accessing it. Makes sense, but I'd argue that it should be a market, meaning the price should go up as more people try to access stuff, but anyone can start to seed popular content and collect revenues also for hosting it (this is better than wasting electricity on accessing stuff or doing proof of work). But isn't FileCoin exactly that for IPFS? MaidSAFE goes a step further and has nodes rebalance autonomously and earn the most safecoin, as something gets more popular it gets seeded more.
5y ago

a-dub 5y ago

it sounds like he wants to be able to archive and respinup/run things like mmoprg game servers as easily as static content can be archived and served today. that would be a huge and expensive paradigm shift for internet service backends that traditionally have been only designed to be run by one entity and typically are a mix of custom, open source and proprietary software that is run in a specific way. i suppose it could be done, but there hasn't been any reason to make that investment. service backends also tend to be more "living software" where part of the system is the team that continuously builds, updates and operates it. basically, it would look something like java applets, but for entire internet service backends. one step and all the services, databases, everything would spin up and start serving. that would be great but is probably a ways out.

jayd16 5y ago
Well there's other static things like FAQs, news feeds, even hosted game content that a game might rely on. It's not just dynamic services. I think Carmack is referring to that.
- a-dub 5y ago
  ..."something like this could combine with a blockchain style technology to make internet applications that could outlive companies. A niche multiuser game that couldn't meet company revenue goals could still be "fed" by anyone that wanted to push resources at it, since the " that's not static assets, that's a multiuser game server. i think he's envisioning entire internet service backends that can be packaged up like java applets and re-run on demand paired with some kind of decentralized serving infrastructure that any user can insert coins and resurrect a sophisticated web service from the past. more likely i suspect we'll see more efforts by hobbyists to resurrect these things and more releases of backends from failed projects into the public domain. with so much physical gear that requires service backends being made today, we may even see regulation that requires release of the source for a service when a service is shut down. crazy to think that if the company who made your car or tractor fails, that your perfectly good car or tractor could cease function when they shut down the service backend.

alberth 5y ago

>>” I wonder if there could be a world where the IA acts as a default host-of-record for startups, with a super-easy CDN relationship such that the content” Doesn’t IA already partner with Cloudflare to do exactly what Carmack is suggesting. https://blog.cloudflare.com/cloudflares-always-online-and-the-internet-archive-team-up-to-fight-origin-errors/ https://blog.cloudflare.com/cloudflares-always-online-and-th...

magila 5y ago
You still need to provide a separate origin server (i.e. host-of-record) when using Always Online. AO is designed to use IA as a backstop when your origin server goes down.

kderbyma 5y ago

I love this idea. I am working on a mini project for a decentralized game which can be self hosted or snapshots - so you can share / modify a portion of it and provide a custom experience

ordinaryradical 5y ago

Imagine someone building this for SaaS hosting--a perma-Heroku, or something like it. That's actually a huge value-add. Suddenly tinyStartupA doesn't need to convince largeCorpB that it's going to be around for forever. The service can exist in perpetuity without the company. Complex repercussions obviously around acquisition, IP, and other business dimensions however. Maybe unworkable even. But I think there's a world where this actually exists and lowers the barrier to building business-critical software and selling to companies that need a 50-year commitment to risk you.

jayd16 5y ago
I wonder if its worth it for these platforms (like Heroku) to simply add a donation portal. It's not as future proof as something fully open, but it wouldn't need the design, implementation and maintenance of a brand new fully open PAAS ecosystem.

mwattsun 5y ago

I think it's interesting to think about what we have lost because we couldn't keep everything from a 100 years ago and what society 100 years from now will be grateful we preserved. Off the top of my head, we lost a lot of common wisdom in dealing with the flu pandemic of 1918 because personal letters and most newspapers were not preserved. I think 100 years from now they might wish we had preserved more from marginal and/or world communities. What folk wisdom is being lost? Perhaps we need to expand our definition of what is worth saving.

lelandfe 5y ago
An excellent place to start is talking to your parents and grandparents and recording their history and stories online.
- mwattsun 5y ago
  That's a good suggestion I've been following
bananamerica 5y ago
Hard to know what will be of interest for future historians. Some things in which we place great value can be considered irrelevant, while some of our junk can become historical gold.
- mwattsun 5y ago
  > some of our junk can become historical gold That's what interests me. For example, there's a cool repository of 12 step speaker meeting talks hosted in Iceland [1] and frankly, some of the talks are junk, but there's a lot of wisdom. What I find interesting is how it showcases how ordinary citizens talk to each other. The words they use, the accents, the little gems of folk wisdom contained, along with some uncommon stories. This will be valuable 100 years from now if, for example, you want to build a virtual world based in the mid to late 20th century and you want to get the accents correct. What phrases did people use? What were some common misconceptions? Maybe 100 years from now addiction will no longer be a problem. If my virtual world is to be accurate I need to know what it was like for ordinary people when it was a problem. Etc... [1] https://xa-speakers.org/ https://xa-speakers.org/

tonymet 5y ago

Is it really worth archiving ?

maxbond 5y ago
It's difficult to know beforehand! A startup might be a total flop that delivers nothing of value to us now, but it might be a valuable datapoint for future historians to understand how startup culture changed and evolved. Or it may be interesting to future founders - I've seen some startups with perfectly fine ideas fail, and then a few years later, someone succeeds doing something very similar. This may not be the best example, but while I'm sure the Rosetta Stone, being a treaty, likely would have seemed like something worth preserving, would anyone have imagined that it would be the pivotal document in understanding ancient Egyptian? That it would be one of the most important documents of all time?
- tonymet 5y ago
  that's optimistic, and I could see the potential. But I've become a bit suspicious of the content on the internet and the costs in archiving it. Maybe Internet Archive could have a curation threshold . I would rather revert to the internet where quality content was published openly on the web. But presently web content, at least that upranked by Google, is very low quality (seo, clickbait, biased, trivial)

soheil 5y ago

Interesting use cases for IA is to go back to early days of when a startup first launched to see their pitch before they raised real money and added tons of nonsensical bs to their homepage. eg. http://web.archive.org/web/20180901110658/https://www.snowflake.com/ http://web.archive.org/web/20180901110658/https://www.snowfl... http://web.archive.org/web/20140701061721/https://databricks.com/ http://web.archive.org/web/20140701061721/https://databricks...

ineedasername 5y ago

I don't understand why Carmack thinks blockchain should be a component of this. Anyone care to elaborate on how that would make this easier/better?

Jonovono 5y ago
.... He says it right after "to make internet applications that could outlive companies". If it's on the blockchain it doesn't matter if the company storing all of the archives shuts down, the content would still exist, forever, until their is a network running the chain. I suppose something like Torrent could be used?
- Mezzie 5y ago
  He has a point, but it's that private companies shouldn't archives of record. I actually think using blockchain for things like ensuring providence is interesting, since in archives being able to have a clean record of what happened to a piece is VERY useful. It just won't earn a ton of money, so we'll need to wait for the capitalism to burn off to see more not-for-profit uses.
ricardobayes 5y ago
Similarly how github is a blockchain. I think he means the ease of version control by this.
- fs111 5y ago
  Github is a software development tooling provider, not a blockchain
  - rat9988 5y ago
    For his defense, he probably meant Git and typed too quickly. It's still obvious what he meant.
    - 5y ago

nickdothutton 5y ago

I wrote a little about this and the alternatives here, especially for sites with user-generated content which may be impossible to find/regenerate from elsewhere. I call it “Beating the Samson Option”, of pulling the temple down upon your head. https://blog.eutopian.io/beating-the-samson-option/ https://blog.eutopian.io/beating-the-samson-option/

zaps 5y ago

I was with him up until “blockchain”.

midoBB 5y ago

908087 5y ago

polote 5y ago

The idea is more interesting when you think about scrapers. Take any ecommerce website, there are several scrapers that download all pages every hours, it would be more efficient if a provider had a live copy of the website and then serve the requests to the scrapers or could even send webhook. A website could handle tons of scrappers without having high bandwidth, only the provider will need high bandwidth. The issue is that scrapers often play with cookies and dynamic websites and such solution wouldn't work on these cases

akouri 5y ago
scraper*
azalemeth 5y ago
(in the nicest way and since your post is recent, please edit s/scrappers/scrapers! It very much changed how I read it first time around -- I thought you were referring to a type of failed startup!)
- polote 5y ago
  Thanks, I need to improve my english haha
atdrummond 5y ago
Is scraper one “p” or two? My inclination would be scrapper is “scrap”-er rather than “scrape”-er.
pjc50 5y ago
Scrapers. A scrAper is the thing you use to remove ice from windscreens or scrape data from websites. A scraPPer is someone who likes getting into fights or collecting scrap metal.

pjc50 5y ago

I'm old enough to remember when the host-of-record for failed startups was fuckedcompany.com ... I do wonder how many startups actually want to be archived, rather than just ditch everything with unseemly speed as soon as they get acquishutdown.

monkeydust 5y ago
There's huge valuable learnings to be had in failure. Through some coordination they could perhaps get compensation for sharing although this goes against it all being open and free.

nur23kg 5y ago

AlexanderTheGr8 5y ago

What compression does IA use to store websites? Using a 2x better compression will allow them to store 2x more websites/content. I am doing some compression research, and would love to help IA in any way I can. There are some amazing SOTA compression algorithms available now. And if IA ignores images/video, and focuses only on text, they can store an insane amount of websites at a very low cost.

jrwr 5y ago
right now, most of it is coming in as ZSTD with a central index. older stuff was gzip
- AlexanderTheGr8 5y ago
  I am working on a ML-based compression which will be slow to compress/decompress but will give much better compression than zstd/gzip (even 2-3x better). Do you think that it's a useful algorithm for archival reasons or for storing huge amounts of data in cloud?

1vuio0pswjnm7 5y ago

Why is there only one IA? Why is IA not globally distributed, like a CDN? I use IA for "problem" websites, e.g., ones that rely on SNI, i.e., ones hosted at certain CDNs. I simply add add these sites to a list and the local proxy does the rest. http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f list } IA "hosts" an enormous number of sites without the need for SNI (plaintext hostnames sent over the wire). EDIT: @sebow the way they (re)format the HTML is less friendly to the text-only browser I use.

sebow 5y ago
archive.today/.ph/etc is vastly superior for doing the intended purpose here: archive[given that you have the URL obviously] IA is more of an curated internet archive + explorer(which granted is very good).
amenghra 5y ago
What’s your issue with SNI/threat model? If you use a non-SNI site, anyone can tell which site you are visiting since there’s only one domain on that IP.
- yellowapple 5y ago
  Seconding that curiosity. Pretty much every web server I've built uses SNI (at least if it's hosting sites under multiple domains), and the only "downside" of which I'm aware is the lack of IE6 support.
- 1vuio0pswjnm7 5y ago
  There is a difference between making something "impossible" and making something "easier". Performing reverse DNS lookups, or otherwise trying to maintain a global table of 1:1 domain:IP mappings and perform lookups in real-time, is nowhere near as easy nor reliable as sniffing SNI. IME, it is neither easy nor reliable, nor worth the effort. SNI is the preferred method. SNI is easier. SNI is 100% reliable for detecting what hostname the user is trying to access. What is the point of so-called "DNS privacy/Private DNS" if "anyone can tell which site you are visiting" simply by observing IP addresses, without any need to see domainnames. If SNI (plaintext hostnames sent over the wire) is a non-issue, then why are people working on encrypted Client Hello in TLS1.3.

msla 5y ago

I'd love it if the Wayback Machiine were less touchy/more reliable. The failure mode I see very often is that the frontend apparently doesn't know what the backend's doing: The part which ingests URLs and tells you what URLs have been archived does not know what archives the backend has, so it will tell you a page has been archived and give you a link to the archive, but when you click the link, it tells you it does not have the page archived, oh, look, it exists online, would you like to archive it now? Archive it again, and it will tell you that you can only archive a page once every 45 minutes. If you're a weird little obsessive like myself, you go through this process a half-dozen times for one page before it acknowledges that, yes, it does have the page archived (once, mind you) and you can actually see it. While I'm filing bitch reports... The Wayback Machine apparently loves setting cookies. It will set cookies until it has exceeded its own ability to accept cookies, at which point it will give you a blank page and you have to look in the developer console to figure out that it sent you a "too many cookies" error in the response header. I've had to force my browsers to not accept any cookies from the Internet Archive to fix this.

markjgraham 5y ago

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is always open to suggestions about how we might do our jobs better, and improve our service. We also host regular meetups (and have hosted summits and a camp) related to the Decentralized Web. See: https://blog.archive.org/tag/dweb/ https://blog.archive.org/tag/dweb/ The Internet Archive also offers archive-it.org, a subscription service, for those who want a higher level of support and more features. We appreciate any support you can offer, financial and otherwise. Please share any bug reports, feature suggestions and other feedback with us via email to info@archive.org Oh/and… checkout the new PDF Search feature we just launched at the bottom of web.archive.org. More to come like that in 2022. Finally, you might also find some of the things I wrote here of interest: https://gijn.org/2021/05/05/tips-for-using-the-internet-archives-wayback-machine-in-your-next-investigation/ https://gijn.org/2021/05/05/tips-for-using-the-internet-arch...

sayhar 5y ago
Thank you for your service!
chaps 5y ago
I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals. Cheers.
- gjs278 5y ago

thrdbndndn 5y ago

I genuinely didn't understand the point he's trying to make. Could someone ELI5?

AtNightWeCode 5y ago

How about a default way of building iterative versioned static web sites where dynamic content is separated and used only for dynamic data. Meaning, most sites can easily be stored offline.

iJohnDoe 5y ago

IA needs to respects robots.txt and they need to make it easier to request data to be removed. Not everyone fully supports the IA mission and they need to respect that view as much as they respect their supporters.

smarx007 5y ago
They clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ctrl+S and uploading it to IA. Removing data from the web in 2021? Hmm… https://web.archive.org/web/20211222032633/https://news.ycombinator.com/item?id=29645135 https://web.archive.org/web/20211222032633/https://news.ycom... oops!
- iJohnDoe 5y ago
  Are you indicating that everything in the IA was added via a person clicking on “archive this page”? I don’t think that is correct. A lot of it was added via automated methods?
  - smarx007 5y ago
    Not necessarily via a button but all of the pages were submitted to them by 3rd parties. So, they don't seem to crawl to discover new URIs eagerly, see https://news.ycombinator.com/item?id=29643506 https://news.ycombinator.com/item?id=29643506 For example, I host my own ArchiveBox at home (you get fulltext search as a bonus) and it is configured to submit every URL I save to IA: https://imgur.com/a/Yhnxo1W https://imgur.com/a/Yhnxo1W IA considers that to be manual submission not subject to robots.txt rules. https://archivebox.io https://archivebox.io