5 ms·

I didn't see any articles relating to how it was decentralized. Is internet archive going for something similar to a blockchain setup, because that would actual

I didn't see any articles relating to how it was decentralized. Is internet archive going for something similar to a blockchain setup, because that would actually be one of the few cases it would make sense I am not familiar with this new dweb subdomain, what is unique about it? Archive has actually lost some of the archives I stored on it, which is weird, because I ran multitude of backups on it a few years back when the site got taken offline

justin_ 8y ago
I wanted to submit an article about this but couldn't find anything either. My understanding is that this loads content from various protocols (listed at the top of the page), many of which support replicating data in a decentralized way. As far as I know, there's no blockchain involved in anything here yet. If you browse to the Community Video section and choose a video, you can see peer information as though downloading through WebTorrent. If I disable WebTorrent and look at a video, I don't see the peer information and it seems to fall back to HTTP. Pretty cool! It looks like almost everything is only seeded by the Internet Archive right now, but hopefully they want to encourage more people to participate.
- Kagerjay 8y ago
  Hm that's interesting, I imagine the internet archive has lots of seedboxes/CDNs/datacenters distributed globally. I have no idea how much information the internet archive is currently backing up though, but its growing at an increasing rate. I would love to see a write up of how their infrastructure works though
  - toomuchtodo 8y ago
    My understanding is all of their digital content is stored and served from their location in SF, although it would be awesome if they started geographically distributing their storage nodes. http://archive.org/web/petabox.php http://archive.org/web/petabox.php
    - ris 8y ago
      Note this is an extremely old webpage.
    - mmt 8y ago
      Although, as a sibling comment points out, that article is very old, I can attest to its continuing general accuracy, having interviewed there last year. Although I didn't get a look at their financials, my overall impression is that they can't afford anything as extravagant as geographic distribution, absent a huge corporate sponsor or two. Even a modest increase in donation revenue would be unlikely to make a difference, as the other impression I had was that spending in that area was far from a priority, especially compared with data acquisition/conversion projects. I think it's great that the IA is attempting to be a broad, general-interest digital library, since nobody else is doing that. However, I also wish for a separate archive focused specifically on The Web. Besides not being subject to distractions or competing interests, I speculate that it would be more attractive for web-dependent companies, such as Google or CDNs, to donate to.
    - sp332 8y ago
      There's been at least a partial copy in Alexandria and Amsterdam for a long time, and they opened a full replica in Canada last year. At least I think the Canada one is done, can't find a blog post saying it was finished. https://blog.archive.org/2016/12/03/faqs-about-the-internet-archive-canada/ https://blog.archive.org/2016/12/03/faqs-about-the-internet-...
      - toomuchtodo 8y ago
        Was not aware; this is excellent news!
- rolandog 8y ago
  Do you think it's built on top of PeerTube Social or is it something completely different?
PhantomGremlin 8y ago
Archive has actually lost some of the archives I stored on it Who currently controls the domain? One of the ways stuff can be lost is if a new domain owner fiddles with robots.txt. (The Archive has recently changed their policy about that) We have also seen an upsurge of the use of robots.txt files to remove entire domains from search engines when they transition from a live web site into a parked domain, which has historically also removed the entire domain from view in the Wayback Machine. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- textmode 8y ago
  Is there a delay between the time the robots.txt changes and the time when the content becomes inaccessible via Wayback Machine? How often does the archive.org_bot crawl robots.txt? Can a script check robots.txt periodically for changes and if changes are detected, then download the content from Wayback Machine before it becomes inaccessible? Additionally, can a script check the domain registration for an anticipated expiration date, or perhaps monitor domainname "drop lists"?
mitraardron 8y ago
There's a video of the talk I did with Arkadiy (from IPFS) and Feross (from WebTorrent) at the Dweb Summit at https://archive.org/details/youtube-eO6pYYWZBs8?start=10447 https://archive.org/details/youtube-eO6pYYWZBs8?start=10447 it goes into quite a lot of detail about how it was implemented and the challenges we faced along the way.

justin_ 8y ago

I wanted to submit an article about this but couldn't find anything either. My understanding is that this loads content from various protocols (listed at the top of the page), many of which support replicating data in a decentralized way. As far as I know, there's no blockchain involved in anything here yet. If you browse to the Community Video section and choose a video, you can see peer information as though downloading through WebTorrent. If I disable WebTorrent and look at a video, I don't see the peer information and it seems to fall back to HTTP. Pretty cool! It looks like almost everything is only seeded by the Internet Archive right now, but hopefully they want to encourage more people to participate.

Kagerjay 8y ago
Hm that's interesting, I imagine the internet archive has lots of seedboxes/CDNs/datacenters distributed globally. I have no idea how much information the internet archive is currently backing up though, but its growing at an increasing rate. I would love to see a write up of how their infrastructure works though
- toomuchtodo 8y ago
  My understanding is all of their digital content is stored and served from their location in SF, although it would be awesome if they started geographically distributing their storage nodes. http://archive.org/web/petabox.php http://archive.org/web/petabox.php
  - ris 8y ago
    Note this is an extremely old webpage.
  - mmt 8y ago
    Although, as a sibling comment points out, that article is very old, I can attest to its continuing general accuracy, having interviewed there last year. Although I didn't get a look at their financials, my overall impression is that they can't afford anything as extravagant as geographic distribution, absent a huge corporate sponsor or two. Even a modest increase in donation revenue would be unlikely to make a difference, as the other impression I had was that spending in that area was far from a priority, especially compared with data acquisition/conversion projects. I think it's great that the IA is attempting to be a broad, general-interest digital library, since nobody else is doing that. However, I also wish for a separate archive focused specifically on The Web. Besides not being subject to distractions or competing interests, I speculate that it would be more attractive for web-dependent companies, such as Google or CDNs, to donate to.
  - sp332 8y ago
    There's been at least a partial copy in Alexandria and Amsterdam for a long time, and they opened a full replica in Canada last year. At least I think the Canada one is done, can't find a blog post saying it was finished. https://blog.archive.org/2016/12/03/faqs-about-the-internet-archive-canada/ https://blog.archive.org/2016/12/03/faqs-about-the-internet-...
    - toomuchtodo 8y ago
      Was not aware; this is excellent news!
rolandog 8y ago
Do you think it's built on top of PeerTube Social or is it something completely different?

PhantomGremlin 8y ago

Archive has actually lost some of the archives I stored on it Who currently controls the domain? One of the ways stuff can be lost is if a new domain owner fiddles with robots.txt. (The Archive has recently changed their policy about that) We have also seen an upsurge of the use of robots.txt files to remove entire domains from search engines when they transition from a live web site into a parked domain, which has historically also removed the entire domain from view in the Wayback Machine. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

textmode 8y ago
Is there a delay between the time the robots.txt changes and the time when the content becomes inaccessible via Wayback Machine? How often does the archive.org_bot crawl robots.txt? Can a script check robots.txt periodically for changes and if changes are detected, then download the content from Wayback Machine before it becomes inaccessible? Additionally, can a script check the domain registration for an anticipated expiration date, or perhaps monitor domainname "drop lists"?

mitraardron 8y ago

There's a video of the talk I did with Arkadiy (from IPFS) and Feross (from WebTorrent) at the Dweb Summit at https://archive.org/details/youtube-eO6pYYWZBs8?start=10447 https://archive.org/details/youtube-eO6pYYWZBs8?start=10447 it goes into quite a lot of detail about how it was implemented and the challenges we faced along the way.