4 ms·

This looks very nice As an aside, wget -r -k -np works surprisingly well for my offline needs. For permanent access I defer to archive.is

by platz 8y ago

toomuchtodo 8y ago
Consider https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site (ArchiveTeam/grab-site) in the future. There is a difference between fetching/mirroring content and recording the http header request and responses when retrieving all objects for a site. I believe there is even a docker container for a quick pull and run.
- j88439h84 8y ago
  When I want to archive a site, don't I want the content, not just the headers? How do I save the content?
  - toomuchtodo 8y ago
    When archiving, you want the request and response headers and the content. Grab-site does that (any tool really that'll write WARC files). Sorry if my comment was ambiguous in that regard. https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... If you wanted to write WARC files out with wget, you'd use the options specified in https://www.archiveteam.org/index.php?title=Wget_with_WARC_output https://www.archiveteam.org/index.php?title=Wget_with_WARC_o...
    - progval 8y ago
      Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.
      - toomuchtodo 8y ago
        Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.
        nikisweeting 8y ago
        We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space. You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130 https://github.com/pirate/ArchiveBox/issues/130

This looks very nice As an aside, wget -r -k -np works surprisingly well for my offline needs. For permanent access I defer to archive.is

by platz 8y ago

This looks very nice As an aside, wget -r -k -np works surprisingly well for my offline needs. For permanent access I defer to archive.is

toomuchtodo 8y ago

Consider https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site (ArchiveTeam/grab-site) in the future. There is a difference between fetching/mirroring content and recording the http header request and responses when retrieving all objects for a site. I believe there is even a docker container for a quick pull and run.

j88439h84 8y ago
When I want to archive a site, don't I want the content, not just the headers? How do I save the content?
- toomuchtodo 8y ago
  When archiving, you want the request and response headers and the content. Grab-site does that (any tool really that'll write WARC files). Sorry if my comment was ambiguous in that regard. https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... If you wanted to write WARC files out with wget, you'd use the options specified in https://www.archiveteam.org/index.php?title=Wget_with_WARC_output https://www.archiveteam.org/index.php?title=Wget_with_WARC_o...
  - progval 8y ago
    Depends on what you want to use your archive for. If it's just for opening it locally in a web browser, mirroring the content is enough.
    - toomuchtodo 8y ago
      Yes, but if the effort to go the WARC route is minimal above mirroring, might as well do it. You never know if you’ll need data you didn’t grab.
      - nikisweeting 8y ago
        We intend to store both the WARC of the headless broswer render and the wget clone. (I'm the creator @pirate on Github). The idea is to archive anything in many redundant formats (HTML clone, WARC, PDF, etc) and for people to run this on a deduplicated filesystem like ZFS/BTRFS or disable specific methods if they care about saving space. You can track the progress on implementing WARC saving for all archived resources here: https://github.com/pirate/ArchiveBox/issues/130 https://github.com/pirate/ArchiveBox/issues/130