4 ms·

Any nice tool to archive web sites using the warc format?

• aussieguy1234 2y ago
  The Wikipedia article on the warc format references wget
• ricopags 2y ago
  May be some here https://github.com/iipc/awesome-web-archiving/blob/main/README.md#warc-io-libraries https://github.com/iipc/awesome-web-archiving/blob/main/READ...
• toomuchtodo 2y ago
  https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem
• andrewmackrodt 2y ago
  If you want a full fidelity warc file, browsertrix-crawler is nice. It is slower than wget on account of using chrome, but it works better for sites with highly dynamic content and can generate wacz files which can be efficiently served via a file/objectstore when used with something like replayweb.page.
• nikisweeting 2y ago
  Webrecorder's are by far the best imo. https://archiveweb.page https://archiveweb.page and Browsertrix Crawler. ArchiveBox uses wget internally for it's WARC generation but I'd love to integrate it with Webrecorder in the future for this part.