I had a look into this as part of a research project. To reduce the burden of writing specific scraping software, I investigated the software listed by the Arc

I had a look into this as part of a research project. To reduce the burden of writing specific scraping software, I investigated the software listed by the Archive Team[0]. The command line tools and libraries aside (as they would require much more specific tailoring to make them work), I was particularly interested in HTTrack and Warrick. Warrick[1] is defined as a tool to recover lost websites using various online archives and caches. Warrick[2] requires some expert knowledge in order to get it up and running. I found Warrick was a bit outdated, so decided to try something similar but more up to date, and came across this (hartator/wayback_machine_downloader). I found this to be a bit easier to work with as it would allow me to download snapshots within a time period, which is what I needed for this project. After running it for ~12 hours on my local machine, it still had not completed, downloading only 11768 of 94518 files. Instead, I found myself writing a Python based tool that could fetch with much more accuracy using the CDX server[3] and filtering by date, targeting only certain fails and allow for multi threading. In order to improve the process, and narrow down the data of what is needed, I scoped to just July and December timeframes per year, from 2010 to this year, only targeting html files. For example: http://web.archive.org/cdx/search/cdx?url=%s&matchType=prefix&limit=900000&output=json&filter=statuscode:200&filter=mimetype:text/html&collapse=timestamp:6&from=2010&to=2020 http://web.archive.org/cdx/search/cdx?url=%s&matchType=prefi... Hopefully someone finds this useful. [0] https://archiveteam.org/index.php?title=Software https://archiveteam.org/index.php?title=Software [1] https://github.com/oduwsdl/warrick https://github.com/oduwsdl/warrick [2] https://code.google.com/archive/p/warrick/wikis/About_Warrick.wiki https://code.google.com/archive/p/warrick/wikis/About_Warric... [3] https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server https://github.com/internetarchive/wayback/tree/master/wayba...