3 ms·
"Is there some straightforward way to list all of archive.org's snapshots (of a particular site) without a javascript-enabled browser?" https://archive.org/ser
by 1vuio0pswjnm7 4y ago
"Is there some straightforward way to list all of archive.org's snapshots (of a particular site) without a javascript-enabled browser?"
https://archive.org/services/docs/api/wayback-cdx-server.html https://archive.org/services/docs/api/wayback-cdx-server.htm...
FWIW, below is a quick and dirty script I use for a variety of purposes, such as accessing www search result URLs so I do not have to (a) use sites that do not support TLS1.3, (b) use sites that require SNI or (c) use DNS. I will call this script "www".
Example usage:
alias links="links -no-connect"
x=https://ranprieur.com
# retrieve first 5 snapshots (default)
echo $x|www >1.htm
# retrieve first 3 snapshots
echo $x|www 3 >1.htm
# retrieve last 3 snapshots
echo $x|www -3 >1.htm
# retrieve all snapshots
echo $x|www 0 >1.htm
links 1.htm
#!/bin/sh
LIMIT=${1-5};
read x0;
x0=$(echo $x0|sed 's/%/&&/g');
x1=web.archive.org;
curl -A "" "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original&limit=$LIMIT&showDupeCount=true" \
|(echo "<h2> ${x0}</h2><ol><pre>";sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/${x1}\/web\/\1if_\/\2>\1<\/a>/;s/ </</;s/ //2p;}");
echo "</ol></pre>";
NB. I used curl here because this is an example for HN. That does not mean I am a curl user.
I also have a small script I use for the Common Crawl archives. They also use CDX but the results are WARC files compressed with gzip. I wrote a small program in C to extract the gzip'd results after HTTP/1.1 pipelining. For retrieving results without pipelining (i.e., many TCP connections), I modified tnftp to accept a Range header.
- marttt 4y agoThis is excellent, many thanks for sharing. I don't know how I didn't think that Wayback Machine might maybe also have an API. :/ Also, lots of interesting stuff for things like the above on Common Crawl: https://commoncrawl.org/the-data/examples/ https://commoncrawl.org/the-data/examples/ I guess my text-only browsing just got a bunch of extra batteries (thus far simply w3m + a few wget-etc scripts).