4 ms·

I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the cha

I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.

nitrogen 5y ago
But the wayback machine is pretty klunky and slow, honestly. This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot. They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.
jandrese 5y ago
This is true, but they're also taking on an absolutely monumental task on a shoestring budget. I continue to be amazed at what they are able to accomplish. There aren't many heroes on the internet, but the Internet Archive team qualifies.
- jonah-archive 5y ago
  Thank you! We're doing what we can with what have where we are, or maybe just what we must because we can, depending on your preferred aphorism :)
bobsmooth 5y ago
Slow, yes. Klunky? Absolutely not. When you make a request to the Internet Archive, you're searching through a massive amount of data. The fact that it only takes a few seconds to pull up a decade-old webpage is amazing.
1vuio0pswjnm7 5y ago
"But the wayback machine is pretty klunky and slow, honestly." archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes) usage 1: (simple html page of all results) echo https://www.theatlantic.com/article/619320/|1.sh >1.htm firefox ./1.htm usage 2: (retrieve last result) echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm firefox ./2.htm #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p); esac haproxy + nc version (965 bytes) maybe it is faster than curl, maybe not; you be the judge #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80; esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
- 1vuio0pswjnm7 5y ago
  Edit: Remove Wayback Machine's Javascript inserts (usage #2) curl version #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"; esac haproxy + nc version #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;" esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
jonah-archive 5y ago
We've got a pretty solid diff view now, notes at https://blog.archive.org/2019/10/18/the-wayback-machine-fighting-digital-extinction-in-new-ways/ https://blog.archive.org/2019/10/18/the-wayback-machine-figh... and an example at https://web.archive.org/web/diff/20170118202526/20170120040337/https://www.ice.gov/speeches https://web.archive.org/web/diff/20170118202526/201701200403... Re slowness, we're able to do a lot with a little, but there's always room for improvement. If you're interested in some of the specific infrastructural challenges, I did a presentation in February: https://archive.org/details/jonah-edwards-presentation https://archive.org/details/jonah-edwards-presentation and my colleague did a fantastic presentation detailing some of the internal workings of the Wayback Machine just last week: https://archive.org/details/bridget-bell-presentation https://archive.org/details/bridget-bell-presentation
- deleted 5y ago
  [deleted]
- neolog 5y ago
  Do you have statistics on what fraction of the pages on major outlets you have archived, and how often they change?

I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.

nitrogen 5y ago

But the wayback machine is pretty klunky and slow, honestly. This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot. They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.

jandrese 5y ago

This is true, but they're also taking on an absolutely monumental task on a shoestring budget. I continue to be amazed at what they are able to accomplish. There aren't many heroes on the internet, but the Internet Archive team qualifies.

jonah-archive 5y ago
Thank you! We're doing what we can with what have where we are, or maybe just what we must because we can, depending on your preferred aphorism :)

bobsmooth 5y ago

Slow, yes. Klunky? Absolutely not. When you make a request to the Internet Archive, you're searching through a massive amount of data. The fact that it only takes a few seconds to pull up a decade-old webpage is amazing.

1vuio0pswjnm7 5y ago

"But the wayback machine is pretty klunky and slow, honestly." archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes) usage 1: (simple html page of all results) echo https://www.theatlantic.com/article/619320/|1.sh >1.htm firefox ./1.htm usage 2: (retrieve last result) echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm firefox ./2.htm #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p); esac haproxy + nc version (965 bytes) maybe it is faster than curl, maybe not; you be the judge #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80; esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

1vuio0pswjnm7 5y ago
Edit: Remove Wayback Machine's Javascript inserts (usage #2) curl version #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"; esac haproxy + nc version #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;" esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

jonah-archive 5y ago

We've got a pretty solid diff view now, notes at https://blog.archive.org/2019/10/18/the-wayback-machine-fighting-digital-extinction-in-new-ways/ https://blog.archive.org/2019/10/18/the-wayback-machine-figh... and an example at https://web.archive.org/web/diff/20170118202526/20170120040337/https://www.ice.gov/speeches https://web.archive.org/web/diff/20170118202526/201701200403... Re slowness, we're able to do a lot with a little, but there's always room for improvement. If you're interested in some of the specific infrastructural challenges, I did a presentation in February: https://archive.org/details/jonah-edwards-presentation https://archive.org/details/jonah-edwards-presentation and my colleague did a fantastic presentation detailing some of the internal workings of the Wayback Machine just last week: https://archive.org/details/bridget-bell-presentation https://archive.org/details/bridget-bell-presentation

deleted 5y ago
[deleted]
neolog 5y ago
Do you have statistics on what fraction of the pages on major outlets you have archived, and how often they change?