5 ms·

On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighti

by tda 5y ago

On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighting the most historically important articles.

throwawayboise 5y ago
I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.
- nitrogen 5y ago
  But the wayback machine is pretty klunky and slow, honestly. This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot. They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.
- jandrese 5y ago
  This is true, but they're also taking on an absolutely monumental task on a shoestring budget. I continue to be amazed at what they are able to accomplish. There aren't many heroes on the internet, but the Internet Archive team qualifies.
  - jonah-archive 5y ago
    Thank you! We're doing what we can with what have where we are, or maybe just what we must because we can, depending on your preferred aphorism :)
- bobsmooth 5y ago
  Slow, yes. Klunky? Absolutely not. When you make a request to the Internet Archive, you're searching through a massive amount of data. The fact that it only takes a few seconds to pull up a decade-old webpage is amazing.
- 1vuio0pswjnm7 5y ago
  "But the wayback machine is pretty klunky and slow, honestly." archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes) usage 1: (simple html page of all results) echo https://www.theatlantic.com/article/619320/|1.sh >1.htm firefox ./1.htm usage 2: (retrieve last result) echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm firefox ./2.htm #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p); esac haproxy + nc version (965 bytes) maybe it is faster than curl, maybe not; you be the judge #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80; esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
  - 1vuio0pswjnm7 5y ago
    Edit: Remove Wayback Machine's Javascript inserts (usage #2) curl version #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"; esac haproxy + nc version #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;" esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
- jonah-archive 5y ago
  We've got a pretty solid diff view now, notes at https://blog.archive.org/2019/10/18/the-wayback-machine-fighting-digital-extinction-in-new-ways/ https://blog.archive.org/2019/10/18/the-wayback-machine-figh... and an example at https://web.archive.org/web/diff/20170118202526/20170120040337/https://www.ice.gov/speeches https://web.archive.org/web/diff/20170118202526/201701200403... Re slowness, we're able to do a lot with a little, but there's always room for improvement. If you're interested in some of the specific infrastructural challenges, I did a presentation in February: https://archive.org/details/jonah-edwards-presentation https://archive.org/details/jonah-edwards-presentation and my colleague did a fantastic presentation detailing some of the internal workings of the Wayback Machine just last week: https://archive.org/details/bridget-bell-presentation https://archive.org/details/bridget-bell-presentation
  - deleted 5y ago
    [deleted]
  - neolog 5y ago
    Do you have statistics on what fraction of the pages on major outlets you have archived, and how often they change?
deadalus 5y ago
Wayback Machine censors many websites(like 4chan) from being 'saved'. Wayback Machine also removes previously archived videos/websites in certain cases. They are not neutral.
- Zababa 5y ago
  Fortunately 4chan has (unofficial) archives, but some content was probably lost.
- thrashh 5y ago
  Don’t spread misinformation. The Wayback Machine is not censoring 4chan. 4chan is ‘censoring’ the Wayback Machine. https://www.4chan.org/robots.txt https://www.4chan.org/robots.txt Also, I let a domain of mine expire and the new domain owner (which just plastered ads) had a robots.txt that retroactively removed my “previously archived website” from the Wayback Machine.
  - a1369209993 5y ago
    > Don't spread misinformation. No you. > The Wayback Machine is not censoring 4chan. 4chan is 'censoring' the Wayback Machine. 4chan is not the operator of the server at web.archive.org. Compliance with robots.txt restrictions is 100% the Internet Archive's fault.
  - anoncake 5y ago
    4chan asks the WM to censor it, but the WM actually does it.
ajsnigrutin 5y ago
This works for major sites like BBC. For minor, local news sites, it doesn't because they don't get archived at all (atleast not fully and in time). news sites should really be mandated to keep previous versions of their newsstories with all the edits, especially the ones paid by taxes.

by tda 5y ago

throwawayboise 5y ago

I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.

nitrogen 5y ago
But the wayback machine is pretty klunky and slow, honestly. This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot. They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.
jandrese 5y ago
This is true, but they're also taking on an absolutely monumental task on a shoestring budget. I continue to be amazed at what they are able to accomplish. There aren't many heroes on the internet, but the Internet Archive team qualifies.
- jonah-archive 5y ago
  Thank you! We're doing what we can with what have where we are, or maybe just what we must because we can, depending on your preferred aphorism :)
bobsmooth 5y ago
Slow, yes. Klunky? Absolutely not. When you make a request to the Internet Archive, you're searching through a massive amount of data. The fact that it only takes a few seconds to pull up a decade-old webpage is amazing.
1vuio0pswjnm7 5y ago
"But the wayback machine is pretty klunky and slow, honestly." archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes) usage 1: (simple html page of all results) echo https://www.theatlantic.com/article/619320/|1.sh >1.htm firefox ./1.htm usage 2: (retrieve last result) echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm firefox ./2.htm #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p); esac haproxy + nc version (965 bytes) maybe it is faster than curl, maybe not; you be the judge #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80; esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
- 1vuio0pswjnm7 5y ago
  Edit: Remove Wayback Machine's Javascript inserts (usage #2) curl version #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"; esac haproxy + nc version #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \ |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/; s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;" esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
jonah-archive 5y ago
We've got a pretty solid diff view now, notes at https://blog.archive.org/2019/10/18/the-wayback-machine-fighting-digital-extinction-in-new-ways/ https://blog.archive.org/2019/10/18/the-wayback-machine-figh... and an example at https://web.archive.org/web/diff/20170118202526/20170120040337/https://www.ice.gov/speeches https://web.archive.org/web/diff/20170118202526/201701200403... Re slowness, we're able to do a lot with a little, but there's always room for improvement. If you're interested in some of the specific infrastructural challenges, I did a presentation in February: https://archive.org/details/jonah-edwards-presentation https://archive.org/details/jonah-edwards-presentation and my colleague did a fantastic presentation detailing some of the internal workings of the Wayback Machine just last week: https://archive.org/details/bridget-bell-presentation https://archive.org/details/bridget-bell-presentation
- deleted 5y ago
  [deleted]
- neolog 5y ago
  Do you have statistics on what fraction of the pages on major outlets you have archived, and how often they change?

deadalus 5y ago

Wayback Machine censors many websites(like 4chan) from being 'saved'. Wayback Machine also removes previously archived videos/websites in certain cases. They are not neutral.

Zababa 5y ago
Fortunately 4chan has (unofficial) archives, but some content was probably lost.
thrashh 5y ago
Don’t spread misinformation. The Wayback Machine is not censoring 4chan. 4chan is ‘censoring’ the Wayback Machine. https://www.4chan.org/robots.txt https://www.4chan.org/robots.txt Also, I let a domain of mine expire and the new domain owner (which just plastered ads) had a robots.txt that retroactively removed my “previously archived website” from the Wayback Machine.
- a1369209993 5y ago
  > Don't spread misinformation. No you. > The Wayback Machine is not censoring 4chan. 4chan is 'censoring' the Wayback Machine. 4chan is not the operator of the server at web.archive.org. Compliance with robots.txt restrictions is 100% the Internet Archive's fault.
- anoncake 5y ago
  4chan asks the WM to censor it, but the WM actually does it.

ajsnigrutin 5y ago

This works for major sites like BBC. For minor, local news sites, it doesn't because they don't get archived at all (atleast not fully and in time). news sites should really be mandated to keep previous versions of their newsstories with all the edits, especially the ones paid by taxes.