11 ms·

It's amazing how little some trusted institutions care about this. For example, the BBC has been bragging about how many people rely on their coverage of the pa

by makomk 5y ago

It's amazing how little some trusted institutions care about this. For example, the BBC has been bragging about how many people rely on their coverage of the pandemic, but have an obnoxious habit of repeatedly overwriting old articles with new ones on similar topics and not keeping the old versions available. The history of a once-in-a-century pandemic with huge local and global impacts is literally being overwritten day by day. Sometimes this helps them whitewash their screw-ups which have lead to widespread false beliefs. For example, after the UK government targetted and hit 100,000 Covid-19 tests in a day, the BBC ran an article falsely claiming Germany had achieved this a month earlier and linked it prominently on their news front page for about a month. A large proportion of the population probably saw this and now falsely believe it, it got brought up all the time as part of the narrative that the government's big "world-leading" achievements were just playing catch up badly, but it was memory-holed from the article in a rewrite and they used that as an excuse for not publishing any correction - so unless historians dig deep in third-party archives, they'd never understand where that belief came from. (Apparently a previous version of the article also wrongly claimed France was carrying out more Covid-19 tests due to mistaking their weekly numbers for daily one, according to a correction which disappeared from the article after a few days and only exists in the Internet Archive now. I haven't been able to find the original version of that claim.)

tda 5y ago
On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighting the most historically important articles.
- throwawayboise 5y ago
  I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.
  - nitrogen 5y ago
    But the wayback machine is pretty klunky and slow, honestly. This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot. They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.
  - jandrese 5y ago
    This is true, but they're also taking on an absolutely monumental task on a shoestring budget. I continue to be amazed at what they are able to accomplish. There aren't many heroes on the internet, but the Internet Archive team qualifies.
    - jonah-archive 5y ago
      Thank you! We're doing what we can with what have where we are, or maybe just what we must because we can, depending on your preferred aphorism :)
  - bobsmooth 5y ago
    Slow, yes. Klunky? Absolutely not. When you make a request to the Internet Archive, you're searching through a massive amount of data. The fact that it only takes a few seconds to pull up a decade-old webpage is amazing.
  - 1vuio0pswjnm7 5y ago
    "But the wayback machine is pretty klunky and slow, honestly." archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes) usage 1: (simple html page of all results) echo https://www.theatlantic.com/article/619320/|1.sh >1.htm firefox ./1.htm usage 2: (retrieve last result) echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm firefox ./2.htm #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p); esac haproxy + nc version (965 bytes) maybe it is faster than curl, maybe not; you be the judge #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80; esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
- deadalus 5y ago
  Wayback Machine censors many websites(like 4chan) from being 'saved'. Wayback Machine also removes previously archived videos/websites in certain cases. They are not neutral.
  - Zababa 5y ago
    Fortunately 4chan has (unofficial) archives, but some content was probably lost.
  - thrashh 5y ago
    Don’t spread misinformation. The Wayback Machine is not censoring 4chan. 4chan is ‘censoring’ the Wayback Machine. https://www.4chan.org/robots.txt https://www.4chan.org/robots.txt Also, I let a domain of mine expire and the new domain owner (which just plastered ads) had a robots.txt that retroactively removed my “previously archived website” from the Wayback Machine.
    - a1369209993 5y ago
      > Don't spread misinformation. No you. > The Wayback Machine is not censoring 4chan. 4chan is 'censoring' the Wayback Machine. 4chan is not the operator of the server at web.archive.org. Compliance with robots.txt restrictions is 100% the Internet Archive's fault.
    - anoncake 5y ago
      4chan asks the WM to censor it, but the WM actually does it.
- ajsnigrutin 5y ago
  This works for major sites like BBC. For minor, local news sites, it doesn't because they don't get archived at all (atleast not fully and in time). news sites should really be mandated to keep previous versions of their newsstories with all the edits, especially the ones paid by taxes.
throwaways885 5y ago
> so unless historians dig deep in third-party archives, they'd never understand where that belief came from I expect future historical tooling will exist to solve exactly this problem. Assuming Archive.org and the like nabbed it, the evidence is all there for future generations to see.
- pessimizer 5y ago
  Assuming archive.org isn't shut down and deleted by court order in some future lawsuit.
mountainb 5y ago
The BBC publishes nothing but garbage and there are other extant sources that are more durable. It's fine to forget things. We're missing entire libraries of classical literature from great authors which would be nice to have. Missing documentary sewage isn't a tragedy. This should show us that most of the web isn't worth preserving anyway, much like McDonald's burger wrappers aren't worth preserving like sacred artifacts. Most web and social media content is worth less than said greasy burger wrappers.
- CodeGlitch 5y ago
  Reading one of the BBC's technical articles, a cyber security news item, they had 3 errors in the first paragraph. I didn't bother reading to the end of the article. I'm glad I no longer pay for a TV license.
  - dotBen 5y ago
    The BBC (News's) tech section isn't aimed at you. Inaccuracies shouldn't be there but often they will dumb down or gloss over stuff for the mainstream audience they are aiming at. You notice it cos you are in tech, but the same happens in financial news, science and even sport. Go read a tech publication. For shits and giggle I did once try to get a technical story on how to copy DVD's published - it got very heavily edited! http://news.bbc.co.uk/2/hi/science/nature/1987665.stm http://news.bbc.co.uk/2/hi/science/nature/1987665.stm (I'm a former + early BBC News website employee)
    - throwaways885 5y ago
      Aside: That old version of BBC News is an absolute gem of history. Especially looking at some of the recommended sidebar stories: > Britons 'baffled over euro rate' > Wireless internet arrives in China > Mobile spam on the rise Fascinating to see how much our problems have stayed the same, despite the changing context. I hope this is considered 'archived' and not 'forgotten'.
  - jandrese 5y ago
    It has always been a constant of journalism that you read an article in your field and go "Wow, this is terrible, they got all of the details wrong". But then you turn around and trust the reporting on everything outside of your field of expertise.
brodouevencode 5y ago
How much of these overwrites are cover ups for failures? Certainly some percentage, just not sure what.

by makomk 5y ago

tda 5y ago

On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighting the most historically important articles.

throwawayboise 5y ago
I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.
- nitrogen 5y ago
  But the wayback machine is pretty klunky and slow, honestly. This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot. They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.
- jandrese 5y ago
  This is true, but they're also taking on an absolutely monumental task on a shoestring budget. I continue to be amazed at what they are able to accomplish. There aren't many heroes on the internet, but the Internet Archive team qualifies.
  - jonah-archive 5y ago
    Thank you! We're doing what we can with what have where we are, or maybe just what we must because we can, depending on your preferred aphorism :)
- bobsmooth 5y ago
  Slow, yes. Klunky? Absolutely not. When you make a request to the Internet Archive, you're searching through a massive amount of data. The fact that it only takes a few seconds to pull up a decade-old webpage is amazing.
- 1vuio0pswjnm7 5y ago
  "But the wayback machine is pretty klunky and slow, honestly." archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes) usage 1: (simple html page of all results) echo https://www.theatlantic.com/article/619320/|1.sh >1.htm firefox ./1.htm usage 2: (retrieve last result) echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm firefox ./2.htm #!/bin/sh read x0; x1=web.archive.org; curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p); esac haproxy + nc version (965 bytes) maybe it is faster than curl, maybe not; you be the judge #!/bin/sh read x0; x1=web.archive.org; printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \ |exec haproxy -D -f /dev/stdin; printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \ \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \ |case $# in :) ;;0)( printf "<h2>     $x0</h2><ol><pre>\n"; sed -n "/[0-9]\{14\} [hf]/{s/$.* $$.*$/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}"; printf "</ol></pre><br>\n" ) ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/$.* $$.*$/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80; esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi
deadalus 5y ago
Wayback Machine censors many websites(like 4chan) from being 'saved'. Wayback Machine also removes previously archived videos/websites in certain cases. They are not neutral.
- Zababa 5y ago
  Fortunately 4chan has (unofficial) archives, but some content was probably lost.
- thrashh 5y ago
  Don’t spread misinformation. The Wayback Machine is not censoring 4chan. 4chan is ‘censoring’ the Wayback Machine. https://www.4chan.org/robots.txt https://www.4chan.org/robots.txt Also, I let a domain of mine expire and the new domain owner (which just plastered ads) had a robots.txt that retroactively removed my “previously archived website” from the Wayback Machine.
  - a1369209993 5y ago
    > Don't spread misinformation. No you. > The Wayback Machine is not censoring 4chan. 4chan is 'censoring' the Wayback Machine. 4chan is not the operator of the server at web.archive.org. Compliance with robots.txt restrictions is 100% the Internet Archive's fault.
  - anoncake 5y ago
    4chan asks the WM to censor it, but the WM actually does it.
ajsnigrutin 5y ago
This works for major sites like BBC. For minor, local news sites, it doesn't because they don't get archived at all (atleast not fully and in time). news sites should really be mandated to keep previous versions of their newsstories with all the edits, especially the ones paid by taxes.

throwaways885 5y ago

> so unless historians dig deep in third-party archives, they'd never understand where that belief came from I expect future historical tooling will exist to solve exactly this problem. Assuming Archive.org and the like nabbed it, the evidence is all there for future generations to see.

pessimizer 5y ago
Assuming archive.org isn't shut down and deleted by court order in some future lawsuit.

mountainb 5y ago

The BBC publishes nothing but garbage and there are other extant sources that are more durable. It's fine to forget things. We're missing entire libraries of classical literature from great authors which would be nice to have. Missing documentary sewage isn't a tragedy. This should show us that most of the web isn't worth preserving anyway, much like McDonald's burger wrappers aren't worth preserving like sacred artifacts. Most web and social media content is worth less than said greasy burger wrappers.

CodeGlitch 5y ago
Reading one of the BBC's technical articles, a cyber security news item, they had 3 errors in the first paragraph. I didn't bother reading to the end of the article. I'm glad I no longer pay for a TV license.
- dotBen 5y ago
  The BBC (News's) tech section isn't aimed at you. Inaccuracies shouldn't be there but often they will dumb down or gloss over stuff for the mainstream audience they are aiming at. You notice it cos you are in tech, but the same happens in financial news, science and even sport. Go read a tech publication. For shits and giggle I did once try to get a technical story on how to copy DVD's published - it got very heavily edited! http://news.bbc.co.uk/2/hi/science/nature/1987665.stm http://news.bbc.co.uk/2/hi/science/nature/1987665.stm (I'm a former + early BBC News website employee)
  - throwaways885 5y ago
    Aside: That old version of BBC News is an absolute gem of history. Especially looking at some of the recommended sidebar stories: > Britons 'baffled over euro rate' > Wireless internet arrives in China > Mobile spam on the rise Fascinating to see how much our problems have stayed the same, despite the changing context. I hope this is considered 'archived' and not 'forgotten'.
- jandrese 5y ago
  It has always been a constant of journalism that you read an article in your field and go "Wow, this is terrible, they got all of the details wrong". But then you turn around and trust the reporting on everything outside of your field of expertise.

brodouevencode 5y ago

How much of these overwrites are cover ups for failures? Certainly some percentage, just not sure what.