11 ms·
To preserve their work journalists take archiving into their own hands
- ghaff 2y agoThis has been true for a long time. Had I not archived a fair bit of my own work, some of it in the CMSs of dead organizations, some of it inaccessible behind paywalls, much would no longer exist. Journalists are probably in better shape than many because they're more likely to have work they've created on a relatively open web.
- wkat4242 2y agoThey should really collaborate with archive.org. They won't shut things down or paywall it.
- shitlord 2y agoArchive.org can and does accept takedown requests, even if the requester wants to avoid public scrutiny. If you're writing about a contentious topic and want to preserve your links (tweets or whatever), there are better options.
- sinkasapa 2y agoWhat are some of those options?
- shitlord 2y agoNot all of these are 1:1 replacements but here are a few options: archive.is/archive.today, ghost archive, save a webpage as html with assets, and taking a screenshot
- tamimio 2y agoThere’s an addon called single file, you can use it to save the whole page with assets (except videos) https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile
- dotancohen 2y agoIs there a to that I can script, to use the cookies on my existing web browser (because I'm logged into some websites) and get the page content text? All while clicking away pop-up banners (newsletter, cookies, etc)? I would script that to go over my local bookmarks file.
- msephton 2y agoReader mode does this in Safari. Adding to Reading List also stores the content for offline reading. I bookmark to a service (Linkding, self-hosted) that automatically sends the URL to be archived at Wayback Machine.
- dotancohen 2y agoThank you. I should have been clearer and mentioned that I'm on Debian )) But in any case, Safari's reader mode can be scripted from the CLI? That is good to know, maybe I'll try to find something similar for Firefox's Reader Mode. Thank you.
- msephton 2y agoYes. On macOS we would generally use osascript on the command line. Apple Script Editor shows which functions each app exposes. Very powerful combo.
- dotancohen 2y agoThat's great to know, I consider an Apple device every few years when I upgrade my current devices. Osascript sounds compelling.
- msephton 2y agoSafari has this, it can save as a Web Archive which is a sort of zip containing the html and assets.
- nikisweeting 2y ago- https://archivebox.io https://archivebox.io - https://archiveweb.page https://archiveweb.page + https://replayweb.page https://replayweb.page / Browsertrix - SingleFile
- ghaff 2y agoThere really aren't. You can publish a book I guess but that has limited reach. And your website isn't going to be around forever.
- msephton 2y agoExcept when https://news.ycombinator.com/item?id=41135450 https://news.ycombinator.com/item?id=41135450
- aspenmayer 2y ago[flagged]
- metadat 2y agoNo discussion == no dupe.
- walterbell 2y ago> “Thank goodness she did that because [otherwise] we would have no records of the early years of the first Women’s Hockey League in Canada,” Azzi said. A few years ago, Canada digitized many older television shows, https://news.ycombinator.com/item?id=35716982 https://news.ycombinator.com/item?id=35716982 With the help of many industry partners, the [Canada Media Fund] CMF team unearthed Canadian gems buried in analog catalogues. Once discovered, we worked to secure permissions and required rights and collaborate with third parties to digitize the works, including an invaluable partnership with Deluxe Canada that covered 40 per cent of the digitization costs. The new, high-quality digital masters were made available to the rights holders and released to the public on the Encore+ YouTube channel in English and French. In late 2022, the channel deleted the entire Youtube Encore archive of Canadian television, with two weeks notice. A few months later, half of the archive resurfaced on https://archive.org/search?query=creator%3A%22Encore%20%2B%22 https://archive.org/search?query=creator%3A%22Encore%20%2B%2.... If anyone independently archived the missing Encore videos from Youtube, please mirror them to Archive.org.
- Bluestein 2y agoArchive.org is such a godsend.- The entire information 'substrate' of society is ephemeral, if digital, and none (at least not enough) seem to have noticed.-
- InsideOutSanta 2y agoI wrote a book in 2010. It had a references section with links to about 100 websites. When I wrote the second edition only about five years later, 50% of those links no longer worked. What we're doing right now is borderline insane. We're putting all of this information on the web, but almost each individual bit of information is dependent on either a company or a human being keeping it online. It's inevitable that companies change their minds, and humans die, so almost all of the information that is online right now will just disappear in the next 80 years. And we essentially only have one single entity that tries to retain that information.
- 2y ago
- ThinkBeat 2y agoI fully support the efforts. but are there not legal problems with this? (No I dont thik legal issues should prevent this) If I worked for CorporateMediaNews as a columnist and reporter for 10 years and they decide ot remove all of it. Does not CMN own the work and can (unfortunately) dispose of it if they so wish? I would not have any rights for the work? Thinking about my own career. I have written a hell of a lot of code and 80% at least are closed source system for various companies. I dont retain any copies of that code. It would be interesting if I heard that System X I wrote 15 years ago is being shut down, and I would try to obtain the source code in order to preserve it. I have never heard of anyone doing it, but probably in games and such it happens more often.
- ghaff 2y agoIt's often probably at least a bit complicated. I cross-posted material between a couple of organizations (one of which is long gone) over a number of years via pretty much informal agreement. I also reused a fair bit of that material for other purposes. Everyone was OK with the state of affairs but who actually held the copyright? Who knows and I was certainly never going to bring it to a head. As a practical matter if CorporateMediaNews or the like don't care about something any longer, they mostly don't care if someone else makes use of it so long as it isn't embarrassing or misrepresenting the organization. In the end, I have created work that I've reused for a variety of organizations as well as independently and someone other than myself probably would claim copyright to but it's often been pretty loose.
- ghaff 2y agoRandom old source code is probably pretty hard to do something useful with. But, honestly, random old work product is pretty hard for just about anyone to do something useful with. And even something self-contained (like writing), most of the the stuff I wrote 10 or 20 years ago is pretty uninteresting even if a few things have some historical interest.
- jjmarr 2y agoJournalists are often judged based on their public output. You get bylines on published articles, and you can use those to apply for jobs. There's not much emphasis on secrecy because the goal of a news outlet is to be read by as many people as possible. Being unable to provide examples of your work is a career-killer if everyone else can point to stuff they wrote. Preserving your work isn't altruistic, it's required. It's a very different environment compared to programming, because oftentimes the source code is valuable because it's a secret. Employees taking source code is dangerous because competitors can copy it; obviously that's bad for one's career. The analogy doesn't work here because the cost/benefits of taking legal action are different.
- whartung 2y agoMy understanding is that some photographers are archiving their digital pictures by basically printing them using the 4 color process, which gives them the 4 "negatives" ("postives?" or whatever they're called) for each color (CMYK I guess). Those sheets are archival quality and should last for quite sometime, given a proper storage environment. They can always use those later to have them scanned back in should they lose their master digital files.
- WalterBright 2y agoWith cheap 20+Tb drives available, just buy a new drive every couple of years and copy the files forward.
- gerdesj 2y agoJournos discover backups! Yay.
- 8bitsrule 2y agoToo bad there's not something like DOI or ARK [0] available for anyone to use to give documents a searchable, permanent ID so that a location can be maintained. IME, the half-life of many URLs (5-10 years?) makes them unreliable. I recently was unable to find (by URL) an entire historical collection at a major southern US university until I discovered that it had been moved to a new server. [0] https://arks.org/about/ https://arks.org/about/
- Bluestein 2y ago> the half-life of many URLs (5-10 years?) makes them unreliable. "Simple" enough experiment, on this very site: Use the "past" feature on the main menu to go back a step at a time, and tally the number of broken links from the external submissions the further you go back.- The amount of dead projects, expired domains, broken links, 404s, etc. is sad.-
- remram 2y agoSomeone did that and found that only 5% of submitted URLs (200k/4M) were dead: https://blog.wilsonl.in/hackerverse/ https://blog.wilsonl.in/hackerverse/ Submitted 3 months ago: https://news.ycombinator.com/item?id=40307519 https://news.ycombinator.com/item?id=40307519
- Bluestein 2y agoThanks for the data. That's actually not bad. Then again, 5% of a "webscale" number of sites is still a lot.- Thanks for pointing to that study ...
- tamimio 2y agoGood thing I’m a hoarder. If I like something, I archive it and back it up locally. For example, a couple of days ago, I needed some digital assets for an Adobe program that I had downloaded a few months ago because I liked them and thought I might need them in the future. When I went back to the company page a couple days ago, everything had vanished! I'm glad I had downloaded them before and checked my backup to retrieve them.
- darby_nine 2y agoDo you engage in any kind of physical archiving? I'm not going so far as to say it's superior to digital archiving but it does simplify maintenance of the assets in the event that you're incapable of it.
- tamimio 2y agoOnly few pictures I have them printed, the rest is all digital for the most part, unless you mean tapes/DVD/etc as a physical archive, then yeah I have that too.
- nashashmi 2y agoMany many years ago I read a Fred Wilson (avc.com) blog post about a founder who vented that a tech journalist's article on the founder's company was a hit piece. The tech journalist was recommended by Fred Wilson who was an investor in the founder's company. Fred wrote the tech journalist rarely ever does their own independent research. The article's position must have come from the founder himself. I can't find that article. And I have looked hard for it. I think now it must have been taken down. And that is sad. It had a valuable lesson that I want to share it with others these days.
- HPsquared 2y agoA lot of the most interesting things are contentious and likely to be deleted. It's almost a law of the internet.
- bzmrgonz 2y agoSomeone should create an immutable article hub repository. They can call it pubark.
- vasco 2y agoA nice social attack is to create an internet archive looking website call it archive.newtld and use it to create social proof of things you didn't actually do. "Oh yeah the Washington Post did a redesign but here are my past 10 posts which I saved in archive: link" In post truth internet, proving archives is going to be tough and unless there's some other form of verification it's going to be useless fast for "proving" purposes. You can think bigger and do this to forge stories about anything you want in any website. Nobody checks authenticity of archive urls and there's several sites already, plus a lot of these services do URL re-writing, so it's hard unless there's some authorative thing.
- HPsquared 2y agoCould this be solved by digital signatures on web content? (Or, a way to store those)
- throwaway48476 2y agoA content addressed network, yes.
- msephton 2y agohttps://en.wikipedia.org/wiki/Content-addressable_network https://en.wikipedia.org/wiki/Content-addressable_network
- vintermann 2y agoYes, pretty easily. In general actively faking old data, and especially faking it to convince the public is not what should be our main concern. Any given archive site can try that once, with high risk of quickly being caught.
- remram 2y agoHow do you obtain the key to check the signature?
- 2y ago
- vertis 2y agoI'm not a particularly good writer, but I've written about how I use the SingleFile extension to capture a perma web version of everything interesting that I read[0]. It's a great open source tool that aids in archiving (even if only at the personal level). I've been taking notes and blogging since the early 2000s and coming back so often to find the content that I'd linked to has disappeared. Archive.org and Archive Team do amazing work, but it's a mistake to put all your archiving eggs in one basket. [0]: https://vertis.io/2024/01/26/how-singlefile-transformed-my-obsidian-workflow/ https://vertis.io/2024/01/26/how-singlefile-transformed-my-o...
- coderatlarge 2y agoHow do you feel about this vs printing a pdf of the content?
- vertis 2y agoI think both work, from a purely information point of view. The SingleFile download preserves more of the original format. For a long while I was using MarkDownload and capturing the content that way, but a bunch is lost that way. I also use Zotero for downloading journal articles (etc), that also has the ability to snapshot, but then I found it was locked up in Zotero. Where my current setup is a Jekyll repo on Vercel that means that the content is almost immediately accessible after the github push and deploy. Something that happens automatically after I click the SingleFile download button (configured in the extension). I need do no more than grab the web link and paste it into Obsidian, where linking to Zotero from Obsidian is a royal pain (not impossible).
- Triphibian 2y agoOne of the great ironies of this situation is many of the now defunct websites had contracts and writing agreements that were absolutely egregious. Often the boilerplate would say that they owned the article (which they paid you a pittance for) until the end of all time. Prior, in the print era, the standard agreement was they'd have the rights to your story upon publication then after a reasonable amount of time the rights would revert to the author.
- kelsey98765431 2y agoEver since the NYT legal case against OpenAI (pronounce: ClosedASI, not FossAGI; free as in your data for them, not free as in beer) there seems to be an underground current pulling into a riptide of closed information access on the web. Humorously enough the zimmit project has been quietly updating the living heck out of itself and awakening from a nearly 6-8 year slumber. The once simple format for making a mediawiki offline archive now is able to mirror any website complete with content such as video, pdf, or other files. It feels a lot like the end of usenet or geocities, but this time without the incentive for the archivists to share their collections as openly. I am certain full scrapes of reddit and twitter exist, even with post API closure changes, but we will likely never see these leave large AI companies internal data holdings. I have taken it upon myself to begin using the updated zimmit docker container to start archiving swaths of the 'useful web', meaning not just high quality language tokens, but high quality citations and knowledge built with sources that are not just links to other places online. I started saving all my starred github repos into a folder and it came out just around 125gb of code. I am terrified that in the very short term future a lot of this content will either become paywalled or the financial incentives of hosting large information repositories will increase past the point of current ad revenue based models as more powerful larger scraping operations seek to fill their petabytes while i try to prevent my few small TB of content i dont want to lose from slipping through my fingers. If anyone actually cares deeply about content preservation, go and buy yourself a few 10+ TB external disks and grab a copy of zimmit and start pulling stuff. Put it on archive.org and tag it. So far the only zim files I see on archive.org are the ones publicly released by the kiwix team yet there is an entire wiki of wikis called wikiindex that remains almost completely unscraped. Fandom and Wikia are gigantic repositories of information and I fear they will close themselves up sooner than later, while many of the smaller info stores we have all come to take for granted as being "at our fingertips" will slowly slip away. I first noticed the deep web deepening when things I used to be able to find on google were no longer showing up no matter how well I knew the content I was searching for, no matter the complex dorking i attempted using operators in the search bar, just like it had vanished. For a time bing was excellent at finding these "scrubbed" sites. Then duckduckgo entered the chat, and bing started to close itself down more. Bing was just a scrape of google, and google stopped being reliable, so downstream "search indexers" just became micro googles that were slightly out of date with slightly worse search accuracy, but those ghost pages were now being "anti-propagated" into these downstream indexers. Yandex became and is still my preferred search engine when I actually need to find something online, especially when using operators to narrow wide pools. I have found some rough edges with zimmit and I am planning on investigating and even submitting some PR upstream, but when an archive attempt takes 3 days to run before crashing and then wiping out the progress it has been hard to debug without the FOMO hitting that I should be spending the time getting what I can now before coming back to work on the code and get everything properly. If any have the time to commit to the project and help make it more stable, perhaps work on some more fault recovery or failure continuation it would make archivists like me who are strapped for time very very happy. Please go and make a dent in this, news is not the only part of the web i feel could be lost forever if we do not act to preserve it. In 5 years time I see generic web searches being considered a legacy software and eventually decommissioned in favor of AI native conversational search (blow my brains out). I know for a fact all AI companies are doing massive data collection and structuring for graphrag style operations, my fear is that when its working well enough search will just vanish until a group of hobbyists make it available to us again.
- JohnFen 2y agoThis is the only way, in my opinion. Not just for journalists, but for all professions. If you haven't archived it yourself, on machines and/or media that you are in possession of, then you can't rely on it to continue to persist.
- jfil 2y agoIt's great to see more non-programmers realize how ephemeral Web content is, and taken bare-bones archiving efforts. If you or someone you know are looking to archive content from the Web, but don't know how, I'll be happy to help. My email is in my profile.