8 ms·
Is stuff online worth saving?
- pabs3 2y agoIf you're interested in that sort of thing, come hang out with ArchiveTeam: https://wiki.archiveteam.org/ https://wiki.archiveteam.org/
- underseacables 2y agoI suppose it comes down to what the purpose of such archiving is. I think it's the preservation of information, but I also believe 90% is absolutely pointless. There is just so much of it, and data storage so cheap, that it makes sense to just save everything.
- sigio 2y agoWell... storage is cheap, but not cheap enough to save everything, with just usenet being in the 400TB/day range these days. Sure, it's cheap enough to save every webpage you visit during your life, but probably not cheap enough to save every video you click on youtube or watch on a streaming-service, and all the music you listen to all day. Though just the music compressed in opus at 128kbit might work ok, 60 years of 24/7 128kbit is 30TB, so that would fit on 1 large HDD currently.
- saulpw 2y agoMusic is actually an ideal candidate. I don't listen to music all day, and when I do listen to it, it's often something I've listened to before. My current collection is about 200GB and that includes a ton of stuff I've never listened to; it seems reasonable that a full life's worth of music could fit in 1TB, easily.
- add-sub-mul-div 2y agoIf that much data comes across Usenet daily then how do services afford the storage to offer years of retention? You can't dedupe the large binary files because they're encoded in small parts likely differently every time they're posted.
- dreamcompiler 2y agoThat data storage is also ephemeral. Nobe of it will last as long as a paper note, unless some human goes to the trouble of copying it all onto new drives with new software every ten years or so.
- Atreiden 2y agoWith a proper NAS and RAID10 for double parity, it's a bit like Theseus ship. Just keep swapping out drives when they become unhealthy and you never have to rebuild or migrate
- ninalanyon 2y agoEventually the controller will die and eventually compatible ones will no longer be produced or will at least be inconvenient to obtain or commission and hence expensive. Paper lasts for centuries without any attention beyond keeping it moderately dry and away from things that eat it.
- emptiestplace 2y agoNo sane person uses hardware RAID in 2024, if that's what you're referring to.
- zamadatix 2y agoWhether you're using hardware RAID or not you still need a hardware storage controller of some type which accepts the new disks you can buy and works with the NAS. What they are saying is eventually that'll be more $ and time than just migrating off the system would be. From ENIAC to now could fit in one lifespan, would you still be maintaining a home floppy drive backup system in the 2040s or just save the time and effort with a migration?
- jpalawaga 2y agosure, you can always move the old storage mechanism to something new if it is too cumbersome. why still back up floppies if you could just move the data to a single dvd, or throw is on the SAN? RAID is just algorithms, the actual transport doesn't matter (i.e. spinning platter and solid state both use SATA connectors).
- danielbln 2y agoData rots though, you can't just save it once and be done with it. You have to migrate it across storage mediums, formats etc. It's a recurrent effort/cost.
- bdhcuidbebe 2y agoMore planning for less effort. Do your research first. Use standards Eg: html, pdf, h264/h265/av1 in mp4 container, chd, zip and so on depending on what you are storing.
- HeatrayEnjoyer 2y agoOn what physical medium? I have 1 terabyte of data in 1860, how do I make sure the storage medium is still intact in 2024?
- TacticalCoder 2y ago> I have 1 terabyte of data in 1860, how do I make sure the storage medium is still intact in 2024? Storage keeps growing and price of storage keeps doing down. My DOS and even some C64 source code made it to this day on backups (DVDs, HDDs, SSDs, USB memory sticks, etc., both online and offline) and to ZFS pools. Medium that didn't exist in the 80s/early 90s. Floppy disks -> 40 MB HDD -> 6.4 GB HDD -> 80 GB HDD -> 500 GB HDD -> 240 GB SSD -> 1 TB NVMe SSD. You get the idea. The way you get sure you still have your data is by not focusing on the medium but by focusing on the fact that data is data. Medium comes and goes. Data can (and should) be copied to new medium. Not unlike: /home/pub/backups/oldBackups/DOSbackups/... ...Conner80MBHDDbackups/backups/oldBackups/Commodore64backups/... Some people are going to complain about the naming but I have all my emails except for six months back since I started using the Internet. And I still have all nearly a lot of my data since I started using computers. 8-bit computers. Do you? I don't care about naming much. "search, don't sort". We've got emulators for just about every and any system. My vintage arcade cab has both real PCBs and a Pi running an emulator with thousands of arcade games on it. You can already, today, emulate, say, the Raspberry Pi model you want using QEMU. There are container file that'll gladly do that for you. Unless civilization ends there's simply a not a world in which, say, PNG, JPG and x265 files aren't readable. This just won't happen. FWIW I'm paranoid integrity of my data: I've got my own naming scheme where a cryptographic hash is added to many of my files. For example: DSC_91394-b3-ae4f2877d3.jpg This means "This file's Blake3 checksum begins with ae4f2877d3". I then have a script doing statistical sampling: I enter a percentage and that percentage of files where a cryptographic hash is part of the filename are checked, randomly (if I enter 100 then 100% of the files are tested). If I enter for example '7', then 7% of the files are tested and then there's high probability all checksums are correct. > On what physical medium? That is the wrong question.
- dleink 2y agoAt this point in history, we can't tell which 90% is absolutely pointless.
- JKCalhoun 2y agoOne hundred twenty-three years ago my great grandmother's first husband died in a hotel in Kansas City from asphyxiation from the gas having been left on over night (the hotel did not yet have electric lighting). A letter was hastily written on a piece of hotel stationary to be delivered to his wife in the neighboring farming community where she lived. It is fortunate to me that someone thought to hang on to that note since I have become interested in genealogy and this was a fairly significant event in family history (had he not died I don't suppose I would be around since it was her second marriage that gave me my grandfather). I long for scraps of anything that my dead relatives, wrote, created, etc. It connects me better to the past — the lives they lived, how they lived them. It somehow grounds me a little better ... well, it's rather hard to explain the draw of genealogy. Sadly very little of the ephemera of everyday life was kept. I get it. It might have seemed like hanging on to junk mail — like you were a hoarder or whatever, but in this digital era we should be able to hold terabytes of what may appear to be ephemera. I'm doing what I can – not for ego, I think, but for future generations that may find a connection to their past interesting.
- willis936 2y ago30 years ago there was no digital world. Nearly all information was in physical artifacts. The things worth saving haven't really changed, but the amount of noise they are buried in has. Imagine if that letter was kept in a two ton pile of ad fliers. Sure, someone would find some of those fliers interesting, but you'd have been much less likely to even know about the letter.
- qwertox 2y agoWhat about robots reading each flier and checking if something is odd about that particular one? It could find the letter and report it to you. Even easier if it was all digital information.
- bongodongobob 2y agoIf only we had search algorithms...
- 2y ago
- asimpletune 2y agoIt reminds me of the cool links page I see now and then.
- mxuribe 2y agoIs this the classic webpage that you're referring to? https://www.w3.org/Provider/Style/URI https://www.w3.org/Provider/Style/URI
- smitelli 2y ago> I got a picture of my great grandfather, thing took six hours to take your picture. [...] Every guy had one picture back then. And it's just him like, "[grimacing] I gotta get back, feed them hogs!" Now, in the future of course it'll be different. 50 years from now, people will be going like, "Hey! You wanna see a hundred thousand pictures of my great grandfather? I got 'em right here plus everything he did every day of his life." --Norm Macdonald[1] There is certainly a quantity of stuff online that is absolutely worth saving, but there's a considerably larger proportion that's just redundant to the point of being unremarkable and pointless. The trick is filtering, which can be capital-H Hard. That's why some may want to err on the side of over-collecting to reduce the possibility of missing something that will actually be important someday. [1]: https://www.youtube.com/watch?v=sY6SjMITHrQ https://www.youtube.com/watch?v=sY6SjMITHrQ
- nytesky 2y agoAnother funny take from Macfarlan Definitely no smiling: https://youtu.be/8SslNMLO0tw https://youtu.be/8SslNMLO0tw
- diggan 2y agoYeah, this is a good point. Isn't it better we save too much, as tooling for filtering stuff out will always get better, rather than saving too little? The latter has no workaround (today at least).
- nilamo 2y agoPersonally, I like that the internet is ephemeral. It matches real life in that way. I would rather see the internet as a means of connecting people over large distances (across space, Mars, etc), maintaining 20,000 copies of every irrelevant thing is just silly.
- qwertox 2y ago> Personally, I like that the internet is ephemeral. It is not. It is only for us normal people. But the companies which log our lives in order to then capitalize on it, for them the internet is not ephemeral. They have copies of videos, pages, podcasts, whatever it is what can be found there. Why would you want those companies to know more about yourself than you do?
- zamadatix 2y agoArchive.org or Google can cache more of the internet than I do while still having the majority of the content be ephemeral. I'd also hazard to guess most people in this camp would want these companies to also not store these things the same as they don't want people to.
- Barrin92 2y ago>Why would you want those companies to know more about yourself than you do? That's not a question of wants, companies will always know more about you than you, for the simple reason that even if you had all their data you have no means to extract any meaning from it. It requires immense organization and resources, increasingly so as the rate of data production increases. For that reason the correct response isn't to engage in the same hoarding and privacy abuse of the companies, it's like bringing a knife to a tank fight, but to 1. make sure you don't produce that data to begin with through privacy protections and technical means and 2. create environments in which you have ownership of your data, instead of businesses.
- lupire 2y agoMaybe I don't, but the solution for that is to destroy what they have, not torture myself by making myself store and know more about myself. I am not a company. As a human, my needs are different.
- paulpauper 2y agoDigital storage is free; yes, save it all
- lxgr 2y agoPlease do share where I can reliably store my backups for free!
- fragmede 2y ago> Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it. — Linus Torvalds
- LinuxBender 2y agoThis does still happen. Microsoft may nuke a git repo and someone has to figure out who has the latest version of the entire repo with all the latest commits of every branch.
- theandrewbailey 2y agoThe vast majority of people aren't privileged enough to have anyone mirror their data.
- lxgr 2y agoBut how do I get everyone to mirror my gigabytes of encrypted photo backups?
- paulpauper 2y agojust upload them to social media accounts. Afik twitter, facebook, and youtube do not have storage limits . no deletion for inactivity either.
- lxgr 2y agoThey don't allow uploading large binary blobs either, though, and steganographically storing gigabytes of data with probably terabytes of overhead sounds like a quick way to get banned.
- Viktoire 2y agoWhen I save things, I try to make sure that it'll be immediately useful to me once I find it again. I'll highlight, summarise and take notes of what I save. Or some combination of those. If I don't find anything new or directly applicable to my life, I'll let it pass by. This approach isn't good for archival purposes, but I hesitate to save a lot of things that I'll never read again.
- ghaff 2y agoI'm going through my file cabinets right now. I'll keep a few things that catch my eye but I'll likely throw out most of it. The odd 25 year old computer magazine is probably interesting but not all of them collectively for the most part. And I'm certainly not going to index them in a way that they'd be useful to me.
- galleywest200 2y agoYou can probably sell or donate those old magazines to a collector, or a kid interested in that stuff. At the very least drop them off at a thirft store instead of just dumping them.
- ghaff 2y agoThrift stores don't want a ton of old paper. There are a lot of things that someone somewhere would probably like but I'm not going to track them down or get them there. Mostly it's not magazines anywway. It's a bunch of articles I ripped out over the years. The one thing I have in my garage I know someone would want is a big pile of laserdiscs. But, again, a thrift shop (or my library) wouldn't want them and I live pretty far out from a major city. Probably will try Craigslist post-winter though as I'm trying to declutter.
- buildsjets 2y agoLaserdiscs appear and gradually disappear at my local thrift, so someone must be buying them. Now in the vinyl records pile, there are copies of Mantovani, Jim Nabors, and Herb Alpert which have been there for years, but anything classic rock or newer sells the same day.
- stared 2y agoI often find myself revisiting old posts and stories. As with any human artifacts, most things aren't worth revisiting or are only meaningful in the moment. If they're gone, few people miss them. I'm a link hoarder myself (over 13k links on Pinboard: https://pinboard.in/u:pmigdal/ https://pinboard.in/u:pmigdal/). While I don't revisit most of them, some have proven invaluable for re-reading and sharing. I'm not sure about the typical half-life of internet content, but a lot disappears—whether because people stop paying for domains, official websites get reorganized (or their content removed), or other reasons. This is where the Internet Archive steps in, doing the essential work of a digital librarian. I often share links from its Wayback Machine, which has been a link-saver more times than I can count.
- swayvil 2y agoCurve smoothing. Chaikin's algorithm and Jarek's tweak etc. Very clever and nice way of making angular geometry curvy. Constructive geometry stuff. There were like a dozen algs. I kept links to nice papers with diagrams. Then they started disappearing. Now I'd be pressed to find 2. This is really useful info that is apparently disappearing. So yes, it happens, and maybe you should save that stuff.
- RajT88 2y agoStuff online is absolutely worth saving. It is a window into the past - what people concerned themselves with, what they loved and hated. Scholars will write papers on this era, speculating what it was like and how it fit into what came after. The web documents the massive societal changes underway which do not relate to the internet directly. Things like changes in transportation technology, medicine, sexuality and gender, and how your average people felt about all of it. Scholars will data mine those opinions to understand who felt what ways and why, with the benefit of hindsight. New knowledge will come of it. So yeah! It is all worth saving.
- greatgib 2y agoSome times you have strange obsessions or a strange mindset related to your technological habits. And you might easily think that it is only you that is weird, not thinking straight. If you are the only one doing something, you are probably wrong. And then, hopefully, there are nice personal blog posts like this one, showing you that you are not alone having some peculiar habits and so that it might make sense even if most people don't even think about it. I have the exact same feeling when I discover through hn, blog posts and events that I'm not the only one having my web browsers full of tabs. Literally having thousand of tabs.
- thefaux 2y agoThere are many things in life that have immense personal value and zero value to nearly everyone else. This creates a lot of misunderstanding and incentive misalignment.
- ozim 2y agoSounds about same what I was going to write. Most likely it is not worth it. But people should not be doing only things that are “worth doing”. Then again if something brought you joy but was complete waste of time - it was worth it. Hate dementors who tell you otherwise, it is limited life time but it is yours. You should be helpful to others but doing only “what is worth” suck the beauty out of existence.
- zimpenfish 2y ago> zero value to nearly everyone else Well, except future historians who may find value in "personal" information (although I guess we've got such a surfeit of recorded "personal" information these days compared to even just 50 years ago, it may not be quite as useful as when they find, e.g., some Babylonian tablet with a shopping list on. But you never know!)
- impure 2y agoThe rise of LLM’s has really devalued saving stuff online. What is the point of saving an article if I could just ask ChatGPT to created it and would probably do a pretty good job? It’s still worth keeping notes and stuff that may be hard to find but the majority of things online can easily be reproduced and are not worth saving.
- vouaobrasil 2y agoI think you are right. But I think the answer goes deeper: we have encouraged a culture where the most supported information is also the most superficial. The essence of individual experience itself has long been discouraged on the web in favour of SEO and the trashy news and the trivial. So the fact that ChatGPT can replace much of the web actually says less about the marvel of ChatGPT and more about the lack of anything really worthwhile because the profound just happens to be the least economically valuable.
- Falkon1313 2y agoI just asked ChatGPT where I could find info about a specific tabletop game that was heavily discussed online some years back and had multiple fan sites with house rules, forum, mailing list, etc. It couldn't help me except to reference a few sites that no longer exist, one that's HTTP only so now causes browser warnings and was mostly only links to more sites that also no longer exist, and an old general gaming forum that doesn't even have a search function. It didn't mention the mailing list archive site which has 21 years of discussion, indexed and searchable, still available. Or the still-active site where some fans archived all that they could from the old sites some years back, along with emulators, binaries, and instructions to get some of the old fan-made software running again on modern systems. Neither of those sites are new since ChatGPT was trained, they have been around much longer than ChatGPT. But it knows nothing about them or their content. I then asked it what it could tell me about the topic of a blog that ran for 10 years, with 314 posts, most of which had around 100+ comments, and the site it most often linked to. ChatGPT's answer was simply: "It seems like I can’t do more browsing right now. Please try again later." So no, you can't "just ask ChatGPT." Contrary to popular belief, it doesn't know much about what is or was on the internet, even at the time it was trained, nor about many topics. Given the way the web has developed over time, it seems quite likely to have huge gaps on anything related to the small web, any niche hobbies or interests, etc. All that non-commercial stuff that you can't easily find in modern search engines, ChatGPT doesn't know about it either. Those big chunks of content that wink out of existence whenever a hosting company goes under or someone just stops paying the bill for a site that they used to love but haven't actively maintained in awhile? It doesn't know any of that either. Entire online communities rose, developed, created a great deal of stuff, then slowly atrophied, and eventually disappeared. ChatGPT knows nothing of them.
- renewiltord 2y agoIn general, I am pro-turnover where there is rivalry: ceteris paribus keep the newer thing. However, information is so cheap as to be effectively non-rivalrous so I am considering running my own archival and to keep kagi's small sites etc. alive. Unfortunately, there is not a good tool for this that matches whatever Archive.org has. ArchiveHub needs routine management to keep the feed up and viewing it is not that easy. I'm sure we'll come up with stuff. The other thing is that searching for the long tail is near impossible. The big sites dominate Google, so I need something like marginalia to actually get to the old stuff that it used to be so easy to find. Because of the median user having simple queries, some questions are no longer answerable on Google: they are dominated by the median user and never show up.
- willjp 2y agoThis resonates so strongly with me. I worked a job where I needed to use outdated Microsoft toolchains to build plugins for software, and the documentation was just -- gone. Good luck. I've been almost compulsively saving the things that feel important to me, while seldom browsing them for years -- all the while hunting for a faster and more intuitive recall system that lets me find them later. My ex, however had a much more fluid relationship with the internet and media in general. They liked new things, and didn't particularly care if they enjoyed something and it faded into obscurity. I feel like that's the winning mentality, but I just can't bring myself to embrace it.
- don-code 2y agoI DVR the nightly news with NextPVR, more as a convenience in case I'm doing something when it's on, want to pause/rewind, want to watch it the next morning instead, etc. Come 2020, I was convinced that the world was going to end. So I simply... turned off the retention rule. One hour of news is around 5GB, but that's a very-high-bitrate MPEG-2 stream with an extra audio channel in Spanish. So I instead wrote a cron job to take that week's news, drop the stuff I don't care about, and H.264 the entire set of them down to 4.7GB, then burn them to a DVD for offline storage, since there's not much value to keeping them online. By 2022, it was obvious the world was not, in fact, ending, but I never stopped this practice because of how simple it was, and how unobtrusive to store they are. I just make sure a fresh DVD is in the NAS every week, and put the DVDs on a spindle - they collectively take up about as much room as a toaster. I could make that even smaller and simpler if I opted for a portable hard drive. Occasionally I'll manually toss something interesting in, like the presidential debates, or special coverage of some newsworthy event. In 20 years, when it comes time to re-burn the earliest of them, maybe I'll make a value judgment on whether that's worth it, but for now it feels like I'd be losing something for not much of a good reason.
- accrual 2y agoReminded me of the story of Marion Marguerite Stokes who recorded TV news from 1977 to her passing in 2012. https://blog.archive.org/2013/11/22/a-dream-to-preserve-tv-news-on-the-road-to-realization-with-your-help/ https://blog.archive.org/2013/11/22/a-dream-to-preserve-tv-n...
- krick 2y agoHow do you backup websites? I mean, it sounds trivial, but I kinda still haven't figured out what is the way. I sometimes think that I'd like some script to automatically make a copy of every webpage I ever link in my notes (it really happens quite often that a blog I linked some years ago is no more), and maybe even replace links to that mirror of my own, but all websites I've actually backed up by now are either "old-web" that are trivial to mirror, or basically required some custom grabber to be writen by myself. If you just want to copy a webdpage, often it either has some broken CSS&JS, missing images, because it was "too shallow", or otherwise it is too deep and has a ton of tiny unnecessary files that are honestly just quite painful to keep on your filesystem as it grows. Add to that cloudaflare, Captchas, ads (that I don't see when browsing with ublock and ideally wouldn't want them in my mirrored sites as well), cookie warning splash-screens, all sorts of really simple (but still above wget's paygrade) anti-scraping measures, you get the idea. Is there something that "just works"?
- Dwedit 2y agoThere are extensions like "Save Page WE" that will dump the current state of the DOM to an HTML file, including CSS and Images, but these are static and don't make the scripting work.
- wis 2y agoFor saving a webpage you have open, I use a browser extension called SingleFile, I've been using it for a while (IIRC I discovered it on HN's front page a few years ago), in my experience it "just works", works really well. You click the "browser action" icon/button of the extension and it saves a single HTML file that looks exactly like the webpage you have open. From its FAQ[1] on GitHub: # What does SingleFile do? SingleFile is a browser extension designed to help users save web pages as complete, self-contained files. The extension's primary function is to capture an entire web page, including its HTML, CSS, JavaScript, images, and other resources, and package them into a single HTML file. # I am a web archivist, is it ok to use SingleFile to archive content? No, SingleFile is not a tool used by professionals to archive content on the Web, especially in the academic field. Professionals prefer to rely on tools based on the WARC specification instead. [1] https://github.com/gildas-lormeau/SingleFile/blob/master/faq.md https://github.com/gildas-lormeau/SingleFile/blob/master/faq...
- tofof 2y agoI've recently come back to a PC game (B-17 the Mighty Eighth) from 2000 that, quite unexpectedly, is getting a remaster and potentially a port to VR. It had a thriving community for several years, with many mods and guides and knowledge contained in the single dominant forum (bombs-away.net). When it shuttered, the vast majority of that information was lost. Old workarounds for bugs in the engine and detailed instructions of exactly how certain mechanics works are unavailable. One popular youtuber who continued playing through at least 2010 maintained a dropbox that had most of the mods that were ever available, but not the forum posts explaining them. So, for example, there's a mod that survives there to let you replace a generic 'sign on the dotted line' handwriting with your own - but gone are the instructions of exactly how to apply it. When I had returned to the game after bombs-away.net had gone defunct, I posted my own personal archive to the GoG forum for the game. Now that I've returned to the Redux version I find my own files, with my personal notes, shared by a single other soul who had similarly maintained an archive, and apparently had collected mine at some point. I'm very glad to have helped preserve knowledge - but not everything of mine was there. Now that I've noticed the 2024 remaster effort and joined that community, I've been able to share files that were otherwise apparently completely lost - in particular, a set of images showing dimensions of certain common features in bombing targets, that allow estimating the total size of the target. Unfortunately, my own personal archive included many forum topics that I just dragged off shortcuts to. I can see the old titles of the pages from the surviving shortcut files. I remember the questions I had (and now have again) that those shortcuts held the answers to. But because I didn't save the page itself, it's.. gone. That's immensely frustrating. Yes, things are worth saving. Especially for topics with extensive information among a small niche audience that have a single point of failure. I've found an extension (SingleFileZ) that does a good job of archiving a web page with all embedded content into what's a zip file under the hood - so futureproof even if the extension disappears and it becomes difficult to simply open the file directly in browser. EDIT - montebicyclelo mentions SingleFile, which apparently is a continuation of SingleFileZ, with new features. SingleFileZ already allowed automatically saving every visted page in a tab (or even among all tabs), batch archiving of a list of urls, etc, so presumably SingleFile has all these capabilities and more.
- btbuildem 2y agoI think some stuff is -- the stuff that is crucial to rebuilding all the other stuff.
- montebicyclelo 2y agoOne approach to this is the SingleFile browser plugin [1], configured to save pages to a GitHub repository - it saves the whole web page as a single HTML file in the repo. (Ok it's probably closer to archiving than bookmarking... but it's not too far off) [1] https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile
- wintermutestwin 2y agoThere is a wealth of live performances on youtube that individuals have uploaded and that likely violate mpaa copyright crap. IMO, this content is of high cultural value and I fear it won’t be long that the goog suffers us to watch “their” content without infecting it with ads. I wish there was an easier way to self host this content with a way to organize and browse using tags.
- wwweston 2y ago5 years ago I was working on a semi-novel crowdfunding platform that relied on video presentations. First iteration we used the YouTube API because hosting our own video seemed daunting and that worked fine for a bit. Over time we started to run into limits/errors/interruptions/audits at inconvenient times until one weekend I was like “screw it, let’s find out what the problems of self transcoding and hosting are.” Spent some time learning to use ffmpeg and throwing the results on our static resource pile. Tagging was a fairly straightforward lift. Honestly worked better than I’d have anticipated and was much less hassle. I’m sure we would have hit the problems if we’d reached a critical scaling point buuut that didn’t happen inside our year or so of having clients.
- zdc1 2y agoThese days whenever I read an interesting article, I will take 2 minutes to copy and paste it into my Obsidian vault under my Articles folder. I'll take care to paste the images as images (and not links) and make sure I've got the author and source URL at the top, and have my separate notes section link to it. It's a bit silly and obsessive, but given how transient content on the Internet is, I think it's necessary to make a copy of anything you care about.
- Feathercrown 2y agoAgreed. I think you could automate some of that too, could save time if you do it often.
- jeofken 2y agoIn my day browsers could save an archive of a page Is this still the case?
- Macha 2y agoThey can but generally that includes any Javascript on the same page which sometimes does funny stuff when you open it up offline or after the remote server goes away.
- compootr 2y agoSingleFile can make a snapshot with just content/styling
- accrual 2y agoIt's not perfect, but Edge will let one take a simple full page screenshot with Ctrl+Shift+S. It results in a hefty PNG but at least it's a visual copy of everything which might suffice for a certain set of purposes (e.g. links will be lost, so it's not good for that). I can still right-click > Save any page as .html, but that doesn't guarantee server streamed stuff, media, images, etc. will be preserved correctly.
- 2y ago
- jscottbee 2y agoI created a local-only web app to wrap up some of my favorite web haunts, with HN being one of them. It allows me to look at the headlines, and save any of them in a locale SQLite db that the app maintains. https://i.postimg.cc/v8znk92x/ycomb-hn.png https://i.postimg.cc/v8znk92x/ycomb-hn.png.
- iamwil 2y agoYes. Sometimes when I'm doing research into recent history of why certain technical decisions were made, and the arguments for or against, I find archive.org invaluable for piecing a line of thought back then. Recently, this was to look up what the debate between React's Functional components vs Signals was. Also, it's helpful to get perspective on the attitudes for or against a new technology in recent history. I remembered there were people that said "If you aren't writing a kernel, you don't have their problems, so you don't need git." Turns out that's not true. Now that git is everywhere, it's harder to remember whether or even if there was pushback against it. This was written about the insights from using git that he needed to highlight to people back then. https://keithp.com/blog/Repository_Formats_Matter/ https://keithp.com/blog/Repository_Formats_Matter/ I often reference it, and if it wasn't still up, I'd have only web archive to rely on. So for me, lots of stuff I look at online (mainly blog posts) are worth saving. Sometimes, if the discussion is on a twitter thread, that too. Which makes me fear for the day Microsoft decides to do Github in, and we'd lose all the issues and comments.
- deskr 2y ago"stuff online" is an exceptionally course filter to deem something worthy of saving.
- neilv 2y agoOne thing that is worth saving is the PDF manuals for physical products that you own. These sometimes disappear from the Web. Or disappear except for some third-party site that modifies and/or paywalls them. Also, save the occasional important support info Web pages for those products. You'll know it when you see it. And if you don't save it now, it might be gone when you need it. You don't need a fancy system for this. I just made a directory `~/doc/`, and started dropping files into it. Someday, I'll take the time to merge this with `~/wiki/`, but for now, I'm capturing the information with low friction, which is most important.
- Groxx 2y agoAnd even when they don't disappear, they still end up dozens of weird pages deep that none of the on-site help text or search points to correctly due to the various pointless redesigns the site has gone through. But hey, there's more whitespace now.
- jll29 2y agoAny information created by humans is part of our "culture". You may consider it of no value, but someone else may beg to differ. I went to a fantastic talk a few years ago at the British Library about digitizing a substantial quantity historic Australian newspapers. It was amazing to be able to read funeral announcements, product advertisements and other signals from the past showing us Australian culture from the 1800s. Since we leave much less behind in terms of physical assets (personal letters, postcards, personal diaries), we should at least aspire to archive more from the digital realm, or to future historians we'd look like a blank century.
- dredmorbius 2y agoYou raise an interesting question: when is recorded knowledge actually cultural? What of the zettabytes of data which today are written but never read? (The old saw of a WORN drive: write once, read never, has never been more apt.) What of a knowledge that is the provenance of a single individual? A recipe, poem, memory, craft, even languages (which are being lost at the rate of several per year: <https://en.wikipedia.org/wiki/List_of_languages_by_time_of_extinction https://en.wikipedia.org/wiki/List_of_languages_by_time_of_e...>)? Maths at the PhD level has been described to me by several people as research which can be understood by only a literal handful of people, a full five if you're extraordinarily lucky. Is that knowledge cultural? I've put some thought into what it takes for a specific skill, art, craft, or technology to be considered "alive". This presumes not only current practitioners, but a new generation who will learn, practice, and pass on that knowledge. Possibly additionally the cultural infrastructure (schools, businesses, markets, etc.) which are necessary to support, sustain, and reward the practice. What's the threshold of truly cultural knowledge?
- Macuyiko 2y agoFrom an age perspective (but the crowd here will not like that): before I trusted myself I could always find it back so I don't need to save it. Now I can't anymore, but I don't care so much.
- mediumsmart 2y agoI used to think so and then I ran out of space
- zelon88 2y agoI was thinking the other day about the longevity of useless data. One idea that floated around in my head was self expiring emails. I recently deleted about 40,000 emails. Most of them were identical, duplicate marketing emails. I was forced to do this to free up storage. That's when I realized something. I am paying my email provider for the full price for every byte of "represented" data. In reality, their distributed file systems could compress an arbitrary number of copies of these emails and only consume the amount of space that one email consumes. So 100,000 duplicate emails on the server are consolidated into one representation of the data, but each customer has to pay for each byte that is represented. The vendor stores a file once and charge full price every time they reproduce it for someone. If you have 10,000 copies of a file they only have to store it once but you will pay for every byte in all 10,000 copies.
- password4321 2y agoThis is the Dropbox business model, especially when they encourage using their service to share files and it counts as space used in source and destination accounts.
- Scoundreller 2y agoThere were some early blog posts by the single person running mailinator. Since they only stored text, they would make a single db entry for each unique line of text that came in and just made more and more references to that. Even different emails… were mostly the same.
- profsummergig 2y agoInstead of saving them as PDFs, I started saving web pages using a Chrome extension called Single File [1] (after testing it, of course). To my dismay, some saved files (.htm extension) didn't open when I wanted to open them. So I'm glad people are discussing ways to archive web pages while that reproduce the original page faithfully. [1] https://chromewebstore.google.com/detail/singlefile/mpiodijhokgodhhofbcjdecpffjipkle?hl=en https://chromewebstore.google.com/detail/singlefile/mpiodijh...
- paulcole 2y agoI’m the opposite of most of the “archivists” on HN. I delete everything and save nothing. I have maybe 25 sheets of paper in my apartment, including social security card and birth certificate. Saving stuff just isn’t fun or useful for me. Never for more than a passing moment have I thought, “Boy I wish I had saved that whatever.” Old people are the worst about this stuff. They think/hope somebody will want it and then just make it the next generation’s problem. I told my dad if he thinks it has value, give it away while he’s alive. I have neither the interest nor the space to deal with it so it’s going straight into the trash.
- dehrmann 2y agoThis feels like confirmation bias to me. The author seemed to genuinely consider the question, but didn't think critically about how little value he got from two decades of bookmarking and instead focused on how he could use this archive in the future.
- og2023 2y agoWe have become so cloud-native (god forbid!). Just recently I realised that I can save an interesting page to my hard drive instead of saving its link. What a wonderful word has opened since! It's so liberating to live without all these bs tools.
- deleted 2y ago[deleted]
- m463 2y agoI think people don't get another perspective on this until someone dies. Mostly it just goes away at death. It might be interesting to read: https://en.wikipedia.org/wiki/Digital_hoarding https://en.wikipedia.org/wiki/Digital_hoarding I have trouble letting go of things, and I found it interesting to read through. There's a part of me that thinks "It would be so useful to 100% automatically log and cache everything I do and be able to search it". But I think maybe being healthly means not doing it.
- EasyMark 2y agoI mean one man's trash is another man's treasure. We can't save it all but if we save even 10% of it, that would be great for future AIs to learn from our mistakes and what humans once did on a day to day basis before they were replaced.
- 8bitsrule 2y agoI started collecting thousands of URLs (as resources for notes) about 16 years ago. In use, I'd estimate that link-rot has affected about 1 or 2 out of four in that time. Out of those two fails, I expect I'd recover about half by feeding the URL to Wayback and asking for their oldest save. That tends to be independent of age. Most successes are from reliable sources.[1] When those sources fail, it's usually because the resource is still there (in identical or revised form) but has a modified URL. They're often recoverable by searching by 'site:www.abcd.efg ' and a string of terms from the original. Each fail requires that I consider whether the time to recover is worth it. I also imagine how difficult it'd be to have saved each resource, and have to search for it. Ideally each resource would have an ID and permanent online home; unlikely. [1] Each fail continues to teach me which specific resources are most reliable over the long term. Short answer:References and WayBack.
- dredmorbius 2y agoIs stuff online worth saving? This supposes a broader question: what is the purpose of recording information generally? The alternatives are observing the here and now (boys, obIsland), conversing, or musing / meditating / cogitating on topics. I'm not hostile to recorded information of many forms, and am in fact something of a packrat myself. But as I acquire archives and observe the increasing enshittification of the Internet as a whole, the value of investing time in searching and saving information only found online and created for online consumption seems ... increasingly questionable. As I've mentioned in numerous earlier HN comments,[1] my preferred information forms are increasingly more traditionally-published books and articles, highly prepared discussions, or conversations amongst experts within a field (or with a strongly discriminating host and an expert). It's worth noting that the origin of much (though of course not all) online content is in fact discussions based around such works: author interviews or exerpts from books, discussion of articles, etc. Another principle, and one that seems worth considering in the context of online content, is that false leads and faulty assumptions are absolutely toxic to learning or training in a domain. This turns up in AI contexts, but is also evident to, say, a survey of the history of philosophy, particularly the roughly 2,000 year period in Western philosophy of the dominance of Christian theological philosophy, much of which was based on utterly misguided premises. It wasn't until such assumptions were dropped that scientific and technological understanding really began advancing.[2] One wonders to what extent our present accumulated document[3] trove might actually weigh down future progress. Or succinctly: What's the balance between retention and study, on the one hand, and observation and reason on the other? Tradition vs. empiricism. -------------------------------- Notes: 1. Searching my handle with "published books articles" turns up a few dozen of these: <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adredmorbius%20published%20books%20articles&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...> 2. I'm not writing this as an utter dismissal of all theological thought, nor am I asserting that what I'm terming "misguided" was utterly useless. There are useful ideas which emerge, there are religiously-aligned philosophers who arrived at keen insights, the process of working through arguments, even if founded on utterly counterfactual premises, can lead to useful developments in reasoning and logic, etc., etc. That's even without getting to "it can always serve as a bad example". But in this and many other areas it becomes clear that relying on early authorities (in the classical sense of that term) heavily retarded intellectual advances. 3. In the Otletian sense of any type of recorded work: books, articles, images, plastic sculpture, video, etc.
- cchi_co 2y ago[dead]