4 ms·
That’s a lot of work.
by lowbloodsugar 2y ago
That’s a lot of work.
- demaga 2y agoYes, exactly. As a data hoarder myself I've been thinking 'what data is _really_ important to me?'. And the answer is - not that much of it. The work, mental space, time, money you have to invest into storing your own data is so much effort, it is probably not worth it.
- overfeed 2y agoI don't consider myself to be a proper data hoarder since I only have tens of TiB at my disposal, but I managed to minimize the work, mental space, and time aspects by automating as much as possible. At first, I had a bunch of scripts running on a raspberry pi, but now I have the entire process managed by Home Assistant. its a rube goldberg machine involving mqtt, VMs, cheap VPSs, rsync, and wifi plugs, but it works. I only get notified if a daily backup fails, and always get full summary of the weeklies. I probably could automate the process of writing to DVD, so the only manual thing I'd need to do would be to insert blank disks, but my quality of life has drastically improved. As for cost, I'm still working on it, and my storage needs are very predictable so I can hunt for deals ahead of time - I still have unused HDDS from last year. It is common to find discounts to about $12/TiB, which is cheap.
- sockaddr 2y agoI'm sure what you rigged up is beautiful, but this sounds stressful haha
- overfeed 2y agoIt helps that I enjoy coding, and that I deeply care about the data I'm preserving. I got burned after losing 2-years worth of unbacked-up data scraped off Twitter right before they closed the API, so I'm never ever going to get that data back again. The backup automation evolved pretty organically, but slowly. I was happy when I finally was able to get the weekly backup process to start automatically.
- sockaddr 2y agoI used to save everything by default and over the years (~20) my storage requirements started getting out of hand. So I had a change in philosophy where I decided to throw everything into a "to delete" folder and start with a single flat folder structure and go through everything file by file and put it in the "keep folder" and really evaluate whether I needed it. As a result I ended up with about a 90% reduction and I don't feel like I'm missing anything.
- pdimitar 2y agoYeah, this. The data I am most concerned about is not even 1GB after compression. That's all my $HOME configs and all my projects I am working on. Then I have some open datasets I like to fiddle with (mostly *.sql.zst compressed DB dumps) which I periodically dump on my Linux server (weekly with rsync) and finally -- video. Video is obviously like 99.99% of everything but I have made sure to store all the sources of it (mostly downloaded playlists from YouTube) and I have scripts that synchronize the videos from the net to my local folders. Even tested that a few times and it worked pretty well. So indeed, in the end, just find what's most valuable and archive that properly. In my case I have one copy in my server and 5+ copies on various cloud storage free tiers. All encrypted and compressed. Tested that setup several times as well, I have a one-line script to restore my dev $HOME, and it works beautifully.
- dharmab 2y agoThe local external backups certainly are. S3 Deep Archive is about one evening to set up rclone and set up a regular job to run a backup. If you don't need Linux support you can also pay Backblaze a flat fee for an easy solution, at a slightly higher price.
- Tostino 2y agoI have my whole unraid server backed up using Backblaze's windows backup tool / subscription. There is a docker container for running the backup app, so it can be setup on a standard Linux desktop as well I believe. You just need to mount the storage you want backed up into the container as another drive and then you can configure the backup software in the fake window environment to back up those drives on "your PC" in the container.