4 ms·
I've had some success in the past using a WARC proxy - it will basically record everything that traverses the browser and can "play it back" on demand. So whil
by tenebrisalietum 5y ago
I've had some success in the past using a WARC proxy - it will basically record everything that traverses the browser and can "play it back" on demand. So while it won't automatically download everything on a site the idea is that whatever you visit with and interact with in a session can be "played back" in the future at some point.
- genewitch 5y agoI actually built a machine to do this, and wanted to use l my "old" spindle drives as storage, but I was and am unhappy with the offerings of replicated or raid like storage solutions. I keep waiting for ceph or something to have a better NFS layer, but I might just have to have scripts that balance/replicate the warc however many times based on usage. For those interested I have 43TB, 210TB, 61-2TB, 212TB. The machine boots and sees all the spindles. ZFS would be ideal but cannot handle differing drive sizes I guess. JBOD misses the replication requirement. Just doing mdraid on all the similar drives and having different folders or JBOD "wastes" too many spindles (at least one per group, so 4-6 wasted spindles by size alone!). So I'll probably handroll something with whatever that triggered rsync program is, or cronjobs. lsyncd, that's the one.