3 ms·
I've been thinking recently it might be interesting/useful to write a simple SOCKS proxy which could be used by my browser. The SOCKS proxy would not just fetc
by stevekemp 5y ago
I've been thinking recently it might be interesting/useful to write a simple SOCKS proxy which could be used by my browser.
The SOCKS proxy would not just fetch the content of the page(s) requested, but would also dump them to ~/Archive/$year/$month/$day/$domain/$id.html.
Of course I'd only want to archive text/plain and text/html, but it seems like it should be a simple thing to write and might be useful. Searching would be a simple matter of grep..
- habibur 5y agoDid that. But then you will find your disk quickly getting filled up with GBs of cached contents that you rarely search within. Rather when you need that same content, you will find yourself going to google, searching that and the page is instantly there unless removed. There's a reason why bookmarks aren't as popular as it had been. People now use google + keywords instead of bookmarks.
- berkes 5y agoIt would also miss all the pages that are built from ajax-requests on the client side. Which, nowadays, is a large amount. The client is the one assembling all the content into the thing you read and so it is the most likely candidate to offer the copy that you want indexed.
- kbenson 5y agoMaybe archive.org should run a subscription service where for a few bucks a month, you can request your page visits be archived (in a timely manner and with some level of assurance) and leverage their system for tracking content over time. That, in conjunction with something like Google, might actually give fairly good assurance that what you're searching for actually exists in a state like you saw it, while also leveraging that 30 people accessing this blog today that use the service don't use significantly more resources to store the data, and also helps archive.org fulfill its mission.