4 ms·
In my visualization of perfect blogging platform, it automatically periodically scrapes all your old links and looks for either redirections or wild size change
by jerf 11y ago
In my visualization of perfect blogging platform, it automatically periodically scrapes all your old links and looks for either redirections or wild size changes in the content. It also auto-archives linked content for you (which used to be easier, but can still be at least tried), and could offer these archives to either the blog owner or post reader. But among the many challenges implementing that provides is that it is hard to make it work at scale, because of the resulting legal issues.
- sohcahtoa 11y agoSeems that would be better built as a stand-alone spider than a bolt-in to a single blogging platform.
- showerst 11y agoIt looks like there are some wordpress plugins that let you check and at least fix the links, but I don't see anything that takes the extra step of archiving. https://wordpress.org/plugins/broken-link-checker/ https://wordpress.org/plugins/broken-link-checker/
- dandelany 11y agoYes, this would be awesome. Even just auto-detecting broken links and redirecting them to archive.org versions instead seems like it would be an easy win that would sidestep the legal problems of archiving them yourself.
- mirimir 11y agoYes, archive.org is an amazing resource. But I'm wondering about the coverage, especially for dynamic and deep content. Maybe those are mainly issues on older snapshots. Also, copyright issues aside, the prospect of a recursively archived Web is a little mind boggling. But storage is always cheaper, so hey.
- toomuchtodo 11y agoOr the links are turned into archive.org links automatically at publish time? With any pages not captured immediately captured at that point? EDIT: This is a hack until the content addressable web arrives.
- isoos 11y agoI've recently started to consider tools like wkhtmltopdf [1] to easily dump a website I'd like to link to. Then I'd include both the original link and the pdf version on the blog page, in order to keep the linked structure and the archived version at the same spot. I haven't done it yet, but would be interested in other's experiences. It would be interesting to have a crawler that does it automatically for each new post. [1] http://wkhtmltopdf.org/ http://wkhtmltopdf.org/
- mirimir 11y agoI tend to keep local PDFs of everything that seems crucial. But linking to them seems ethically questionable. But maybe as a last resort, with an explanatory comment.
- dangrossman 11y agoThis would be problematic legally, even when what you're linking to offers a permissive license of some sort.