3 ms·

My current experiment in fighting linkrot is pre-emptive local archiving, hosting static snapshots of links, serialized using SingleFile: https://www.reddit.com

by gwern 6y ago

My current experiment in fighting linkrot is pre-emptive local archiving, hosting static snapshots of links, serialized using SingleFile: https://www.reddit.com/r/gwern/comments/f5y8xl/new_gwernnet_feature_local_link_archivesmirrors/ https://www.reddit.com/r/gwern/comments/f5y8xl/new_gwernnet_... I am still finetuning it: there are a lot of domains which should be whitelisted because archiving doesn't work or isn't useful, and sometimes pages break when SingleFile'd (the content is there, but the CSS or JS is broken, and I muck around manually trying to fix them). The resource requirements are not that bad so far (21GB) but are probably a deal-breaker for anyone who wants the usual personal-static-site budget strategy of hosting on Amazon S3 for <$5/monthly, as even assuming way fewer links than gwern.net, cloud bandwidth is so expensive they'll quickly blow their budget. It also makes server logs messy as various bots try to fetch broken relative links, which were of resources which didn't get inlined by SingleFile (not sure what exactly triggers those). However, so far it seems like a viable strategy.

gildas 6y ago
Author of SingleFile here, Thanks for your feedback! Feel free to contact me or create posts on Github about these bugs, I will do my best to fix them.
- gwern 6y ago
  I haven't mentioned it because logs are full of garbage from crawlers even without any bugs (there's a shocking amount of malformed, malicious, or just crazy bugs - I think there must be crawlers which use n-grams or something to predict possible URLs to speculatively fetch), and I haven't tracked down whether it's anything to do with SingleFile or just the crazy old Internet being crazy. At some point I'll have to dig into it more. It's probably something simple like "lots of websites encode addresses with absolute paths, so bots follow those; SingleFile ought to rewrite absolute paths to point to the original domain."
  - gildas 6y ago
    Thanks again for your comments. I also own a SaaS related to SEO and I can confirm what you see. Some bots request URLs that don't exist by trying to guess them. One the other hand, I also often see errors in links where the HTML contains for example <a href="www.example.com/another-page"> and is resolved as "https://www.example.com/www.example.com/another-page" https://www.example.com/www.example.com/another-page". FYI, note that SingleFile is supposed to resolve all the links and that this should be reliable since it delegates this resolution to the browser.
    - gildas 6y ago
      I'm also realizing your issue could be related to some <noscript> tags which are not processed by SingleFile and could contain relative URLs. I should maybe add an option to remove them.

by gwern 6y ago

gildas 6y ago

Author of SingleFile here, Thanks for your feedback! Feel free to contact me or create posts on Github about these bugs, I will do my best to fix them.

gwern 6y ago
I haven't mentioned it because logs are full of garbage from crawlers even without any bugs (there's a shocking amount of malformed, malicious, or just crazy bugs - I think there must be crawlers which use n-grams or something to predict possible URLs to speculatively fetch), and I haven't tracked down whether it's anything to do with SingleFile or just the crazy old Internet being crazy. At some point I'll have to dig into it more. It's probably something simple like "lots of websites encode addresses with absolute paths, so bots follow those; SingleFile ought to rewrite absolute paths to point to the original domain."
- gildas 6y ago
  Thanks again for your comments. I also own a SaaS related to SEO and I can confirm what you see. Some bots request URLs that don't exist by trying to guess them. One the other hand, I also often see errors in links where the HTML contains for example <a href="www.example.com/another-page"> and is resolved as "https://www.example.com/www.example.com/another-page" https://www.example.com/www.example.com/another-page". FYI, note that SingleFile is supposed to resolve all the links and that this should be reliable since it delegates this resolution to the browser.
  - gildas 6y ago
    I'm also realizing your issue could be related to some <noscript> tags which are not processed by SingleFile and could contain relative URLs. I should maybe add an option to remove them.