3 ms·

Yet another proof that having `wget` to download websites, `git` to control for versions (what changed since last pull) (or restic/borg to do more of less the s

Yet another proof that having `wget` to download websites, `git` to control for versions (what changed since last pull) (or restic/borg to do more of less the same) and/or host them somewhere is (a) extremely easy and (b) somehow mind-blowing to a lot of people who call themselves "technical" or even "programmers". Weird. I get what not having time and energy is, I truly do, but if you cared so much about this community and its history, surely SOMEBODY will find the time AT LEAST ONCE and archive these precious bits of history? I have a very and I really mean VERY benevolent spider bot -- we're talking from 2s all the way to 10s waiting between each link visit -- that crawls sites I care about weekly and does exactly that. The only thing I am worried about is if I can rely on my GIT hosting but I'll get to that at one point. I already have a NAS and everything goes in it, and everything on it gets deduplicated / compressed / encrypted and sent to several cloud storage services, but still, you never know.

blincoln 4y ago
Based on your description (wget + git), I think your bot is going to miss a lot of content on modern websites. Have you tried viewing the content you've archived recently? A lot of modern websites don't even download the actual page/article content until some JavaScript runs. I noticed this in particular when I was collecting daily Monkeypox stats from the CDC website.[1] The Internet Archive had indexed the public URL every day, but that page used JavaScript to download the actual daily stats from a separate JSON or XML file and used that data to render the page. The Internet Archive bot was only downloading that data file maybe every 2-3 weeks, so even though they had archives of the page for almost every day, nearly all of the per-day data was missing. I don't write bots often anymore, but for the last ten years or so I've defaulted to using an automated instance of an actual browser, because it's the only way you'll get all of the page content to load accurately, especially on sites that use the "lazy loading" anti-pattern.[2] Even if you do that, it's still another step (probably several steps) to archive the content, because the browser just loads the page and resources, it doesn't save them, and in a lot of cases images in articles are downscaled and you want to archive the full-res version as well. TLDR: make sure you're really backing up what you think you're backing up. [1] At the time, the CDC was only showing the current figures, not the historical data, so if you wanted a graph based for sure on their real data, not some third party, you needed to collect it yourself. [2] I know a lot of devs think lazy loading is great, because it reduces load on their servers, but it's an anti-pattern for UX IMO because it causes the page to take longer to load than if the browser were allowed to continue loading resources before the user even started scrolling.
- pdimitar 4y ago
  Good points, yep. I'm too indifferent to try and fight to have an automated browser download stuff for me I admit, plus the websites I archive are server-side rendered, but if I ever get to the point you speak of then I'm much more likely to find how are they downloading the data that they're rendering, download their CSS and turn the page to a static one after downloading the data myself. They're interesting problems for sure but I don't have as much time and energy for it if I'm not get paid to do it. But I also feel it's a dying art so I try to keep up-to-date every now and then. (If you can share how you do the automated browser workflow, that will be valuable.)
  - tiagod 4y ago
    Check out the SingleFile browser extension. It's open-source and automates Firefox/chrome to turn any content page into a static HTML file with everything bundled in (js is stripped out). In my experience it handles anything you throw at it remarkably well. It can be automated from the command line too.
    - pdimitar 4y ago
      Oh right, I completely forgot about that one even though I have it bookmarked. Thanks.

blincoln 4y ago

Based on your description (wget + git), I think your bot is going to miss a lot of content on modern websites. Have you tried viewing the content you've archived recently? A lot of modern websites don't even download the actual page/article content until some JavaScript runs. I noticed this in particular when I was collecting daily Monkeypox stats from the CDC website.[1] The Internet Archive had indexed the public URL every day, but that page used JavaScript to download the actual daily stats from a separate JSON or XML file and used that data to render the page. The Internet Archive bot was only downloading that data file maybe every 2-3 weeks, so even though they had archives of the page for almost every day, nearly all of the per-day data was missing. I don't write bots often anymore, but for the last ten years or so I've defaulted to using an automated instance of an actual browser, because it's the only way you'll get all of the page content to load accurately, especially on sites that use the "lazy loading" anti-pattern.[2] Even if you do that, it's still another step (probably several steps) to archive the content, because the browser just loads the page and resources, it doesn't save them, and in a lot of cases images in articles are downscaled and you want to archive the full-res version as well. TLDR: make sure you're really backing up what you think you're backing up. [1] At the time, the CDC was only showing the current figures, not the historical data, so if you wanted a graph based for sure on their real data, not some third party, you needed to collect it yourself. [2] I know a lot of devs think lazy loading is great, because it reduces load on their servers, but it's an anti-pattern for UX IMO because it causes the page to take longer to load than if the browser were allowed to continue loading resources before the user even started scrolling.

pdimitar 4y ago
Good points, yep. I'm too indifferent to try and fight to have an automated browser download stuff for me I admit, plus the websites I archive are server-side rendered, but if I ever get to the point you speak of then I'm much more likely to find how are they downloading the data that they're rendering, download their CSS and turn the page to a static one after downloading the data myself. They're interesting problems for sure but I don't have as much time and energy for it if I'm not get paid to do it. But I also feel it's a dying art so I try to keep up-to-date every now and then. (If you can share how you do the automated browser workflow, that will be valuable.)
- tiagod 4y ago
  Check out the SingleFile browser extension. It's open-source and automates Firefox/chrome to turn any content page into a static HTML file with everything bundled in (js is stripped out). In my experience it handles anything you throw at it remarkably well. It can be automated from the command line too.
  - pdimitar 4y ago
    Oh right, I completely forgot about that one even though I have it bookmarked. Thanks.