Based on your description (wget + git), I think your bot is going to miss a lot of content on modern websites. Have you tried viewing the content you've archive

Based on your description (wget + git), I think your bot is going to miss a lot of content on modern websites. Have you tried viewing the content you've archived recently? A lot of modern websites don't even download the actual page/article content until some JavaScript runs. I noticed this in particular when I was collecting daily Monkeypox stats from the CDC website.[1] The Internet Archive had indexed the public URL every day, but that page used JavaScript to download the actual daily stats from a separate JSON or XML file and used that data to render the page. The Internet Archive bot was only downloading that data file maybe every 2-3 weeks, so even though they had archives of the page for almost every day, nearly all of the per-day data was missing. I don't write bots often anymore, but for the last ten years or so I've defaulted to using an automated instance of an actual browser, because it's the only way you'll get all of the page content to load accurately, especially on sites that use the "lazy loading" anti-pattern.[2] Even if you do that, it's still another step (probably several steps) to archive the content, because the browser just loads the page and resources, it doesn't save them, and in a lot of cases images in articles are downscaled and you want to archive the full-res version as well. TLDR: make sure you're really backing up what you think you're backing up. [1] At the time, the CDC was only showing the current figures, not the historical data, so if you wanted a graph based for sure on their real data, not some third party, you needed to collect it yourself. [2] I know a lot of devs think lazy loading is great, because it reduces load on their servers, but it's an anti-pattern for UX IMO because it causes the page to take longer to load than if the browser were allowed to continue loading resources before the user even started scrolling.