5 ms·

Archiving data from the Internet Archive is useful, but I would argue there's a greater need going unfulfilled: local direct site archiving. Twenty years ago yo

by causi 4y ago

Archiving data from the Internet Archive is useful, but I would argue there's a greater need going unfulfilled: local direct site archiving. Twenty years ago you could punch a URL into HTTRACK and get a local site copy that was indistinguishable from the live version. As far as I know there's nothing like that available today. The closest methods I know involve stringing together four or five entirely separate utilities to get one saved site.

vorpalhex 4y ago
ArchiveBox https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox
- causi 4y ago
  If I'm reading this right, ArchiveBox is intended for archiving individual web pages, not entire sites.
Brajeshwar 4y ago
Not to archive, but this was my way to learn HTML/CSS to build websites in 1999-2000.
phiresky 4y ago
https://github.com/webrecorder/browsertrix-crawler https://github.com/webrecorder/browsertrix-crawler works pretty well. Same tech as https://archiveweb.page/ https://archiveweb.page/ but non-interactive
marban 4y ago
Cloudflare has more or less killed white-hat crawling en large. On top, you have sites like Bloomberg where even solutions like Puppeteer-stealth will fail at some point.
- xwolfi 4y ago
  And Selenium with clever clicking ? I never got why crawlers never took the approach to just be a slow human clicking around chrome.
  - traverseda 4y ago
    Anyone looking into this, I'd recommend python-playwright. Good browser engines and the ability to hot patch every request means you don't need to write a separate proxy to intercept requests. Writing something like this would be a pretty big project, at least a few solid weeks of work for someone who knew what they were doing, and there's a lot of edge cases.
- marginalia_nu 4y ago
  I run a crawler and I dunno if I agree with this description. Usually anti-bot measures are in place for a reason, either to protect some paywalled content from scraping, or to prevent bots from getting at forms and causing a mess. Few webmasters have cloudflare (or similar) anti-bot measures turned up higher than it needs to be (because it 100% will annoy people), and content that isn't a high-value target for blackhat bot activity is rarely difficult to crawl.
deleted 4y ago
[deleted]
walterbell 4y ago
On iOS/macOS, DevonThink includes a web scraper with full-text local search.
- janandonly 4y ago
  This piqued my interest until I saw the $99 price.
  - walterbell 4y ago
    iOS version is $3/mo or $50 for non-subscription. Cross-device sync via WebDAV, Dropbox or iCloud. Pricy, but they are an indie dev team who have been around for ~20 years, provide email support by competent humans, continuously add features, and most importantly, have been reliable with large databases (e.g. 75GB) on iOS. Web scraper can optionally convert to plaintext, markdown or PDF.
taffit 4y ago
SingleFile [0] and SingleFileZ [1] are my tools of choice for archiving pages. Often using an adjusted version of the aardvark-bookmarklet for cutting out unwanted parts. [0] https://chrome.google.com/webstore/detail/singlefile/mpiodijhokgodhhofbcjdecpffjipkle https://chrome.google.com/webstore/detail/singlefile/mpiodij... [1] https://chrome.google.com/webstore/detail/singlefilez/offkdfbbigofcgdokjemgjpdockaafjg https://chrome.google.com/webstore/detail/singlefilez/offkdf...
aliqot 4y ago
I just used HTTrack last week, it still works fine
suramya_tomar 4y ago
Well, wget still works and allows you to mirror a website with a single command The following command mirrors a given website: wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://site-to-download.com https://site-to-download.com So not sure what you mean by there are no tools available to mirror a website.

by causi 4y ago

vorpalhex 4y ago

ArchiveBox https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox

causi 4y ago
If I'm reading this right, ArchiveBox is intended for archiving individual web pages, not entire sites.

Brajeshwar 4y ago

Not to archive, but this was my way to learn HTML/CSS to build websites in 1999-2000.

phiresky 4y ago

https://github.com/webrecorder/browsertrix-crawler https://github.com/webrecorder/browsertrix-crawler works pretty well. Same tech as https://archiveweb.page/ https://archiveweb.page/ but non-interactive

marban 4y ago

Cloudflare has more or less killed white-hat crawling en large. On top, you have sites like Bloomberg where even solutions like Puppeteer-stealth will fail at some point.

xwolfi 4y ago
And Selenium with clever clicking ? I never got why crawlers never took the approach to just be a slow human clicking around chrome.
- traverseda 4y ago
  Anyone looking into this, I'd recommend python-playwright. Good browser engines and the ability to hot patch every request means you don't need to write a separate proxy to intercept requests. Writing something like this would be a pretty big project, at least a few solid weeks of work for someone who knew what they were doing, and there's a lot of edge cases.
marginalia_nu 4y ago
I run a crawler and I dunno if I agree with this description. Usually anti-bot measures are in place for a reason, either to protect some paywalled content from scraping, or to prevent bots from getting at forms and causing a mess. Few webmasters have cloudflare (or similar) anti-bot measures turned up higher than it needs to be (because it 100% will annoy people), and content that isn't a high-value target for blackhat bot activity is rarely difficult to crawl.

deleted 4y ago

[deleted]

walterbell 4y ago

On iOS/macOS, DevonThink includes a web scraper with full-text local search.

janandonly 4y ago
This piqued my interest until I saw the $99 price.
- walterbell 4y ago
  iOS version is $3/mo or $50 for non-subscription. Cross-device sync via WebDAV, Dropbox or iCloud. Pricy, but they are an indie dev team who have been around for ~20 years, provide email support by competent humans, continuously add features, and most importantly, have been reliable with large databases (e.g. 75GB) on iOS. Web scraper can optionally convert to plaintext, markdown or PDF.

taffit 4y ago

SingleFile [0] and SingleFileZ [1] are my tools of choice for archiving pages. Often using an adjusted version of the aardvark-bookmarklet for cutting out unwanted parts. [0] https://chrome.google.com/webstore/detail/singlefile/mpiodijhokgodhhofbcjdecpffjipkle https://chrome.google.com/webstore/detail/singlefile/mpiodij... [1] https://chrome.google.com/webstore/detail/singlefilez/offkdfbbigofcgdokjemgjpdockaafjg https://chrome.google.com/webstore/detail/singlefilez/offkdf...

aliqot 4y ago

I just used HTTrack last week, it still works fine

suramya_tomar 4y ago

Well, wget still works and allows you to mirror a website with a single command The following command mirrors a given website: wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://site-to-download.com https://site-to-download.com So not sure what you mean by there are no tools available to mirror a website.