6 ms·
My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point loc
by falcolas 11y ago
My favorite use of wget: mirroring web documentation to my local machine.
wget -r -l5 -k -np -p https://docs.python.org/2/
Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation.
I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects by default, `curl -O` will not. Yes, I could remember yet another argument to curl... but why?
That said, I love curl (combined with `jq`) for playing with rest interfaces.
- look_lookatme 11y agoN.B., this will make curl redirect by default: echo "-L" >> ~/.curlrc
- tomphoolery 11y agoI bet that would be extra useful for creating Dash.app docsets...
- pyre 11y agoif you look at some of the default docsets you can tell that this was most likely done. All of the directories under the file root in the docset are named after domain names (one of the things whet does when crawling a site and traversing multiple domains).
- jsmthrowaway 11y agoIs this the part where I tell you about pydoc and blow your mind? I had a patch at one point to make pydoc style itself just like docs.p.o, but I'm not working much in Python these days. Maybe it's not even necessary any more, which would be cool.
- falcolas 11y agoSure. Does it work with golang docs, the w3 HTML5 spec, Mozilla's JS reference, etc? ;) Python's built in documentation tools are good, but the command I referenced also gives you the full language spec locally, as well as the FAQs and packaging docs.
- daveguy 11y agoThanks for the jq tip. I hadn't seen that before. Link if anyone's interested: https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- k3n 11y agoUnderscore CLI looks interesting as well, though I haven't personally had much of a chance to play with it. It does require NodeJS, which might be a deal-breaker for some, but if it's already in your toolchain then it might come in handy. https://github.com/ddopson/underscore-cli https://github.com/ddopson/underscore-cli
- daxelrod 11y agoI've written a similar tool underscore-cli and jq called jowl. It's designed to be easier to learn (for JavaScript developers) than underscore-cli or jq. The README includes a comparison. It's still early in its development, and would benefit from a tutorial and a few more features, but it's getting there. https://www.npmjs.com/package/jowl https://www.npmjs.com/package/jowl
- robk 11y agoOr just curl -OL
- thrownaway2424 11y agowget is ludicrously good for mirroring. I use it to mirror the entirety of EDGAR. For my money another tool that belongs in the toolbox is perl's WWW::Mechanize and its component WWW::Mechanize::Shell.
- iglookid 11y agoHow does wget compare with httrack for mirroring?
- timClicks 11y agoIt's somewhat hard to compare CLI vs GUI programs.. but in my experience, wget --mirror is simpler than HTTrack for simple tasks, which takes a little bit of mucking around to start downloading. HTTrack has a smaller learning curve for more complex situations though.
- mkobit 11y agoWow. This is so incredibly useful to me. The general workflow I have followed was, "Download PDF if present. If documentation can be shown in single HTML page, download it, change all the anchors to point to my own filesystem". This changes everything in such a simple, useful, way.
- heinrich5991 11y agoFor docs specifically, you could also try Zeal, which allows you to browse documentation for quite a few programming languages offline, including Mozilla's JS docs and an HTML spec, as you mentioned in some other comment in this thread. https://zealdocs.org/ https://zealdocs.org/
- tripzilch 11y agoThis is a great use, and I've used it for a lot of other sites and documentation. However I just want to point out that Python's documentation happens to come in an easily downloadable archive of HTML files: https://docs.python.org/2.7/download.html https://docs.python.org/2.7/download.html (you can also find archives for Python 3 and SciPy and NumPy)