4 ms·
> Mr. Sweeting explained in his email that a Linux command, “wget,” allows anyone to download the contents of a website, “including all images, HTML, CSS, etc.”
by danso 9y ago
> Mr. Sweeting explained in his email that a Linux command, “wget,” allows anyone to download the contents of a website, “including all images, HTML, CSS, etc.”
According to my research [0], this is the second time in New York Times history that the word "wget" has appeared in the NYT.
The first time was in 2014:
https://www.nytimes.com/2014/02/09/us/snowden-used-low-cost-tool-to-best-nsa.html https://www.nytimes.com/2014/02/09/us/snowden-used-low-cost-...
> Evidence presented during Private Manning’s court-martial for his role as the source for large archives of military and diplomatic files given to WikiLeaks revealed that he had used a program called “wget” to download the batches of files. That program automates the retrieval of large numbers of files, but it is considered less powerful than the tool Mr. Snowden used.
[0] http://query.nytimes.com/search/sitesearch/#/wget/since1851/allresults/1/allauthors/newest/ http://query.nytimes.com/search/sitesearch/#/wget/since1851/...
- Redoubts 9y agoIt would be interesting to see a count for cURL :)
- Flammy 9y agohttp://query.nytimes.com/search/sitesearch/#/cURL/since1851/allresults/1/allauthors/newest/ http://query.nytimes.com/search/sitesearch/#/cURL/since1851/... Sadly the search is not case sensitive, so a lot of the recent hits are hurricane paths curling...
- nikisweeting 9y agoI'm a huge wget fan! It's the core tech inside my archiving tool: https://github.com/pirate/bookmark-archiver https://github.com/pirate/bookmark-archiver
- danso 9y agoVery nice. I've been using wget by itself to archive various government pages, but that approach is reaching its end-life with JS-heavy sites becoming more prevalent. You use headless Chromium for screenshots; is it possible to use it to execute a page's JS and save the resulting HTML?
- nikisweeting 9y agoYes, headless chrome `--dump-dom` allows you to dump the <body> html after the page loads. I opted not to do that in bookmark-archiver since glueing it back to the <head> code to get a working static page was complicated and error prone.