4 ms·
> I see no mention of the wayback machine. The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA. > Nor digging into raw gi
by gwern 8y ago
> I see no mention of the wayback machine.
The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA.
> Nor digging into raw git and mercurial repositories.
I have never had to do that for any research task I've undertaken, so that would be both too obscure to mention and I wouldn't know anything about it.
> Universities often have blanket access to journals if you use their wifi.
A good point. My proxy superseded this for me but I used to do this, and simply forgot about it. I'll add that. (Another good trick I know is using the old Google Reader RSS exports hosted on IA to get fulltext of webpages. I'll have to add that too: https://www.gwern.net/Search#searching-the-google-reader-archives https://www.gwern.net/Search#searching-the-google-reader-arc... )
> get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc)
I was really just thinking of grep and some other CLI utilities there. :)
- sitkack 8y ago> I was really just thinking of grep and some other CLI utilities there. :) Another technique I use is to spider a site (respectfully) and then use https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep or load the spidered data into a local Elastic search. Some of what you outline in searching other parts dovetail into handling poor access to the Internet. For folks with bad or transient access pip install --upgrade youtube_dl To download youtube videos (or complete playlists) for offline viewing.
- greglindahl 8y agoBTW "pip install warcio" is the latest hotness for processing warc files. Also, you can add a header to wget to download a byte range. Unfortunately archival tools are mostly targeted at crawling and then playback, not so much special collections like this one.
- gwern 8y agoHow do you use wget? I know you can specify a start position, intended for resuming downloads, but you need to specify an end position as well (to extract just the single compressed file), and I don't see any way to specify an end in the wget manpage.
- greglindahl 8y agoHm, come to think of it, I guess I was programming in python at the time. The internets say that wget doesn't support range requests and while curl's manpage says --range works, it doesn't work for me.