6 ms·

These are some interesting tips. And as a bonus, the page is giving me serious flashbacks to Fravia's Web Searchlores! http://search.lores.eu/indexo.htm http://

These are some interesting tips. And as a bonus, the page is giving me serious flashbacks to Fravia's Web Searchlores! http://search.lores.eu/indexo.htm http://search.lores.eu/indexo.htm

gwern 8y ago
Hopefully my tips are a little easier to read and learn from than Fravia's. And much more up to date...
- sitkack 8y ago
  Thanks for writing this up. I see no mention of the wayback machine. Nor digging into raw git and mercurial repositories. I have also had luck emailing folks that either inquired about document or last good email for the author. Universities often have blanket access to journals if you use their wifi. SH has mostly supplanted the need for this, but not always. Universities are still great for interlibrary loans, if you are friendly with the library staff, they might let you IL a book and read it in the library if they don't checkout to citizens. Last one on mining a local search database is like you said, to determine the boolean operators, get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) and some lightweight un-obtrusive scripting (using in browser js console). Like this snippet that extracts the pdfs and doi.org urls from a page. `copy` is a chrome specific helper. This snippet is great for extracting results that are lazily loaded in, something wget cannot do for you. function gpdfs() { aref = document.getElementsByTagName("a"); var result = []; for(i = 0; i < aref.length; i++) { var t = aref[i]; if (t.href.endsWith(".pdf") || t.href.includes("doi.org")) { result.push(t.href); } } return result; } function pc() { copy(gpdfs()); } pc();
  - gwern 8y ago
    > I see no mention of the wayback machine. The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA. > Nor digging into raw git and mercurial repositories. I have never had to do that for any research task I've undertaken, so that would be both too obscure to mention and I wouldn't know anything about it. > Universities often have blanket access to journals if you use their wifi. A good point. My proxy superseded this for me but I used to do this, and simply forgot about it. I'll add that. (Another good trick I know is using the old Google Reader RSS exports hosted on IA to get fulltext of webpages. I'll have to add that too: https://www.gwern.net/Search#searching-the-google-reader-archives https://www.gwern.net/Search#searching-the-google-reader-arc... ) > get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) I was really just thinking of grep and some other CLI utilities there. :)
    - sitkack 8y ago
      > I was really just thinking of grep and some other CLI utilities there. :) Another technique I use is to spider a site (respectfully) and then use https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep or load the spidered data into a local Elastic search. Some of what you outline in searching other parts dovetail into handling poor access to the Internet. For folks with bad or transient access pip install --upgrade youtube_dl To download youtube videos (or complete playlists) for offline viewing.
    - greglindahl 8y ago
      BTW "pip install warcio" is the latest hotness for processing warc files. Also, you can add a header to wget to download a byte range. Unfortunately archival tools are mostly targeted at crawling and then playback, not so much special collections like this one.
      - gwern 8y ago
        How do you use wget? I know you can specify a start position, intended for resuming downloads, but you need to specify an end position as well (to extract just the single compressed file), and I don't see any way to specify an end in the wget manpage.
        greglindahl 8y ago
        Hm, come to think of it, I guess I was programming in python at the time. The internets say that wget doesn't support range requests and while curl's manpage says --range works, it doesn't work for me.

These are some interesting tips. And as a bonus, the page is giving me serious flashbacks to Fravia's Web Searchlores! http://search.lores.eu/indexo.htm http://search.lores.eu/indexo.htm

gwern 8y ago

Hopefully my tips are a little easier to read and learn from than Fravia's. And much more up to date...

sitkack 8y ago
Thanks for writing this up. I see no mention of the wayback machine. Nor digging into raw git and mercurial repositories. I have also had luck emailing folks that either inquired about document or last good email for the author. Universities often have blanket access to journals if you use their wifi. SH has mostly supplanted the need for this, but not always. Universities are still great for interlibrary loans, if you are friendly with the library staff, they might let you IL a book and read it in the library if they don't checkout to citizens. Last one on mining a local search database is like you said, to determine the boolean operators, get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) and some lightweight un-obtrusive scripting (using in browser js console). Like this snippet that extracts the pdfs and doi.org urls from a page. `copy` is a chrome specific helper. This snippet is great for extracting results that are lazily loaded in, something wget cannot do for you. function gpdfs() { aref = document.getElementsByTagName("a"); var result = []; for(i = 0; i < aref.length; i++) { var t = aref[i]; if (t.href.endsWith(".pdf") || t.href.includes("doi.org")) { result.push(t.href); } } return result; } function pc() { copy(gpdfs()); } pc();
- gwern 8y ago
  > I see no mention of the wayback machine. The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA. > Nor digging into raw git and mercurial repositories. I have never had to do that for any research task I've undertaken, so that would be both too obscure to mention and I wouldn't know anything about it. > Universities often have blanket access to journals if you use their wifi. A good point. My proxy superseded this for me but I used to do this, and simply forgot about it. I'll add that. (Another good trick I know is using the old Google Reader RSS exports hosted on IA to get fulltext of webpages. I'll have to add that too: https://www.gwern.net/Search#searching-the-google-reader-archives https://www.gwern.net/Search#searching-the-google-reader-arc... ) > get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) I was really just thinking of grep and some other CLI utilities there. :)
  - sitkack 8y ago
    > I was really just thinking of grep and some other CLI utilities there. :) Another technique I use is to spider a site (respectfully) and then use https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep or load the spidered data into a local Elastic search. Some of what you outline in searching other parts dovetail into handling poor access to the Internet. For folks with bad or transient access pip install --upgrade youtube_dl To download youtube videos (or complete playlists) for offline viewing.
  - greglindahl 8y ago
    BTW "pip install warcio" is the latest hotness for processing warc files. Also, you can add a header to wget to download a byte range. Unfortunately archival tools are mostly targeted at crawling and then playback, not so much special collections like this one.
    - gwern 8y ago
      How do you use wget? I know you can specify a start position, intended for resuming downloads, but you need to specify an end position as well (to extract just the single compressed file), and I don't see any way to specify an end in the wget manpage.
      - greglindahl 8y ago
        Hm, come to think of it, I guess I was programming in python at the time. The internets say that wget doesn't support range requests and while curl's manpage says --range works, it doesn't work for me.