5 ms·

Internet Search Tips

174 points by telotortium 8y ago

gigama 8y ago
"If you can’t use it while chatting without the other person noting your pauses, it’s not fast enough." Still takes time to read, comprehend and reply.
- glomph 8y ago
  I think they might mean chatting online.
  - yorwba 8y ago
    Gwern has now replaced "chatting" by "IRCing".
guidovranken 8y ago
> master/PhD theses: sorry. It’s probably hopeless if it’s pre-2000. If you have a university proxy, you may be able to get a copy off ProQuest. Otherwise, you need full university ILL services, and even that might not be enough (a surprising number of universities appear to restrict access only to the university students/faculty, with the complicating factor of most theses being stored on microfilm). At least in the Netherlands there is [1] which indexes many papers/theses including very old ones, and many are freely accessible. [1] https://www.narcis.nl/ https://www.narcis.nl/
- btrettel 8y ago
  I've used my (US) university's interlibrary loan (ILL) service many times. In my experience theses and dissertations are treated like any other book in interlibrary loan. Local public libraries also often have interlibrary loan, perhaps for a fee. So I can't see why theses would be inaccessible via ILL outside of a university. I have run into some theses and dissertations being restricted to people affiliated with the particular university which holds the book, but that's rare in my experience. More common is the case of rare books (few copies available, like in the case of theses/dissertations) being marked as "library use only", so you can't take the book outside of the library. But I have received many of these via ILL, and my university simply respects the wishes of the book's owner by not allowing me to take the book outside of the library I requested the book from.
  - gwern 8y ago
    As a kid, I made heavy use of my public library's ILL (sorry, taxpayers), but I never heard any whisper that university-level ILL of any documents was possible, and the forms my librarians filled out strongly implied that only books & CDs were ever contemplated. > In my experience theses and dissertations are treated like any other book in interlibrary loan. Yes, but you have to be at a university in the first place. Life is grand if you're a student with full privileges - ILL is, IMO, one of the most underrated fringe benefits of being a student - but once you're out, you're out in the cold. I've discussed this with any number of people, including people at think-tanks without university affiliations, and no one's come up with a good solution how to get back into the ILL system short of hiring students to do requests for you. > I have run into some theses and dissertations being restricted to people affiliated with the particular university which holds the book, but that's rare in my experience. Yes, fairly rare, but it still happens. Unfortunately, I don't know what happens when you ILL them because I didn't run into any examples until after I graduated. I'm also puzzled when I run into online open access theses which are, however, embargoed for a year... (I simply schedule a reminder to go back, but I'm perplexed as to what could possibly be the point of that.)
    - btrettel 8y ago
      Have you tried requesting a thesis or dissertation via a public library's interlibrary loan service? I am fairly confident that most universities would lend a thesis or dissertation to a public library. I don't see any clear reason why they would not other than snobbery. (Edit: And I can recall a few instances where my university got a book via ILL from a public library. So the converse does happen, at least.) While I have not used public library ILL, my impressions is that the difference is more in scale than access. (Though access surely is reduced.) I recall reading public library ILL policies that charged for requests and only allowed one request at a time. That would be a lot worse than what I have right now, but much better than nothing. (Also, I have used ILL at two government labs and I would say the difference again was more in scale than access, and that these labs were somewhere between public and university libraries.) There are many paid document delivery services. Here's my university's one: https://www.lib.utexas.edu/find-borrow-request/interlibrary-loan/outside-orgs-individuals https://www.lib.utexas.edu/find-borrow-request/interlibrary-... I've used a few paid document delivery services before. They weren't cheap, but I was able to get some things that my university's ILL wasn't able to. Personally, I'd prefer some sort of scan request barter system. I scan X pages for you, you scan X pages for me. I've done similar things informally. r/scholar hasn't worked out well for anything not already digitized. I've thought before that someone should make a scan request website that gives you credits for fulfilling a request. Hopefully the lawyers would stay away from this as these items can't really be obtained any other way; if they were available for sale, people would buy them. As for embargoed theses, my impression is that usually the author wants to get a patent on something discussed in the thesis. In US law, you can only get a patent within a 1 year after publicly disclosing the invention. So the embargo gives them more time. There likely are other reasons as well, but this is the only one I've encountered.
- copperx 8y ago
  Regarding the hopeless comment: if the author is still alive the easiest way to get a copy is, well, by emailing the author. Unless the author is a hotshot, they should be able to procure an electronic copy, even if it's was done in pre-word processing times. Most authors would be flattered, not inconvenienced.
yusuf-lan3 8y ago
some good tips but the site is super hard to read (content structure wise)
- gwern 8y ago
  I've reorganized it a bit.
mrspeaker 8y ago
These are some interesting tips. And as a bonus, the page is giving me serious flashbacks to Fravia's Web Searchlores! http://search.lores.eu/indexo.htm http://search.lores.eu/indexo.htm
- gwern 8y ago
  Hopefully my tips are a little easier to read and learn from than Fravia's. And much more up to date...
  - sitkack 8y ago
    Thanks for writing this up. I see no mention of the wayback machine. Nor digging into raw git and mercurial repositories. I have also had luck emailing folks that either inquired about document or last good email for the author. Universities often have blanket access to journals if you use their wifi. SH has mostly supplanted the need for this, but not always. Universities are still great for interlibrary loans, if you are friendly with the library staff, they might let you IL a book and read it in the library if they don't checkout to citizens. Last one on mining a local search database is like you said, to determine the boolean operators, get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) and some lightweight un-obtrusive scripting (using in browser js console). Like this snippet that extracts the pdfs and doi.org urls from a page. `copy` is a chrome specific helper. This snippet is great for extracting results that are lazily loaded in, something wget cannot do for you. function gpdfs() { aref = document.getElementsByTagName("a"); var result = []; for(i = 0; i < aref.length; i++) { var t = aref[i]; if (t.href.endsWith(".pdf") || t.href.includes("doi.org")) { result.push(t.href); } } return result; } function pc() { copy(gpdfs()); } pc();
    - gwern 8y ago
      > I see no mention of the wayback machine. The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA. > Nor digging into raw git and mercurial repositories. I have never had to do that for any research task I've undertaken, so that would be both too obscure to mention and I wouldn't know anything about it. > Universities often have blanket access to journals if you use their wifi. A good point. My proxy superseded this for me but I used to do this, and simply forgot about it. I'll add that. (Another good trick I know is using the old Google Reader RSS exports hosted on IA to get fulltext of webpages. I'll have to add that too: https://www.gwern.net/Search#searching-the-google-reader-archives https://www.gwern.net/Search#searching-the-google-reader-arc... ) > get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) I was really just thinking of grep and some other CLI utilities there. :)
8bitsrule 8y ago
Most important in my experience are the terms that you choose for a search. Sometimes it helps to think about how most people would phrase a question. Often results can be too broad ... so use them to choose more specific terms to add to your query to 'focus in' on the result you need. (E.g. adding years (even specific dates) can add focus -and quality- to your results.) It helps to keep a growing list of bookmarks for high-quality 'specialist' sites with -a lot- of content.
saagarjha 8y ago
Great article, but I’m left disappointed after looking at the “obstacles” that were presented. For example, paywalls: why is it ok for legal documents to cost money to access? This should be free for everyone–if they need money for “technology” (somehow RECAP apparently doesn’t, or can figure out a way to cover their expenses without charging people) they should roll it into taxes. Same goes for university research, especially if it’s publicly funded. So much effort wasted for information that should be freely available.
etaioinshrdlu 8y ago
I am honestly concerned for Gwern's mental health. So much personal optimization taken to such an extreme, doing blatantly illegal things and blogging about it. Some are probably quite unhealthy. I am all for being the best person we all can be -- but Gwern seems to have made personal optimization the end goal itself. I think we should aim to be happy well adjusted people, not work ourselves to death, not drug ourselves to our physical limit. There were times in my life when I behaved similarly and it was rooted in deep dissatisfaction with life. To be clear, this is not an ad hominem attack. The article is amazing.
- sitkack 8y ago
  The ability to wield the power of the Internet to answer questions, not just optimizing a shopping cart, but the ability to answer deep questions is a needed and important skill. I have zero issue if someone wants to optimize themselves, even with no other end other than "optimal".
- vonseel 8y ago
  Gwern is amazing and what a fascinating website. I hear you and understand where you are coming from, but some people thrive under extreme organization like this. These are the kind of people I want writing technical documentation on my teams.
- Recursing 8y ago
  > doing blatantly illegal things and blogging about it What are the legal risks of using (downloading from and uploading to) libgen, or buying from ebook.farm?
throwaway4790 8y ago
Worth adding in my opinion: booksc/bookfi/bookzz IRC bot rooms (#bookz, #ebooks and others) Private trackers (Bibliotik et al.) DHT search engines, eMule, DC++ In my country you can also just walk into a public library, get a free membership card and start browsing

174 points by telotortium 8y ago

"If you can’t use it while chatting without the other person noting your pauses, it’s not fast enough." Still takes time to read, comprehend and reply.

glomph 8y ago
I think they might mean chatting online.
- yorwba 8y ago
  Gwern has now replaced "chatting" by "IRCing".

guidovranken 8y ago

> master/PhD theses: sorry. It’s probably hopeless if it’s pre-2000. If you have a university proxy, you may be able to get a copy off ProQuest. Otherwise, you need full university ILL services, and even that might not be enough (a surprising number of universities appear to restrict access only to the university students/faculty, with the complicating factor of most theses being stored on microfilm). At least in the Netherlands there is [1] which indexes many papers/theses including very old ones, and many are freely accessible. [1] https://www.narcis.nl/ https://www.narcis.nl/

btrettel 8y ago
I've used my (US) university's interlibrary loan (ILL) service many times. In my experience theses and dissertations are treated like any other book in interlibrary loan. Local public libraries also often have interlibrary loan, perhaps for a fee. So I can't see why theses would be inaccessible via ILL outside of a university. I have run into some theses and dissertations being restricted to people affiliated with the particular university which holds the book, but that's rare in my experience. More common is the case of rare books (few copies available, like in the case of theses/dissertations) being marked as "library use only", so you can't take the book outside of the library. But I have received many of these via ILL, and my university simply respects the wishes of the book's owner by not allowing me to take the book outside of the library I requested the book from.
- gwern 8y ago
  As a kid, I made heavy use of my public library's ILL (sorry, taxpayers), but I never heard any whisper that university-level ILL of any documents was possible, and the forms my librarians filled out strongly implied that only books & CDs were ever contemplated. > In my experience theses and dissertations are treated like any other book in interlibrary loan. Yes, but you have to be at a university in the first place. Life is grand if you're a student with full privileges - ILL is, IMO, one of the most underrated fringe benefits of being a student - but once you're out, you're out in the cold. I've discussed this with any number of people, including people at think-tanks without university affiliations, and no one's come up with a good solution how to get back into the ILL system short of hiring students to do requests for you. > I have run into some theses and dissertations being restricted to people affiliated with the particular university which holds the book, but that's rare in my experience. Yes, fairly rare, but it still happens. Unfortunately, I don't know what happens when you ILL them because I didn't run into any examples until after I graduated. I'm also puzzled when I run into online open access theses which are, however, embargoed for a year... (I simply schedule a reminder to go back, but I'm perplexed as to what could possibly be the point of that.)
  - btrettel 8y ago
    Have you tried requesting a thesis or dissertation via a public library's interlibrary loan service? I am fairly confident that most universities would lend a thesis or dissertation to a public library. I don't see any clear reason why they would not other than snobbery. (Edit: And I can recall a few instances where my university got a book via ILL from a public library. So the converse does happen, at least.) While I have not used public library ILL, my impressions is that the difference is more in scale than access. (Though access surely is reduced.) I recall reading public library ILL policies that charged for requests and only allowed one request at a time. That would be a lot worse than what I have right now, but much better than nothing. (Also, I have used ILL at two government labs and I would say the difference again was more in scale than access, and that these labs were somewhere between public and university libraries.) There are many paid document delivery services. Here's my university's one: https://www.lib.utexas.edu/find-borrow-request/interlibrary-loan/outside-orgs-individuals https://www.lib.utexas.edu/find-borrow-request/interlibrary-... I've used a few paid document delivery services before. They weren't cheap, but I was able to get some things that my university's ILL wasn't able to. Personally, I'd prefer some sort of scan request barter system. I scan X pages for you, you scan X pages for me. I've done similar things informally. r/scholar hasn't worked out well for anything not already digitized. I've thought before that someone should make a scan request website that gives you credits for fulfilling a request. Hopefully the lawyers would stay away from this as these items can't really be obtained any other way; if they were available for sale, people would buy them. As for embargoed theses, my impression is that usually the author wants to get a patent on something discussed in the thesis. In US law, you can only get a patent within a 1 year after publicly disclosing the invention. So the embargo gives them more time. There likely are other reasons as well, but this is the only one I've encountered.
copperx 8y ago
Regarding the hopeless comment: if the author is still alive the easiest way to get a copy is, well, by emailing the author. Unless the author is a hotshot, they should be able to procure an electronic copy, even if it's was done in pre-word processing times. Most authors would be flattered, not inconvenienced.

yusuf-lan3 8y ago

some good tips but the site is super hard to read (content structure wise)

gwern 8y ago
I've reorganized it a bit.

mrspeaker 8y ago

These are some interesting tips. And as a bonus, the page is giving me serious flashbacks to Fravia's Web Searchlores! http://search.lores.eu/indexo.htm http://search.lores.eu/indexo.htm

gwern 8y ago
Hopefully my tips are a little easier to read and learn from than Fravia's. And much more up to date...
- sitkack 8y ago
  Thanks for writing this up. I see no mention of the wayback machine. Nor digging into raw git and mercurial repositories. I have also had luck emailing folks that either inquired about document or last good email for the author. Universities often have blanket access to journals if you use their wifi. SH has mostly supplanted the need for this, but not always. Universities are still great for interlibrary loans, if you are friendly with the library staff, they might let you IL a book and read it in the library if they don't checkout to citizens. Last one on mining a local search database is like you said, to determine the boolean operators, get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) and some lightweight un-obtrusive scripting (using in browser js console). Like this snippet that extracts the pdfs and doi.org urls from a page. `copy` is a chrome specific helper. This snippet is great for extracting results that are lazily loaded in, something wget cannot do for you. function gpdfs() { aref = document.getElementsByTagName("a"); var result = []; for(i = 0; i < aref.length; i++) { var t = aref[i]; if (t.href.endsWith(".pdf") || t.href.includes("doi.org")) { result.push(t.href); } } return result; } function pc() { copy(gpdfs()); } pc();
  - gwern 8y ago
    > I see no mention of the wayback machine. The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA. > Nor digging into raw git and mercurial repositories. I have never had to do that for any research task I've undertaken, so that would be both too obscure to mention and I wouldn't know anything about it. > Universities often have blanket access to journals if you use their wifi. A good point. My proxy superseded this for me but I used to do this, and simply forgot about it. I'll add that. (Another good trick I know is using the old Google Reader RSS exports hosted on IA to get fulltext of webpages. I'll have to add that too: https://www.gwern.net/Search#searching-the-google-reader-archives https://www.gwern.net/Search#searching-the-google-reader-arc... ) > get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) I was really just thinking of grep and some other CLI utilities there. :)

8bitsrule 8y ago

Most important in my experience are the terms that you choose for a search. Sometimes it helps to think about how most people would phrase a question. Often results can be too broad ... so use them to choose more specific terms to add to your query to 'focus in' on the result you need. (E.g. adding years (even specific dates) can add focus -and quality- to your results.) It helps to keep a growing list of bookmarks for high-quality 'specialist' sites with -a lot- of content.

saagarjha 8y ago

Great article, but I’m left disappointed after looking at the “obstacles” that were presented. For example, paywalls: why is it ok for legal documents to cost money to access? This should be free for everyone–if they need money for “technology” (somehow RECAP apparently doesn’t, or can figure out a way to cover their expenses without charging people) they should roll it into taxes. Same goes for university research, especially if it’s publicly funded. So much effort wasted for information that should be freely available.

etaioinshrdlu 8y ago

I am honestly concerned for Gwern's mental health. So much personal optimization taken to such an extreme, doing blatantly illegal things and blogging about it. Some are probably quite unhealthy. I am all for being the best person we all can be -- but Gwern seems to have made personal optimization the end goal itself. I think we should aim to be happy well adjusted people, not work ourselves to death, not drug ourselves to our physical limit. There were times in my life when I behaved similarly and it was rooted in deep dissatisfaction with life. To be clear, this is not an ad hominem attack. The article is amazing.

sitkack 8y ago
The ability to wield the power of the Internet to answer questions, not just optimizing a shopping cart, but the ability to answer deep questions is a needed and important skill. I have zero issue if someone wants to optimize themselves, even with no other end other than "optimal".
vonseel 8y ago
Gwern is amazing and what a fascinating website. I hear you and understand where you are coming from, but some people thrive under extreme organization like this. These are the kind of people I want writing technical documentation on my teams.
Recursing 8y ago
> doing blatantly illegal things and blogging about it What are the legal risks of using (downloading from and uploading to) libgen, or buying from ebook.farm?

throwaway4790 8y ago

Worth adding in my opinion: booksc/bookfi/bookzz IRC bot rooms (#bookz, #ebooks and others) Private trackers (Bibliotik et al.) DHT search engines, eMule, DC++ In my country you can also just walk into a public library, get a free membership card and start browsing