4 ms·

A key part of a scientific-research education is learning how to filter the vast pool of literature and recover interesting and meaningful books, reviews and pr

A key part of a scientific-research education is learning how to filter the vast pool of literature and recover interesting and meaningful books, reviews and primary publications. It's not that trivial and is definitely a learned skill. Take the author's example, 'entropy'. Well, a one-word search is of course going to generate a massive pile of unsorted material. Internet Archive Scholar gives ~873,000 hits. Nice resource by the way: https://news.ycombinator.com/item?id=33919823 https://news.ycombinator.com/item?id=33919823 If you knew nothing about entropy, but knew how to research, you might start with, "okay, when is the first appearance of the word 'entropy' in the literature" and you quickly discover it's Ludwig Boltzmann, 1872. Seems to have something to do with energy and information, and if we add in those terms, we're down to 284K hits. At this point, you might think "I bet someone has written a good book on this broad topic, maybe I shouldn't be looking at the primary literature until later." Notice that adding search terms narrows results? Keep doing that until you get a smaller number of hits. Now, you can grab the top dozen or so papers, and flip to the bibliography, and look to see if they all reference a landmark paper (this is related to searching by citation count). That's probably one you want to look at. With non-research-literature-focused internet search engines, the same general rules apply. More search terms tends to get better results. If you find a site with a good article, search that site for more. For example, this gives a lot of good results, for the role of entropy in machine learning concepts at least: 'site:towardsdatascience.com entropy Shannon' It's basically mining a haystack for needles. There are lots of techniques and strategies one can use to speed up the process.

marcosdumay 4y ago
> With non-research-literature-focused internet search engines, the same general rules apply. More search terms tends to get better results. Not anymore. For a decade or so. Nowadays adding terms on general purpose internet search engines makes your results less focused, not more.
- photochemsyn 4y ago
  I don't understand your comment. Something like: 'entropy shannon machine learning differential' will give more focused results than just 'entropy', certainly. I suppose there is some limit, is that what you mean? Even there, if I add 'literature review' the results get more focused.
  - marcosdumay 4y ago
    If you add terms until a search engine gets out of results it considers relevant, it will start rewriting your query until it finds relevant results again. Your examples only work because there are plenty of popular sites that talk about this stuff.
    - pixl97 4y ago
      I think what you're addressing here is that search engines are not infinite. I think one of the things that's changed here in the past view years is 'infinite shit SEO engines'. Some of these sites are obvious and you can type random statements and see that some series of garbage sites attempt to give hits on it. There are less obvious versions of these sites, and I'm assuming they are much harder for Google to detect. Now things boil down to the Chinese room problem. Take a topic with no 'popular' sites that can work as traffic directors. How can Google determine if you're authoritative on the subject, or if you're just a SEO site spitting out spam?
sitkack 4y ago
I point people to ESR's "How To Ask Questions The Smart Way" [1], using a search engine (and libraries) is indeed an important skill that must be learned and honed. I really enjoyed Fravia's [2] SearchLores [3] site on how to dig deep into the knowledge of the internet. Most people rarely cross the boundary of the SEO sludge into the really good stuff. There are a couple easy patterns, starting from a seed to iteratively widen and deepen the context. Wikipedia, Archive.org, site:edu, filetype:pdf, libgen, and now chat.openai.com I just asked chatgpt, 'What is a good home experiment to show the concept of "entropy"' and the answer was excellent. I then tried the same question on DDG but appended "site:edu" and found https://paradigms.oregonstate.edu/activity/325/ https://paradigms.oregonstate.edu/activity/325/ Now I am off to do the "Time Dilation Clock Skit" with my kid https://paradigms.oregonstate.edu/activity/548/ https://paradigms.oregonstate.edu/activity/548/ [1] https://www.catb.org/esr/faqs/smart-questions.html https://www.catb.org/esr/faqs/smart-questions.html (something wrong with his certs) [2] https://en.wikipedia.org/wiki/Fravia https://en.wikipedia.org/wiki/Fravia [3] https://web.archive.org/web/20191201105758/http://search.lores.eu/indexo.htm https://web.archive.org/web/20191201105758/http://search.lor...
bgilroy26 4y ago
I'm only commenting because by coincidence, I wrote a similar google search yesterday ----------------------- The first page of results for "dan dill entropy site:bu.edu" Yields https://www.bu.edu/genchem/ch131-summer-1-2021/notes/SecondLaw.pdf https://www.bu.edu/genchem/ch131-summer-1-2021/notes/SecondL...

marcosdumay 4y ago

> With non-research-literature-focused internet search engines, the same general rules apply. More search terms tends to get better results. Not anymore. For a decade or so. Nowadays adding terms on general purpose internet search engines makes your results less focused, not more.

photochemsyn 4y ago
I don't understand your comment. Something like: 'entropy shannon machine learning differential' will give more focused results than just 'entropy', certainly. I suppose there is some limit, is that what you mean? Even there, if I add 'literature review' the results get more focused.
- marcosdumay 4y ago
  If you add terms until a search engine gets out of results it considers relevant, it will start rewriting your query until it finds relevant results again. Your examples only work because there are plenty of popular sites that talk about this stuff.
  - pixl97 4y ago
    I think what you're addressing here is that search engines are not infinite. I think one of the things that's changed here in the past view years is 'infinite shit SEO engines'. Some of these sites are obvious and you can type random statements and see that some series of garbage sites attempt to give hits on it. There are less obvious versions of these sites, and I'm assuming they are much harder for Google to detect. Now things boil down to the Chinese room problem. Take a topic with no 'popular' sites that can work as traffic directors. How can Google determine if you're authoritative on the subject, or if you're just a SEO site spitting out spam?

sitkack 4y ago

I point people to ESR's "How To Ask Questions The Smart Way" [1], using a search engine (and libraries) is indeed an important skill that must be learned and honed. I really enjoyed Fravia's [2] SearchLores [3] site on how to dig deep into the knowledge of the internet. Most people rarely cross the boundary of the SEO sludge into the really good stuff. There are a couple easy patterns, starting from a seed to iteratively widen and deepen the context. Wikipedia, Archive.org, site:edu, filetype:pdf, libgen, and now chat.openai.com I just asked chatgpt, 'What is a good home experiment to show the concept of "entropy"' and the answer was excellent. I then tried the same question on DDG but appended "site:edu" and found https://paradigms.oregonstate.edu/activity/325/ https://paradigms.oregonstate.edu/activity/325/ Now I am off to do the "Time Dilation Clock Skit" with my kid https://paradigms.oregonstate.edu/activity/548/ https://paradigms.oregonstate.edu/activity/548/ [1] https://www.catb.org/esr/faqs/smart-questions.html https://www.catb.org/esr/faqs/smart-questions.html (something wrong with his certs) [2] https://en.wikipedia.org/wiki/Fravia https://en.wikipedia.org/wiki/Fravia [3] https://web.archive.org/web/20191201105758/http://search.lores.eu/indexo.htm https://web.archive.org/web/20191201105758/http://search.lor...

bgilroy26 4y ago

I'm only commenting because by coincidence, I wrote a similar google search yesterday ----------------------- The first page of results for "dan dill entropy site:bu.edu" Yields https://www.bu.edu/genchem/ch131-summer-1-2021/notes/SecondLaw.pdf https://www.bu.edu/genchem/ch131-summer-1-2021/notes/SecondL...