3 ms·

"Here’s the only technique I’ve found that works, which I call scrobbling even though that means something totally different: "1. Search seed terms you know, l

"Here’s the only technique I’ve found that works, which I call scrobbling even though that means something totally different: "1. Search seed terms you know, like “cost of bugs”, in an appropriate journal (here’s one). "2. Find papers that look kinda relevant, skim their abstracts and conclusions. "3. Make a list of all the papers that either cite or are cited by these papers and repeat. "4. Find more useful terms and repeat." Congratulations! You're a grad student now! P.S. When I graduated, I had a filing cabinet with two drawers full of papers that felt important enough to save printed copies, from just this process. A couple of months ago, I carted them out to the burn pile, threw them on the pile of leaves and brush, and held a Viking funeral. And I didn't even feel bad. :-)

hwayne 5y ago
I got into software specifically because I didn't want to go to physics grad school, and now I read more papers than I ever did in physics. Man plans, God laughs.
titzer 5y ago
My impression on reading this list was, uh yeah, there's no avoiding that. Except if you know an expert. Then just ask an expert. Search engines for papers suck, bad. At the fringes of human knowledge, you need an actual brain to process cutting-edge research. Either your own non-expert brain, or an expert brain to help guide you. That process is the same in every field. Older fields have textbooks with knowledge distilled from experts. That's a good place to start to get the state of the art > 20 years ago.
- frickinLasers 5y ago
  | Search engines for papers suck, bad. This screams the need for an intelligent search engine, with summarizing features that point out relevant parts of the paper, and (imho) a UI and algorithms that reach across disciplines and encourage discovery. So, for instance (I'm making this up), someone looking for a way to understand interferometry data might stumble on a useful regularization technique from image processing. I imagine academic libraries would pay really good money for that. Of course, it would help if a mostly complete body of scientific literature was available to crawl--like SciHub.
  - rmetzler 5y ago
    Wouldn't be some graph-traversing be enough? You find an important paper and then just search for papers which cite this one? And then sort by how many cite these (similar to the pagerank algorithm) and filter them by year and keyword? There aren't that many different citing stiles, so I'm pretty sure it would be possible.
    - frickinLasers 5y ago
      It is usually possible to do this (a digital journal can include links to cited/citing papers), but citations happen for a variety of reasons (background information, a single statistic, boosting a colleague's work...). Every citation applies to a different sentence or paragraph in an article. To understand the content and whether it is relevant, we still need to digest the text. There are also less-cited papers and journals that can be just as relevant--every article starts with zero citations, and the vast majority of good work out there isn't exciting enough to be accepted for Nature. Further, the process is rather incestuous: Previous person solved X problem with Y approach because that's the first thing someone came up with and it's just how it's done in our field, so let's continue doing this inefficient thing and citing that paper, maybe working on a better way to do things. Meanwhile in some other field, a mathematician or whatever came up with a far better solution to a similar problem long ago, but nobody in this field ever knew that, so it goes unnoticed. I do want to see what is highly cited because it's probably interesting, but I also want to see things that don't get that kind of attention but are applicable to my work, and things that I wouldn't know are applicable to my work. Another example: Currently, authors enter relevant keywords when they submit a paper. Maybe that works when someone searches the right combination of words, or maybe it doesn't because the search engines suck. Or because in one part of the world the topic has a completely different vocabulary, and we miss a whole library's worth of useful papers. I speak from a background in STEM. I can't vouch for other disciplines, but I imagine they have similar pain points. Heck, I don't even know what others in STEM think, other than "that's just how research works." I don't think it's an easy problem, or we wouldn't still be sifting through mounds of crap to find a few relevant, reproducible works worth reading and citing. I think it would involve figuring out the overarching themes and important methods in a paper and sorting them by their importance, and a little bit of fuzziness to say "hey, this isn't exactly what you're looking for, but it sure seems useful." This could even allow un-cited works a second chance. I think I'm describing two separate goals, and the fuzzy part could wait until the relevance problem is addressed. I don't think the problems are particularly easy, or Google Scholar would have solved the problem and monetized the solution already. But it seems like they're solvable problems.
hyperhopper 5y ago
The problem stated is that instead of being able to find widely cited or good papers, any amount of research involves trudging through hundreds of bad, off topic, outdated, unrelated, or otherwise useless papers, just to get an idea of what is being talked about in the field. And even when getting there, what you find may be completely non-practically relevant in the end. Just saying "yeah thats how things work in academia" Doesn't help anything; this article is pointing out a very real problem: the state of things now is a mess.
anonymousDan 5y ago
If you're really in a hurry, filter out all those not published at top venues (e.g. ICSE or FSE as top tier for SW engineering, or ASE/ISSTA the tier below), or at the very least cited by several papers at top venues. In the vast majority of cases everything else is likely to be noise, incremental, or preliminary work. Of course consider the above venues as (perhaps) necessary but not necessarily sufficient condition for quality. You have to realise that academia is hyper competitive. The best work tends to get published at top venues because they are very difficult to get published at, and offer hiring and promotion committees an easy filter. Use that in your favour to filter out much of the dross and/or poorly executed ideas.
joppy 5y ago
Yeah I was going to say, this sounds like a normal way of getting around the literature. We were taught to start mostly with review/summary papers from high-quality places, and work our way out from there.
chakkepolja 5y ago
Now I know why my professor explained breadth first search for an entire week!!

hwayne 5y ago

I got into software specifically because I didn't want to go to physics grad school, and now I read more papers than I ever did in physics. Man plans, God laughs.

titzer 5y ago

My impression on reading this list was, uh yeah, there's no avoiding that. Except if you know an expert. Then just ask an expert. Search engines for papers suck, bad. At the fringes of human knowledge, you need an actual brain to process cutting-edge research. Either your own non-expert brain, or an expert brain to help guide you. That process is the same in every field. Older fields have textbooks with knowledge distilled from experts. That's a good place to start to get the state of the art > 20 years ago.

frickinLasers 5y ago
| Search engines for papers suck, bad. This screams the need for an intelligent search engine, with summarizing features that point out relevant parts of the paper, and (imho) a UI and algorithms that reach across disciplines and encourage discovery. So, for instance (I'm making this up), someone looking for a way to understand interferometry data might stumble on a useful regularization technique from image processing. I imagine academic libraries would pay really good money for that. Of course, it would help if a mostly complete body of scientific literature was available to crawl--like SciHub.
- rmetzler 5y ago
  Wouldn't be some graph-traversing be enough? You find an important paper and then just search for papers which cite this one? And then sort by how many cite these (similar to the pagerank algorithm) and filter them by year and keyword? There aren't that many different citing stiles, so I'm pretty sure it would be possible.
  - frickinLasers 5y ago
    It is usually possible to do this (a digital journal can include links to cited/citing papers), but citations happen for a variety of reasons (background information, a single statistic, boosting a colleague's work...). Every citation applies to a different sentence or paragraph in an article. To understand the content and whether it is relevant, we still need to digest the text. There are also less-cited papers and journals that can be just as relevant--every article starts with zero citations, and the vast majority of good work out there isn't exciting enough to be accepted for Nature. Further, the process is rather incestuous: Previous person solved X problem with Y approach because that's the first thing someone came up with and it's just how it's done in our field, so let's continue doing this inefficient thing and citing that paper, maybe working on a better way to do things. Meanwhile in some other field, a mathematician or whatever came up with a far better solution to a similar problem long ago, but nobody in this field ever knew that, so it goes unnoticed. I do want to see what is highly cited because it's probably interesting, but I also want to see things that don't get that kind of attention but are applicable to my work, and things that I wouldn't know are applicable to my work. Another example: Currently, authors enter relevant keywords when they submit a paper. Maybe that works when someone searches the right combination of words, or maybe it doesn't because the search engines suck. Or because in one part of the world the topic has a completely different vocabulary, and we miss a whole library's worth of useful papers. I speak from a background in STEM. I can't vouch for other disciplines, but I imagine they have similar pain points. Heck, I don't even know what others in STEM think, other than "that's just how research works." I don't think it's an easy problem, or we wouldn't still be sifting through mounds of crap to find a few relevant, reproducible works worth reading and citing. I think it would involve figuring out the overarching themes and important methods in a paper and sorting them by their importance, and a little bit of fuzziness to say "hey, this isn't exactly what you're looking for, but it sure seems useful." This could even allow un-cited works a second chance. I think I'm describing two separate goals, and the fuzzy part could wait until the relevance problem is addressed. I don't think the problems are particularly easy, or Google Scholar would have solved the problem and monetized the solution already. But it seems like they're solvable problems.

hyperhopper 5y ago

The problem stated is that instead of being able to find widely cited or good papers, any amount of research involves trudging through hundreds of bad, off topic, outdated, unrelated, or otherwise useless papers, just to get an idea of what is being talked about in the field. And even when getting there, what you find may be completely non-practically relevant in the end. Just saying "yeah thats how things work in academia" Doesn't help anything; this article is pointing out a very real problem: the state of things now is a mess.

anonymousDan 5y ago

If you're really in a hurry, filter out all those not published at top venues (e.g. ICSE or FSE as top tier for SW engineering, or ASE/ISSTA the tier below), or at the very least cited by several papers at top venues. In the vast majority of cases everything else is likely to be noise, incremental, or preliminary work. Of course consider the above venues as (perhaps) necessary but not necessarily sufficient condition for quality. You have to realise that academia is hyper competitive. The best work tends to get published at top venues because they are very difficult to get published at, and offer hiring and promotion committees an easy filter. Use that in your favour to filter out much of the dross and/or poorly executed ideas.

joppy 5y ago

Yeah I was going to say, this sounds like a normal way of getting around the literature. We were taught to start mostly with review/summary papers from high-quality places, and work our way out from there.

chakkepolja 5y ago

Now I know why my professor explained breadth first search for an entire week!!