4 ms·

History has shown that heuristics like this are a fantastically good idea. They've been part of science more or less since science started, and science has acco

by Steuard 12y ago

History has shown that heuristics like this are a fantastically good idea. They've been part of science more or less since science started, and science has accomplished an enormous amount. The number of bold, exciting claims that are entirely wrong is far larger than the number of bold, exciting claims that are even partly correct. Heuristics like "put less trust in single-author papers" (among others) are essential for science to actually make progress rather than spending all its effort chasing mirages. Yes, sometimes an important idea gets overlooked for a decade or two along the way, but the alternative would be to accomplish nothing at all. (Unless you've been a physicist at a major university, you have no idea how many emails those folks get claiming to have quantized gravity or to have proved Einstein wrong.)

tjradcliffe 12y ago
As a colleague once put it, "I believe papers published by people I know. Failing that, by people who have worked with people I know. In third place, by people who are at institutions I know have a history of good work." Science has a deep internal tension that drives it forward, because it is both inherently elitist and necessarily public. If results are not published they aren't science, because they aren't incorporated by the scientific community. And yet while a) anyone can do science and b) anyone can publish, it is still the case that c) the scientific community is overwhelmingly tightly knit and fairly conservative when it comes to adopting new ideas, and the number of new ideas that get generated is so high that this sort of heuristic really is extremely valuable. Sure there are lone geniuses. There are also lone nutjobs. It's not difficult to guess which one predominates, and these sort of filters do a good enough job of bringing the nut/genius ratio down closer to unity that we are always going to use them.
api 12y ago
Statistically, if you always dismiss novel claims you will be right almost all the time. Yet this requires no intelligence at all. It's akin to a wall following msze solver. Wall following maze solvers do work, but only for the subset of mazes that are simply and directly connected end to end. You are correct that some way of sorting signal from noise is required, but I think it's got to be something up to the task. There is also clearly a breadth/depth tradeoff at work here that demands some amout of cost/benefit analysis. At one extreme you have rote dogma O(1), and at another you have full iterative search of the combinatorial space of all theory O(n^n^...). The way I think about issues like this is greatly informed by my study of learning theory, which if studied deeply infuses one with a great phobia of the dreaded local maximum and a suspicion of any and all deterministic non-adaptive learning strategies... As these tend to converge at local maxima. See: no free lunch theorem, among other things. My maze solver analogy is a pretty good simple one for how the space of learning algorithms maps to the space of fitness landscapes. The ideal strategy would probably be to run multiple heuristics concurrently or switch them up over time. For example, science might embark upon multi decade investigations of orthodoxy punctuated by jubilees where an accumulated queue of minority and fringe theories are investigated. Successes here form the next set of orthodoxies, rinse and repeat.

by Steuard 12y ago

tjradcliffe 12y ago

As a colleague once put it, "I believe papers published by people I know. Failing that, by people who have worked with people I know. In third place, by people who are at institutions I know have a history of good work." Science has a deep internal tension that drives it forward, because it is both inherently elitist and necessarily public. If results are not published they aren't science, because they aren't incorporated by the scientific community. And yet while a) anyone can do science and b) anyone can publish, it is still the case that c) the scientific community is overwhelmingly tightly knit and fairly conservative when it comes to adopting new ideas, and the number of new ideas that get generated is so high that this sort of heuristic really is extremely valuable. Sure there are lone geniuses. There are also lone nutjobs. It's not difficult to guess which one predominates, and these sort of filters do a good enough job of bringing the nut/genius ratio down closer to unity that we are always going to use them.

api 12y ago

Statistically, if you always dismiss novel claims you will be right almost all the time. Yet this requires no intelligence at all. It's akin to a wall following msze solver. Wall following maze solvers do work, but only for the subset of mazes that are simply and directly connected end to end. You are correct that some way of sorting signal from noise is required, but I think it's got to be something up to the task. There is also clearly a breadth/depth tradeoff at work here that demands some amout of cost/benefit analysis. At one extreme you have rote dogma O(1), and at another you have full iterative search of the combinatorial space of all theory O(n^n^...). The way I think about issues like this is greatly informed by my study of learning theory, which if studied deeply infuses one with a great phobia of the dreaded local maximum and a suspicion of any and all deterministic non-adaptive learning strategies... As these tend to converge at local maxima. See: no free lunch theorem, among other things. My maze solver analogy is a pretty good simple one for how the space of learning algorithms maps to the space of fitness landscapes. The ideal strategy would probably be to run multiple heuristics concurrently or switch them up over time. For example, science might embark upon multi decade investigations of orthodoxy punctuated by jubilees where an accumulated queue of minority and fringe theories are investigated. Successes here form the next set of orthodoxies, rinse and repeat.