12 ms·

Fund ideas, not pedigree, to find fresh insight

263 points by seagullz 9y ago

dalbasal 9y ago
I think people are hesitant to over-criticise academia. Maybe they can already hear the resulting anti-science sentiment, and fear losing hard fought ground if academia is called into question. However, I think there are serious signs that we should be looking for radical changes to the metstructures of academia. They're not necessarily related, but they point to something being really wrong. First is the replicability crisis. Have we actually been accumulating knowledge in fields where this is bad? Do we know more about human behaviour due to the human behaviour research done in the last generation? What's the point of all these "more research needed" conclusions. The process of academic publishing and peer reveiw directs this whole thing, and it doesn't seem to be all that well directed.. While we're on publishing, everyone in academia complains bitterly about the publish-or-perish problem and about grant/funding politics. One of the big complaints about grants is that senior academics spend all their time on grants and administration. Success in many fields is more a function of being good on administration and politics, than being good at research. While we're on administration... the number of administrators (not teachers or researchers) has skyrocketed over the last 2 generations. Ultimately, I think that the "scientific method" in practice is in large part embodied the academic publishing system. ...the algorithm determining how science works.
- wgerard 9y ago
  > I think people are hesitant to over-criticise academia. Really? Maybe inside academia itself that's true, but I think people outside are fairly eager to criticize academia and put down academics. I feel like anytime a study with even a slightly intriguing/controversial abstract gets circulated, one of the following happens (in order of frequency): 1) Narrow criticism of something very specific (usually sample sizes) with very little understanding about why that criticism might be misinformed (again, usually relating to sample sizes and how statistical significance works) which they use to discredit the whole paper 2) Pointing to previous papers that may (but often don't) contradict what the paper in question is suggesting - and "suggesting" is a loose term here since people tend to draw all sorts of conclusions that the author often doesn't. This is then used to basically throw hands up and say "it's unknowable." 3) Just plain dismissal, especially when the paper's implication is something that the reader doesn't want to hear. This comes in all forms, from the pedigree-based "oh who cares what someone from Montana State says what do they know" to the more direct ones like "why the hell would I listen to Bernanke talk about business, he's an ivory tower economist"
  - dalbasal 9y ago
    I guess hesitance is a relative :) In any case, I hesitate a little, for the above reasons. Criticizing papers is..erm.. good and proper, I think. Sometimes the criticism will be crap, but a critical instinct isn't a bad thing here. Bias is, c'est la vie. Critising academics... I feel we're far too comfortable dishing out personal criticism in the internet era. Leave that one there. Criticising the system... All those "flags" that I raised are actually common and recurring criticisms of the system, within academia. But, they seem to fall short on how meta they go. Publish-or-perish & grant systems, the insider criticism sounds more like "my boss is a moron" talk, cheap shots. What I'm hoping for is big ideas from academia, I guess. Ideas about how to structure the institutions of academia: publishing, grants, whatever else determines what research gets done, by whom. Something that promotes efficiency.^ Something that promotes completion/conclusion so that we end up with knowledge, not a collection of research findings. Negative results and data pooling. Seperation of data & interpretation. Replication. This is not an excel-vs-R problem. It's an "incentives-within-academia" issue. ^Remember, there're problems that these grant bureaucracies are clumsily trying to solve. This is a market, and a bad market can be a lot worse than a good one.
  - castle-bravo 9y ago
    I think there are two main kinds of criticism here; the first comes from people who think a lot of published work is misleading at best and fraudulent at worst (see replicability crisis); the second comes from people who think there's something horribly wrong with the culture of academia, notwithstanding the quality of output. Criticism in the second category touches on topics like adjunct professors (most instructors of undergraduate courses are temp workers making near minimum wage with no benefits), the takeover of universities by a class of professional administrators (many of whom have unbelievably cushy jobs), the distribution of research grants to researchers with the most political pull rather than those doing the most interesting work, the pressure put on researchers to acquire grants (see Stephan Grimm), and the crushing debt burden most undergraduates take on to get a university degree (most of which is spent frivolously while instructors and researchers work themselves to death). I think what you've described sits more in the first category than the second, while the parent belongs more to the second.
- m-watson 9y ago
  I think it is reasonable to question academia and research but I think a problem, often, is people want science and research to give us answers. It does not, it provides evidence, and once that evidence gets supported enough times it is viewed as closer and closer to reality. I would say it is asymptotic though, we never reach what is fact because there can always be a missing piece of evidence that shows a different result. There is one statement you make that I do want to directly comment on. The point of more research is needed is simply, that more research is needed. It goes directly to your statement of "do we know more about human behavior due to the ..." We learn more with all research and the way research currently gets disseminated is through publication. More research is needed publications as well as null result research is extremely useful in the research community and often does not get the attention outside (or inside sadly) the scientific community that it deserves.
  - dalbasal 9y ago
    I'm not arguing at the philosophical level. My views are fairly mundane on that. I'm talking about the implementation of that philosophy (where I agree with you) in academia, as a system. When I say "build knowledge", I'm speaking colloquially. For a contrived example, we know traits are inherited by animals via genetics. We're confident enough to treat this as knowledge and design other expirements that take this as an assumption. Are we building this kind of knowledge in social science, currently? Does "more research needed" lead to more research being done, typically in a field? If not, why? Why are we moving on to a new question, when the last one is still unanswered. Take a "willpower is a limited resource" hypothesis (replication issues aside). It does not seem impossible to answer this question. Nothing in science is "knowledge" in an absolute sense, but we can get to some level of confidence. So.. do we want to know the answer to this question? If not, why do the the first expirement? If yes, then what would it take to get to an answer? We can answer that partially in advance. Independant replication will be part of the answer. Sufficient sample size also. If we know in advance that these things won't be done, what is the point? I'm an outsider and I realize I may be throwing unwarranted dirt, so if/where I am, apologies. I think that in some fields we systematically do not do the things necessary to get to useful knowledge, a sufficient accumulation of evidence to treat a result as more than anecdote.
- Vinnl 9y ago
  I've been working full time on trying to help solve one of the problems in academia (access to research), and although I was superficially aware of most of the problems in academia, once you start to dive into one you will learn a lot about the others as well, as they're somewhat intertwined. And really, the scope and breadth of the problems is astonishing. Personally, my hunch is that the root cause for most of them is a somewhat misguided focus on "excellence" combined with it being practically impossible to measure researcher performance with a reasonable amount of effort. Then again, the problems are so intertwined and have so many different actors, that finding the root cause is really hard, so who knows whether I'm correct.
  - dalbasal 9y ago
    I think measuring research performance is a dead end. People can do this, although they carry a ton of blindness and blindspots. Bureaucracies, systems and such can't. They end up measuring what they can measure, like grades, attendance, number of published papers, number of citations maybe... It just won't end well. But, we can have systems that promote performance without measuring it. I think those are the kinds of ideas we should look at.
    - Vinnl 9y ago
      I agree with your first paragraph, but I don't see how your second paragraph would work. Do you have any concrete examples/ideas of how such a system could work? (Without requiring e.g. those on tenure committees to read and understand all the research of all applications - unless you believe that could work somehow.)
      - dalbasal 9y ago
        Nothing specific. I'm also not an academic, so anything I say will be a little contrived but... ideas generally... I think the first step is look outside the basic, low-level structure and look a level or two above it. I think the best ideas will likely touch on a few of the related problems and happen outside of individual departments. Either the publication or the funding structures seem like good places to start. Publishing... This is obviously a centrepiece institution. It's central to replication/quality issues, to publish-or-perish career incentives and to funding allocation. So, I guess any radical ideas here will likely impact on a lot. Can we modernize publication, make it more open, and update the peer review system? One step is to move away from seeing "peer reviewed" as a binary. Another might be increasing the role of peer review. Maybe papers should have critical "reviewer notes." Maybe publications could take more of a 'director' role, influencing follow up, replications, being critical of practices... ..If a paper concludes that "more research is needed," does this mean anything? Will the research be done? What exactly needs to be done (eg, same study @ 5X the sample size). Is this likely to yield a result? Is that result important? A big part of the replication problem (from my perspective, safely on my couch) seems to be that very little replication is done, because the incentives are all wrong. If institutions were different (eg capitalism, for example) I could imagine the problem being reversed: replication and further work could be overinvested in, impoverishing exploratory science instead of the other way around. Replication is a lot more concrete (and money men love concrete).. you know what you need to do specifically, and what you might achieve by doing it. The problem in academia is that original work, even unreplicated an inconclusive is valued higher than subsequent work. We have to incentivize replication. In social sciences, we're sitting on a mountain of inconclusive results. Unless this work is done, we can't build on. Funding... Measuring researcher output well enough to base funding on is not possible, imo. But, maybe we'd have better luck with portfolios. Ie, structure funding bodies as competing portfolio managers, looking to impress based on portfolio performance. This might give us a better mix of predictable returns (eg replicate these 175 studies) and long tail risky stuff.
- api 9y ago
  Treating science like a religion is also anti-science.
- losvedir 9y ago
  I think this comment is spot on. In particular, > I think people are hesitant to over-criticise academia. is a message to people here. Of course, there's all the "anti-global warming, intelligent design flat earther republicans" who have always criticized academia, but that's not what you're talking about (I think). I think you mean that people here - i.e. relatively educated, orthodox liberals for the most part - should really start to look hard at academia. I'm someone with a traditional academic pedigree (double major in Econ + Math from MIT, most of my friends have PhDs) but I, too, am growing increasingly wary of much published research. For example, there's this eye-opening blog post[0] by Andrew Gelman, a professor of statistics at Columbia who collaborates on a lot of psychology research, about the problem of bad statistics in research in psychology. There's this [1] famous paper by the Fed about problems of replication in what I studied, economics. Or how the NIMH "is re-orienting its research away from DSM categories".[2] And then there's the endless pop-science ideas that show up and get shot down like Power Pose, Air Rage, or Learning Styles. Or the constantly changing recommendations on nutrition. I don't really take for granted anymore that academia is generally working towards greater understanding of the world. It's an imperfect system that has done some incredible things, but it definitely is dealing with some serious issues at the moment. I hope that initiatives like journals for null results, changes in the way grants are apportioned (like in this posted article), etc, can put it back on track. [0] http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he... [1] https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf https://www.federalreserve.gov/econresdata/feds/2015/files/2... [2] https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2013/transforming-diagnosis.shtml https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2...
yetihehe 9y ago
The biggest problem with funding ideas - they fail more than proven conservative projects. When you select idea that fails, you can be accused of wasting funds, and not selected for next funding meeting.
- microtherion 9y ago
  It seems from an article that the funding agencies were pleased with the return they got from an "ideas" portfolio. Maybe more ideas failed, but the projects that succeeded did so in more spectacular ways — a bit like a VC portfolio. The problem, it seems to me, is similar to VC: The funding agencies can spread their risk across a pool of ideas, but the investigators generally have to commit to one or a small number of projects for some time, so they risk spending several years without much to show for it if the idea fails.
danieltillett 9y ago
Peer review is not about funding the best science, it is about funding the best connected and ensuring those that have climbed to the top keep their position even if they are out of ideas. More seriously, the problem with approaches like this is the people who are chosen would be mad to actually work on the idea they put up. At most the funding is for 18 month and what happens if at the end of that time you have nothing? If you want this idea to work the funding has to be for at least 10 years so that people can afford to take real risk and have enough time to build a track record if the idea doesn't pan out.
- cassowary37 9y ago
  not sure this is entirely fair. While there's certainly some cronyism (see my earlier comment), study sections are mostly about funding the safest science - i.e., the project with the least chance of failure and the fewest flaws. But they don't have to be. Many of the 'big idea'-focused mechanisms at NIH are 4 or 5 years - plenty of time to make real headway. These would work if we supported enough of them. Parenthetically, high-risk science does not mean shooting the moon - a well-designed project, even if it's high risk, will still yield /something/ valuable both for the field, and for the investigator's career development. That's why reviewers tend to look closely at the alternatives/pitfalls section of grants - what will you do if the experiment blows up?
  - danieltillett 9y ago
    I am somewhat biased by having been through the Australian system where cronyism is alive and well. If you have 4 or 5 years you really only have 2 years before you have to switch to low risk if things aren't working out to give yourself time to get enough papers out before you need to apply for the next grant. What I did when I was an academic was spend 70% of my and my students/postdocs time on low risk activities and 30% on high risk. This keeps the risk down and still allow for some dreams.
Torai 9y ago
- I started defending the value of good ideas 10 years ago in Quora. I debated with venture capitalist and entrepreneurs. They took me as a fool. - I debated about the value of ideas in HN. They took me as a fool. "Ideas are worthless, Execution is king". Now, I have a brain used to find good ideas quickly. My fuzzy process of analizing something complex is well advanced. And no venture capitalist, coder or entrepreneur will be able to compete with me cause I have years of advantage. I've also trained emotional resilience and self-confidence cause I knew I was right even when the successful people didn't agree. I don't even need to read the article to comment on this. Ideas are my area of expertise. I spent 7 years without working and letting my mind free, thinking about thinking. Hacker News shepple: venture capitalists and entrepreneur sheeple: FUCK OFF!!! Anyway, I know this sheeple will find another way to attract money for projects, cause they say just what others repeat and what people wants to hear. No what people need to hear. Nature or another scientific magazine will publish an article in 2 years time about the power and value of being frank, blunt and honest. And how it helps the mind. Meanwhile, I will be taken again as a fool again. Really: FUCK OFF!!!!!!!!! I will do my fucking projects without any external money.
- RobertRoberts 9y ago
  While I generally agree with your sentiment about building without external money (ie, bootstrapping), you may come across with your points better if you were a little more civilized and decent about it.
- microtherion 9y ago
  I recommend practicing your manic laughter, it should go well with that kind of mindset.
  - lostcolony 9y ago
    It really does read that way. 'They called me mad! MAD! Well I'll show them! I'll SHOW THEM ALL!'
    - arethuza 9y ago
cassowary37 9y ago
Over the past decade I've sat on 10+ NIH study sections, and chaired a few. This commentary is spot on. Consider the response to a flaw in a grant. For a newer investigator or one no one has heard of, there will be much hemming and hawing about concern about ability to execute the project. This is code for, "who is this person in our sandbox anyway?" For one of us older guys (yes, unfortunately the pronoun is mostly right), the response is, "but I'm sure he can get this done, he must have thought about this." Even a modest scoring 'bonus' for new investigators doesn't correct this bias. On the novelty front, reviewers are trained to focus on finding flaws or absence of detail or preliminary data. The presence of any of these requires that we score down - so it's just about impossible for something truly new to get funded. Until we do the project, we don't necessarily know what problems we'll encounter or what the data will look like. The small number of NIH grants aimed at funding high-risk work by earlier-career folks (eg, the DP2 mechanism) are a great development but a tiny fraction of the portfolio. The key thing to recognize is that any move to distribute funding towards younger folks or higher-risk work causes the old guard to scream bloody murder - "what do you mean I can't have 5 R01's at a time?". For study sections, I've observed first hand that old habits favoring established investigators and low-risk, incremental work are hard to break.
- sharemywin 9y ago
  could you explain 5 R01's?
  - cassowary37 9y ago
    sure - they're the most common NIH grant mechanism - generally 4 or 5 years @ up to $500k/year (though they can be smaller, which NIH program officers always appreciate). If you plot the number of NIH grants per investigator, there are a small but significant number of investigators out in the right tail with 4+ R01's. In general grants beget grants, such that bigger and better-funded labs are better positioned to get the next grant. Details re R01 mechanism are here [1]. A recent proposal that would cap the number of grants an investigator could lead at one time was shot down by the old guard [2]. NIH continues to struggle for alternatives, because unless the overall budget grows, which it will not in the foreseeable future, any strategy will involve taking from the rich(er) and giving to the poor(er). [1] https://grants.nih.gov/grants/funding/r01.htm https://grants.nih.gov/grants/funding/r01.htm [2] http://www.sciencemag.org/news/2017/06/updated-nih-abandons-controversial-plan-cap-grants-big-labs-creates-new-fund-younger http://www.sciencemag.org/news/2017/06/updated-nih-abandons-...
abhv 9y ago
I have served on dozens of NSF computer science funding panels at the small (500k), medium (1.5m), and large (3m) levels. These review panels are not blind, and decisions are based on (a) what problems are explained in the grant, and (b) prior track record and proven expertise in the area. In my own circumstance and what I know from my colleagues, CS researchers rarely write their best ideas in grants---not because they are afraid the ideas are too bold---but rather because those ideas are often not fully worked out, and nobody wants to just give those away to a review panel full of top-rank scientists who might make connections faster than you! The problem with "blind review" is that project proposals can rarely be anonymous because just explaining the work and citing the relevant prior work leaks a lot about the possible author. So making review blind can often be an advantage to the higher profile researcher and a disadvantage to the capable but slightly less well known one. That said, as my own experiment, during my next NSF review, I am going to tear away the first pages of all the proposals, and make my first pass without knowing the authors to see if it makes a difference.
deleted 9y ago
[deleted]
EGreg 9y ago
I think this goes against the YC thesis.
- david927 9y ago
  And the thesis of almost all of venture capital, all of whom have made a lot of money on the insight and innovation of the past, but which has failed to created any new innovation of any significance. This should be better said as, "Fund fresh insight, to find fresh insight." And while that sounds tautologically obvious, almost no one is doing it.
supernova87a 9y ago
Upvoted the comment that says "I think this goes against YC philosophy". The YC guide explicitly calls out team experience and likelihood to succeed as a criterion. I think there's a fetishization of complete blind objectivity in so many fields right now, which has at the root of it trying to iron out inequality. A laudable goal, but you shouldn't let that ideal completely upend the practical question to yourself of, where am I going to place my bets? An unavoidable fact is that ideas alone are not enough to make some technology or venture successful. Sure, some one-off inventor may have a brilliant idea, but the idea is just like 1% of the problem. If your model is to fund a team to develop an idea, the track record / personalities of the team are quite important. Most people in this world are barely able to get their own lives under control. What's the likelihood that a random person with a great idea can take that to something commercially viable? There's a reason that who it is matters, and unfortunately, that still leads to people being selected who reflect the starting set of less-than-diverse people.
- neuromantik8086 9y ago
  Most researchers applying for NIH grants aren't "random people"; all of them have gotten through PhD programs and potentially have done one or two postdocs, which already indicates that they have sufficient "team experience" and are already quite accomplished in their own right. This article isn't saying that anyone should be able to get an NIH grant, but rather that grants should be awarded less due to politics and more based upon potential merit and the promise of an idea. I.e., This editorial is being published in Nature, not viXra. The central issue is not one of team dynamics, or rooting out inequality on a societal level, but rather about rooting out inequality among equally capable candidates. Currently, many academic fields are dominated by cronyism and that is what the article is advocating against.
- qdoop 9y ago
  YC's philosophy is basically make noise lots of noise and the rest shall follow. They are never going to pay attention to something that cannot generate lots of buzz. I would never expect something fresh or innovative to come out from what we would call "proper" people because these are the people less willing to change and question anything. The problem is much deeper. If I am having good time the bast policy to follow is the one involving fewer changes so I could continue having good time. Change comes mostly from the ones not having good time the "loosers" you might call
mempko 9y ago
This might be a fine way to fund innovation (taking invention and turning it into capital ) but it's a terrible way to fund invention. By definition there is no idea when you set out to invent. innovation produces billions of dollars of wealth . Invention produces trillions. Are we sure we have our priorities straight as a species?
pmiller2 9y ago
When do we get to see the corresponding article for hiring? "Stop hiring exclusively from the same handful of schools"?
mankash666 9y ago
I think this applies to YC-type incubators and VC investment as well. Often times pedigree seems to outweigh fundamentals regarding execution in VC funding
mkollo 9y ago
One other thing worth mentioning is the obsession of funders with hypothesis-driven research. They do this out of defensiveness to justify spending, but in the long term, it reinforces a daisy-chain of stale ideas and a selection bias that distorts data to adhere to popular models.

263 points by seagullz 9y ago

I think people are hesitant to over-criticise academia. Maybe they can already hear the resulting anti-science sentiment, and fear losing hard fought ground if academia is called into question. However, I think there are serious signs that we should be looking for radical changes to the metstructures of academia. They're not necessarily related, but they point to something being really wrong. First is the replicability crisis. Have we actually been accumulating knowledge in fields where this is bad? Do we know more about human behaviour due to the human behaviour research done in the last generation? What's the point of all these "more research needed" conclusions. The process of academic publishing and peer reveiw directs this whole thing, and it doesn't seem to be all that well directed.. While we're on publishing, everyone in academia complains bitterly about the publish-or-perish problem and about grant/funding politics. One of the big complaints about grants is that senior academics spend all their time on grants and administration. Success in many fields is more a function of being good on administration and politics, than being good at research. While we're on administration... the number of administrators (not teachers or researchers) has skyrocketed over the last 2 generations. Ultimately, I think that the "scientific method" in practice is in large part embodied the academic publishing system. ...the algorithm determining how science works.

wgerard 9y ago
> I think people are hesitant to over-criticise academia. Really? Maybe inside academia itself that's true, but I think people outside are fairly eager to criticize academia and put down academics. I feel like anytime a study with even a slightly intriguing/controversial abstract gets circulated, one of the following happens (in order of frequency): 1) Narrow criticism of something very specific (usually sample sizes) with very little understanding about why that criticism might be misinformed (again, usually relating to sample sizes and how statistical significance works) which they use to discredit the whole paper 2) Pointing to previous papers that may (but often don't) contradict what the paper in question is suggesting - and "suggesting" is a loose term here since people tend to draw all sorts of conclusions that the author often doesn't. This is then used to basically throw hands up and say "it's unknowable." 3) Just plain dismissal, especially when the paper's implication is something that the reader doesn't want to hear. This comes in all forms, from the pedigree-based "oh who cares what someone from Montana State says what do they know" to the more direct ones like "why the hell would I listen to Bernanke talk about business, he's an ivory tower economist"
- dalbasal 9y ago
  I guess hesitance is a relative :) In any case, I hesitate a little, for the above reasons. Criticizing papers is..erm.. good and proper, I think. Sometimes the criticism will be crap, but a critical instinct isn't a bad thing here. Bias is, c'est la vie. Critising academics... I feel we're far too comfortable dishing out personal criticism in the internet era. Leave that one there. Criticising the system... All those "flags" that I raised are actually common and recurring criticisms of the system, within academia. But, they seem to fall short on how meta they go. Publish-or-perish & grant systems, the insider criticism sounds more like "my boss is a moron" talk, cheap shots. What I'm hoping for is big ideas from academia, I guess. Ideas about how to structure the institutions of academia: publishing, grants, whatever else determines what research gets done, by whom. Something that promotes efficiency.^ Something that promotes completion/conclusion so that we end up with knowledge, not a collection of research findings. Negative results and data pooling. Seperation of data & interpretation. Replication. This is not an excel-vs-R problem. It's an "incentives-within-academia" issue. ^Remember, there're problems that these grant bureaucracies are clumsily trying to solve. This is a market, and a bad market can be a lot worse than a good one.
- castle-bravo 9y ago
  I think there are two main kinds of criticism here; the first comes from people who think a lot of published work is misleading at best and fraudulent at worst (see replicability crisis); the second comes from people who think there's something horribly wrong with the culture of academia, notwithstanding the quality of output. Criticism in the second category touches on topics like adjunct professors (most instructors of undergraduate courses are temp workers making near minimum wage with no benefits), the takeover of universities by a class of professional administrators (many of whom have unbelievably cushy jobs), the distribution of research grants to researchers with the most political pull rather than those doing the most interesting work, the pressure put on researchers to acquire grants (see Stephan Grimm), and the crushing debt burden most undergraduates take on to get a university degree (most of which is spent frivolously while instructors and researchers work themselves to death). I think what you've described sits more in the first category than the second, while the parent belongs more to the second.
m-watson 9y ago
I think it is reasonable to question academia and research but I think a problem, often, is people want science and research to give us answers. It does not, it provides evidence, and once that evidence gets supported enough times it is viewed as closer and closer to reality. I would say it is asymptotic though, we never reach what is fact because there can always be a missing piece of evidence that shows a different result. There is one statement you make that I do want to directly comment on. The point of more research is needed is simply, that more research is needed. It goes directly to your statement of "do we know more about human behavior due to the ..." We learn more with all research and the way research currently gets disseminated is through publication. More research is needed publications as well as null result research is extremely useful in the research community and often does not get the attention outside (or inside sadly) the scientific community that it deserves.
- dalbasal 9y ago
  I'm not arguing at the philosophical level. My views are fairly mundane on that. I'm talking about the implementation of that philosophy (where I agree with you) in academia, as a system. When I say "build knowledge", I'm speaking colloquially. For a contrived example, we know traits are inherited by animals via genetics. We're confident enough to treat this as knowledge and design other expirements that take this as an assumption. Are we building this kind of knowledge in social science, currently? Does "more research needed" lead to more research being done, typically in a field? If not, why? Why are we moving on to a new question, when the last one is still unanswered. Take a "willpower is a limited resource" hypothesis (replication issues aside). It does not seem impossible to answer this question. Nothing in science is "knowledge" in an absolute sense, but we can get to some level of confidence. So.. do we want to know the answer to this question? If not, why do the the first expirement? If yes, then what would it take to get to an answer? We can answer that partially in advance. Independant replication will be part of the answer. Sufficient sample size also. If we know in advance that these things won't be done, what is the point? I'm an outsider and I realize I may be throwing unwarranted dirt, so if/where I am, apologies. I think that in some fields we systematically do not do the things necessary to get to useful knowledge, a sufficient accumulation of evidence to treat a result as more than anecdote.
Vinnl 9y ago
I've been working full time on trying to help solve one of the problems in academia (access to research), and although I was superficially aware of most of the problems in academia, once you start to dive into one you will learn a lot about the others as well, as they're somewhat intertwined. And really, the scope and breadth of the problems is astonishing. Personally, my hunch is that the root cause for most of them is a somewhat misguided focus on "excellence" combined with it being practically impossible to measure researcher performance with a reasonable amount of effort. Then again, the problems are so intertwined and have so many different actors, that finding the root cause is really hard, so who knows whether I'm correct.
- dalbasal 9y ago
  I think measuring research performance is a dead end. People can do this, although they carry a ton of blindness and blindspots. Bureaucracies, systems and such can't. They end up measuring what they can measure, like grades, attendance, number of published papers, number of citations maybe... It just won't end well. But, we can have systems that promote performance without measuring it. I think those are the kinds of ideas we should look at.
  - Vinnl 9y ago
    I agree with your first paragraph, but I don't see how your second paragraph would work. Do you have any concrete examples/ideas of how such a system could work? (Without requiring e.g. those on tenure committees to read and understand all the research of all applications - unless you believe that could work somehow.)
    - dalbasal 9y ago
      Nothing specific. I'm also not an academic, so anything I say will be a little contrived but... ideas generally... I think the first step is look outside the basic, low-level structure and look a level or two above it. I think the best ideas will likely touch on a few of the related problems and happen outside of individual departments. Either the publication or the funding structures seem like good places to start. Publishing... This is obviously a centrepiece institution. It's central to replication/quality issues, to publish-or-perish career incentives and to funding allocation. So, I guess any radical ideas here will likely impact on a lot. Can we modernize publication, make it more open, and update the peer review system? One step is to move away from seeing "peer reviewed" as a binary. Another might be increasing the role of peer review. Maybe papers should have critical "reviewer notes." Maybe publications could take more of a 'director' role, influencing follow up, replications, being critical of practices... ..If a paper concludes that "more research is needed," does this mean anything? Will the research be done? What exactly needs to be done (eg, same study @ 5X the sample size). Is this likely to yield a result? Is that result important? A big part of the replication problem (from my perspective, safely on my couch) seems to be that very little replication is done, because the incentives are all wrong. If institutions were different (eg capitalism, for example) I could imagine the problem being reversed: replication and further work could be overinvested in, impoverishing exploratory science instead of the other way around. Replication is a lot more concrete (and money men love concrete).. you know what you need to do specifically, and what you might achieve by doing it. The problem in academia is that original work, even unreplicated an inconclusive is valued higher than subsequent work. We have to incentivize replication. In social sciences, we're sitting on a mountain of inconclusive results. Unless this work is done, we can't build on. Funding... Measuring researcher output well enough to base funding on is not possible, imo. But, maybe we'd have better luck with portfolios. Ie, structure funding bodies as competing portfolio managers, looking to impress based on portfolio performance. This might give us a better mix of predictable returns (eg replicate these 175 studies) and long tail risky stuff.
api 9y ago
Treating science like a religion is also anti-science.
losvedir 9y ago
I think this comment is spot on. In particular, > I think people are hesitant to over-criticise academia. is a message to people here. Of course, there's all the "anti-global warming, intelligent design flat earther republicans" who have always criticized academia, but that's not what you're talking about (I think). I think you mean that people here - i.e. relatively educated, orthodox liberals for the most part - should really start to look hard at academia. I'm someone with a traditional academic pedigree (double major in Econ + Math from MIT, most of my friends have PhDs) but I, too, am growing increasingly wary of much published research. For example, there's this eye-opening blog post[0] by Andrew Gelman, a professor of statistics at Columbia who collaborates on a lot of psychology research, about the problem of bad statistics in research in psychology. There's this [1] famous paper by the Fed about problems of replication in what I studied, economics. Or how the NIMH "is re-orienting its research away from DSM categories".[2] And then there's the endless pop-science ideas that show up and get shot down like Power Pose, Air Rage, or Learning Styles. Or the constantly changing recommendations on nutrition. I don't really take for granted anymore that academia is generally working towards greater understanding of the world. It's an imperfect system that has done some incredible things, but it definitely is dealing with some serious issues at the moment. I hope that initiatives like journals for null results, changes in the way grants are apportioned (like in this posted article), etc, can put it back on track. [0] http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ http://andrewgelman.com/2016/09/21/what-has-happened-down-he... [1] https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf https://www.federalreserve.gov/econresdata/feds/2015/files/2... [2] https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2013/transforming-diagnosis.shtml https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2...

yetihehe 9y ago

The biggest problem with funding ideas - they fail more than proven conservative projects. When you select idea that fails, you can be accused of wasting funds, and not selected for next funding meeting.

microtherion 9y ago
It seems from an article that the funding agencies were pleased with the return they got from an "ideas" portfolio. Maybe more ideas failed, but the projects that succeeded did so in more spectacular ways — a bit like a VC portfolio. The problem, it seems to me, is similar to VC: The funding agencies can spread their risk across a pool of ideas, but the investigators generally have to commit to one or a small number of projects for some time, so they risk spending several years without much to show for it if the idea fails.

danieltillett 9y ago

Peer review is not about funding the best science, it is about funding the best connected and ensuring those that have climbed to the top keep their position even if they are out of ideas. More seriously, the problem with approaches like this is the people who are chosen would be mad to actually work on the idea they put up. At most the funding is for 18 month and what happens if at the end of that time you have nothing? If you want this idea to work the funding has to be for at least 10 years so that people can afford to take real risk and have enough time to build a track record if the idea doesn't pan out.

cassowary37 9y ago
not sure this is entirely fair. While there's certainly some cronyism (see my earlier comment), study sections are mostly about funding the safest science - i.e., the project with the least chance of failure and the fewest flaws. But they don't have to be. Many of the 'big idea'-focused mechanisms at NIH are 4 or 5 years - plenty of time to make real headway. These would work if we supported enough of them. Parenthetically, high-risk science does not mean shooting the moon - a well-designed project, even if it's high risk, will still yield /something/ valuable both for the field, and for the investigator's career development. That's why reviewers tend to look closely at the alternatives/pitfalls section of grants - what will you do if the experiment blows up?
- danieltillett 9y ago
  I am somewhat biased by having been through the Australian system where cronyism is alive and well. If you have 4 or 5 years you really only have 2 years before you have to switch to low risk if things aren't working out to give yourself time to get enough papers out before you need to apply for the next grant. What I did when I was an academic was spend 70% of my and my students/postdocs time on low risk activities and 30% on high risk. This keeps the risk down and still allow for some dreams.

Torai 9y ago

- I started defending the value of good ideas 10 years ago in Quora. I debated with venture capitalist and entrepreneurs. They took me as a fool. - I debated about the value of ideas in HN. They took me as a fool. "Ideas are worthless, Execution is king". Now, I have a brain used to find good ideas quickly. My fuzzy process of analizing something complex is well advanced. And no venture capitalist, coder or entrepreneur will be able to compete with me cause I have years of advantage. I've also trained emotional resilience and self-confidence cause I knew I was right even when the successful people didn't agree. I don't even need to read the article to comment on this. Ideas are my area of expertise. I spent 7 years without working and letting my mind free, thinking about thinking. Hacker News shepple: venture capitalists and entrepreneur sheeple: FUCK OFF!!! Anyway, I know this sheeple will find another way to attract money for projects, cause they say just what others repeat and what people wants to hear. No what people need to hear. Nature or another scientific magazine will publish an article in 2 years time about the power and value of being frank, blunt and honest. And how it helps the mind. Meanwhile, I will be taken again as a fool again. Really: FUCK OFF!!!!!!!!! I will do my fucking projects without any external money.

RobertRoberts 9y ago
While I generally agree with your sentiment about building without external money (ie, bootstrapping), you may come across with your points better if you were a little more civilized and decent about it.
microtherion 9y ago
I recommend practicing your manic laughter, it should go well with that kind of mindset.
- lostcolony 9y ago
  It really does read that way. 'They called me mad! MAD! Well I'll show them! I'll SHOW THEM ALL!'
  - arethuza 9y ago

cassowary37 9y ago

Over the past decade I've sat on 10+ NIH study sections, and chaired a few. This commentary is spot on. Consider the response to a flaw in a grant. For a newer investigator or one no one has heard of, there will be much hemming and hawing about concern about ability to execute the project. This is code for, "who is this person in our sandbox anyway?" For one of us older guys (yes, unfortunately the pronoun is mostly right), the response is, "but I'm sure he can get this done, he must have thought about this." Even a modest scoring 'bonus' for new investigators doesn't correct this bias. On the novelty front, reviewers are trained to focus on finding flaws or absence of detail or preliminary data. The presence of any of these requires that we score down - so it's just about impossible for something truly new to get funded. Until we do the project, we don't necessarily know what problems we'll encounter or what the data will look like. The small number of NIH grants aimed at funding high-risk work by earlier-career folks (eg, the DP2 mechanism) are a great development but a tiny fraction of the portfolio. The key thing to recognize is that any move to distribute funding towards younger folks or higher-risk work causes the old guard to scream bloody murder - "what do you mean I can't have 5 R01's at a time?". For study sections, I've observed first hand that old habits favoring established investigators and low-risk, incremental work are hard to break.

sharemywin 9y ago
could you explain 5 R01's?
- cassowary37 9y ago
  sure - they're the most common NIH grant mechanism - generally 4 or 5 years @ up to $500k/year (though they can be smaller, which NIH program officers always appreciate). If you plot the number of NIH grants per investigator, there are a small but significant number of investigators out in the right tail with 4+ R01's. In general grants beget grants, such that bigger and better-funded labs are better positioned to get the next grant. Details re R01 mechanism are here [1]. A recent proposal that would cap the number of grants an investigator could lead at one time was shot down by the old guard [2]. NIH continues to struggle for alternatives, because unless the overall budget grows, which it will not in the foreseeable future, any strategy will involve taking from the rich(er) and giving to the poor(er). [1] https://grants.nih.gov/grants/funding/r01.htm https://grants.nih.gov/grants/funding/r01.htm [2] http://www.sciencemag.org/news/2017/06/updated-nih-abandons-controversial-plan-cap-grants-big-labs-creates-new-fund-younger http://www.sciencemag.org/news/2017/06/updated-nih-abandons-...

abhv 9y ago

I have served on dozens of NSF computer science funding panels at the small (500k), medium (1.5m), and large (3m) levels. These review panels are not blind, and decisions are based on (a) what problems are explained in the grant, and (b) prior track record and proven expertise in the area. In my own circumstance and what I know from my colleagues, CS researchers rarely write their best ideas in grants---not because they are afraid the ideas are too bold---but rather because those ideas are often not fully worked out, and nobody wants to just give those away to a review panel full of top-rank scientists who might make connections faster than you! The problem with "blind review" is that project proposals can rarely be anonymous because just explaining the work and citing the relevant prior work leaks a lot about the possible author. So making review blind can often be an advantage to the higher profile researcher and a disadvantage to the capable but slightly less well known one. That said, as my own experiment, during my next NSF review, I am going to tear away the first pages of all the proposals, and make my first pass without knowing the authors to see if it makes a difference.

deleted 9y ago

[deleted]

EGreg 9y ago

I think this goes against the YC thesis.

david927 9y ago
And the thesis of almost all of venture capital, all of whom have made a lot of money on the insight and innovation of the past, but which has failed to created any new innovation of any significance. This should be better said as, "Fund fresh insight, to find fresh insight." And while that sounds tautologically obvious, almost no one is doing it.

supernova87a 9y ago

Upvoted the comment that says "I think this goes against YC philosophy". The YC guide explicitly calls out team experience and likelihood to succeed as a criterion. I think there's a fetishization of complete blind objectivity in so many fields right now, which has at the root of it trying to iron out inequality. A laudable goal, but you shouldn't let that ideal completely upend the practical question to yourself of, where am I going to place my bets? An unavoidable fact is that ideas alone are not enough to make some technology or venture successful. Sure, some one-off inventor may have a brilliant idea, but the idea is just like 1% of the problem. If your model is to fund a team to develop an idea, the track record / personalities of the team are quite important. Most people in this world are barely able to get their own lives under control. What's the likelihood that a random person with a great idea can take that to something commercially viable? There's a reason that who it is matters, and unfortunately, that still leads to people being selected who reflect the starting set of less-than-diverse people.

neuromantik8086 9y ago
Most researchers applying for NIH grants aren't "random people"; all of them have gotten through PhD programs and potentially have done one or two postdocs, which already indicates that they have sufficient "team experience" and are already quite accomplished in their own right. This article isn't saying that anyone should be able to get an NIH grant, but rather that grants should be awarded less due to politics and more based upon potential merit and the promise of an idea. I.e., This editorial is being published in Nature, not viXra. The central issue is not one of team dynamics, or rooting out inequality on a societal level, but rather about rooting out inequality among equally capable candidates. Currently, many academic fields are dominated by cronyism and that is what the article is advocating against.
qdoop 9y ago
YC's philosophy is basically make noise lots of noise and the rest shall follow. They are never going to pay attention to something that cannot generate lots of buzz. I would never expect something fresh or innovative to come out from what we would call "proper" people because these are the people less willing to change and question anything. The problem is much deeper. If I am having good time the bast policy to follow is the one involving fewer changes so I could continue having good time. Change comes mostly from the ones not having good time the "loosers" you might call

mempko 9y ago

This might be a fine way to fund innovation (taking invention and turning it into capital ) but it's a terrible way to fund invention. By definition there is no idea when you set out to invent. innovation produces billions of dollars of wealth . Invention produces trillions. Are we sure we have our priorities straight as a species?

pmiller2 9y ago

When do we get to see the corresponding article for hiring? "Stop hiring exclusively from the same handful of schools"?

mankash666 9y ago

I think this applies to YC-type incubators and VC investment as well. Often times pedigree seems to outweigh fundamentals regarding execution in VC funding

mkollo 9y ago

One other thing worth mentioning is the obsession of funders with hypothesis-driven research. They do this out of defensiveness to justify spending, but in the long term, it reinforces a daisy-chain of stale ideas and a selection bias that distorts data to adhere to popular models.