7 ms·

I hadn't even dared to think that Sturgeon's Law might apply to output in the field of science. Yikes. "Anyway, science right now is in an existential crisis.

I hadn't even dared to think that Sturgeon's Law might apply to output in the field of science. Yikes. "Anyway, science right now is in an existential crisis. This is a real, real issue, and there’s now a generation of scientists who specialize in pointing this out and analyzing it. Andrew Gelman and others. I’ll give you an example: I had a conversation with the long-time head of one of the big federal funding agencies for healthcare research who is also a very accomplished entrepreneur, and I said, “do you really think it’s true that 50-70% of biomedical research is fake?” This is a guy who has spent his life in this world. And he said “oh no, that’s not true at all. It’s 90%.” [Richard laughs]. I was like “holy shit,” I was flabbergasted that it could be 90%. He’s like “well look, 90% of everything is shit”, which is literally this thing called Sturgeon’s law which says that 90% of everything is bad. 90% of every novel written is bad, 90% of music, 90% of art… 90% of everything is bad. So his analysis was, anything you get in the field of medical experimentation, biomedical development, and this is going to be true of any field, there’s like five labs total in the world that are really good at what they’re doing and doing really cutting edge work. And this is true of quantum computing; pick any field for advanced technology you want. So those five labs have a pretty good shot at doing interesting work, even some of that is going to reproduce and some of it isn’t. But once you get out of those top five labs, it’s pretty much make-work, incremental, marginal improvements at best, and a complete waste of time otherwise. And I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Anyway, a longwinded way of saying that we have pretty serious structural and incentive problems in the research complex. "

satellite2 5y ago
As if we are capable of determining who is doing the top 10% best science and are not glued to the 10% most shining project according to the fad of the day.
dotcommand 5y ago
> He’s like “well look, 90% of everything is shit” I think there is a difference between "being shit" and "being fake" though. 'do you really think it’s true that 50-70% of biomedical research is fake?' Fake seems more insidious and problematic. Also this brings up a very troubling issue. If 90% of the "research is fake/shit" and lets say that means that 90% of the researchers are "fake/shit", then it means that 'scientific' consensus is also likely "fake/shit". This is why I'm always skeptical of 'scientific' consensus. Science is about evidence, testing, etc. Not consensus - which is the realm of politics, law, etc. And I'd suspect 99.99% of social 'sciences' is probably 'fake/shit' and that is used to push/change society/government/etc.
- techfoolery 5y ago
  I think it's irrelevant to what was being said in their conversation. In Marc's context, being fake/shit equated to non-replicable simply. Before that quote, I believe he was talking about the replication crisis briefly.
petermcneeley 5y ago
https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law This law seems to lack any kind of rigor or theoretical basis and seems like a purely rhetorical device.
- ysavir 5y ago
  As are most of these "laws". The idea isn't to do a conclusive, thorough examination of all things under the sun and render a statement that reflects that particular moment in time. The idea is to make an observation--perhaps true, perhaps false--that helps us maintain a healthy mindset when examining and comprehending things. It's not meant to be insightful, it's meant to encourage us to strive for better insight.
  - vickspicks 5y ago
    90% of laws are bad
    - capiki 5y ago
      What law is that?
      - ysavir 5y ago
        Technically, Sturgeon's law.
    - breck 5y ago
      84.3% of statistics are made up on the spot
  - blueboo 5y ago
    OK, but let’s be clear: it’s not rational. It’s “thoughts and prayers” for the Star Slate Codex set.
    - ysavir 5y ago
nonameiguess 5y ago
Not only this, but it appears to me that diagnoses of the state of science as a problem are making the classic problem a person like Andrew Gelman would usually quickly point out, which is ignoring base rates. Saying 70% of published research can't replicate sounds bad in a vacuum, but how does that compare to 50 years ago? We have no idea if this number that sounds terrible outside of any context represents an improvement, degradation, or no difference at all from what we should expect. That isn't by any means to suggest we shouldn't try to make science better by being more replicable and reducing fraud, but if it is an improvement, we're already doing something right and there is far less cause for despair and cynicism. It also, of course, doesn't do any base rate comparison to non-scientific ways of generating facts. If 90% of published psychology research turns out to be wrong, but 99% of folk psychology turns out to be wrong, then science is still winning there.
- jollybean 5y ago
  "Saying 70% of published research can't replicate sounds bad in a vacuum, but how does that compare to 50 years ago?" It doesn't matter a whole lot with respect to 'how it compares to 50 years ago' if the assumption that science is supposed to be objective and credible. It's supposed to be 'approaching 100%' i.e. they are publishing findings, not fake findings. i.e. it's not a game of 'getting a higher percentage' it's a game of integrity. The implications that 'most science is rubbish' is pretty scary actually and it has profound consequences for the future both in practical terms (i.e. we can't trust a lot of it), but especially in populist terms (i.e. people not trusting climate change, vaccine research etc.). "If 90% of published psychology research turns out to be wrong, but 99% of folk psychology turns out to be wrong, then science is still winning there. " Is it though? We spend bazillions on psychology and nothing on pop psychology, so we'd have to contemplate how valuable that '10%' is to society - especially in the context that we may not even know 'which 10%'. This is kind of a big problem.
  - majormajor 5y ago
    There's a big, important difference between "fake fraudulent findings" and "not-replicable findings." The latter suggests problems stemming from a big nasty complicated poorly-understood world, and can be improved with introspection and change over time. The first step is the greatly increased discussion over it now than in the past - we're discovering the size of the problem. The next step will be to do things with that. Publishing negative results, getting everyone to have a better understanding of prob/stat... I think those are the directions to go in. If, on the other hand, you just dumped all those psychology researchers out of the academic environment and sent them to marketing and advertising companies... are we better off?
ren_engineer 5y ago
it's all about incentives, the modern academic system has the incentives all wrong for creating innovation and quality work. Replication crisis is just a symptom of the real problem
u385639 5y ago
Interesting. Marc is a nuanced guy of course but I sense he may have shifted in the last while. I recall in his 1:1 w/ Peter Thiel he took the opposing side of this argument. Peter does not mince words re: the progress of "science". Perhaps Marc was just doing it for the sake of discussion.
j7ake 5y ago
Yes, but this sturgeons law (maybe less extreme extent) often applies to within labs or within scientists as well. Newton did first rate work, but he also spent a lot of time working on alchemy and theology which didn’t get anywhere. Research grants are generally competitive, if you haven’t shown you’ve ever done something my significant or haven’t done something significant in a long time, it is difficult to survive. But indeed even the top scientists spend a lot of time on dead ends. It is true that often only top 5 labs or so doing good work, but those top 5 labs are not constant over time, they wax and wane like fashion trends. The lesson is just because over last few years the top work came from only a few labs, it’s premature to eliminate all other labs because those top labs could be lower in the next cycle.
- lisper 5y ago
  It's also important to distinguish between things that turn out to be crap in retrospect and things that can be known to be crap given the information available at the time. Those are two very different things. Yes, alchemy is crap, but there's no way Newton could have known that given the data he had at his disposal at the time.
  - WalterBright 5y ago
    > things that can be known to be crap given the information available at the time Sometimes people do investigate the crap, and discover it's actually fertilizer. The trouble is, one cannot tell in advance.
  - lambdatronics 5y ago
    You understate your point. Many advances were 'known' in advance to be crap.
- tmn 5y ago
  It’s somewhat laughable to say he worked on alchemy and theology and that got no where, while we are discussing him and his accomplishments 400 years later. Alchemy is an esoteric inner journey. Ramanujan was by all accounts a one of a kind genius. Just from wiki ‘Ramanujan credited his substantial mathematical capacities to divinity, and said the mathematical knowledge he displayed was revealed to him by his family goddess Namagiri Thayer’. Belief systems can be very powerful or destructive to ones own accomplishments
version_five 5y ago
I don't see a better system. Maybe I'm misunderstanding his or your point. To me this is essentially capitalism. 90% or more of businesses fail, and of those that don't, 90% or more suck. But if you tried to call that 99% "waste" that could be avoided by only running the good business, you would fail miserably. Research is exactly the same. We can't just identify and fund the good research. (But this should be obvious to a VC so I may be reading it wrong)
- yuy910616 5y ago
  Well said - Perhaps a way to look at it is that 'good' is a subjective target that should be define as 'distance from average/medium'. So let's be thankful of the crappy medium haha
- jollybean 5y ago
  This is definitely not the right analogy because one is a matter of risk, the other a matter of integrity. New business ventures are inherently risky. Research is inherently risky, and we would expect 90% to 'fail' but for 90% of papers to be rubbish would be a matter of 'extreme corruption' along the lines of 90% of judges taking bribes etc.. "I don't see a better system." Well, one thing we could do is punish people for knowingly publishing crap results, i.e. have a 'secondary review system' that investigates 'why' results were rubbish, and if there was malfeasance, well, put in place some kind of retribution. We could also change our tune with respect to expectations, and come to 'expect' that the majority of research should publish then 'null' result and that should just be it.
  - yuy910616 5y ago
    I think this is a good point too - venture companies and research SHOULD have two very different distributions
  - nostrademons 5y ago
    Be careful about assuming malice where ignorance exists. It's possible for 90% of papers to be rubbish without any ill intent or dishonesty on the part of the experimenter. They might simply suffer from "works for me" syndrome, as it afflicts software developers. In science, this equates to failing to control for all the possible confounding factors that might affect the result of the experiment. They might have carefully controlled for everything they thought of and gotten methodologically valid results, but then when another experimenter tries to replicate it, they find their experimental environment is different in a way that neither considered important but alters the results. Finding all these conflaters is the point of science. If you can't replicate the experiment, it means the results don't hold, but it doesn't mean the investigators were bad people - and assuming they are is unlikely to get better science.
majormajor 5y ago
> So those five labs have a pretty good shot at doing interesting work, even some of that is going to reproduce and some of it isn’t. But once you get out of those top five labs, it’s pretty much make-work, incremental, marginal improvements at best, and a complete waste of time otherwise. And I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Anyway, a longwinded way of saying that we have pretty serious structural and incentive problems in the research complex. " This seems like an non-thorough and fairly uninteresting answer to the question. I would want to know how much useful-but-not-groundbreaking stuff still comes out of the "make work" crowd, how wide you need the funnel to be at the top to make sure you catch the people who can end up in the top labs and who make breakthroughs, how much of a "safety net" that you need to make sure people are motivated to jump into the funnel in the first place rather than going into something safer, etc. A world where we have fewer "wasteful" make-work research jobs but also have fewer breakthroughs isn't great. Hell, you could call the majority of today's service- and information-oriented non-academic jobs "make-work" too (let's up those clickthrough rates! let's stream this video in 8k instead of 4k! let's move this money around slightly faster!). So is there necessarily much utility in moving "make-work" researchers into other fields?
deleted 5y ago
[deleted]
yuy910616 5y ago
It should be point out that Marc Andreessen of all people should deeply believe Sturgeon's law - after all, return on investment for venture firms are dominated by few 'hits' and lots of 'duds'. Of course - I think if anyone act on the belief that 90% of their bets are duds, they would not be in the venture business. It's really interesting that a process could lead to a distribution - but knowing the outcome distribution is actually...useless?
helixc 5y ago
I believe, this is how scaling works, and exactly the price we pay to afford the top-notch. In any given field, company, pro sport team, there are top 10% labs/groups/players that matters the most. However, to make the 10% emerge, there has to be a pyramid where the rest 90% is responsible for forming a stable infrastructure. Then it comes down to how well we can design the pyramid. Our result should be evaluated not by how thin we cut down the infrastructure, but how easy we make top-players emerge. On another note, I have seen small no-name labs published impactful scientific results, and then went back to stealth mode again. It's super cool. Science labs, unlike a business entity, should not be measured by recurrent revenue and growth.
- 908B64B197 5y ago
  > On another note, I have seen small no-name labs published impactful scientific results, and then went back to stealth mode again. It's super cool. Science labs, unlike a business entity, should not be measured by recurrent revenue and growth. Did the individual contributors stay at these labs?
- habitue 5y ago
  > the price we pay to afford the top-notch. So, usually when people say this about sturgeon's law type stuff, it's because we genuinely don't know how to determine what's crap and what's not overall. For example, if we're talking about 90% of academic papers are crap, ok, it's true, but we pay the cost because we don't apriori which 10% will be the good ones. The issue is that in academics, we actually have a pretty good idea which programs produce the best and most impactful work. Now it doesn't seem like a cost we have to pay, but rather a conscious decision not to cut out the crap (or to keep funding the crap). I think the ideal outcome is that the culture and resources of all programs are approximately the same, and each produces 90% crap, and 10% good stuff.
  - lambdatronics 5y ago
    There's a difference between high-impact and high quality. You'd hope that the peer review process selects for high quality and then the impact is TBD by history, but it's usually the other way around.
deleted 5y ago
[deleted]
majormajor 5y ago
Playing with some numbers... here's an interesting way setting a bar for publication can backfire: if you don't publish negative results, and you define positive results as 0.05-level, you're going to greatly inflate the amount of false-positives you publish, in fields that rely on statistical significance. (Fudging here around the 0.05 threshold threshold, since stuff on that boundary is the most likely to produce false positives, not sure what how to use a more realistic range... I think this would just be upper-bound on false positives? Please correct/clarify if your probability/statistics knowledge here is deeper than mine!) Let's say twenty different researchers have a similar idea, they all pursue it, and one of them gets (un)lucky and gets apparently statistically significant results. They're the only ones that try to publish, and nobody ever has the knowledge that it only worked 1 time out of 20 (or more?) - even the people who's experiments failed who see it are only aware of their own attempt, and maybe they just messed something up. Let's say 90% of all published results are non-reproducible. What sort of pipeline would be required to create that? For every reproducible, significant finding, there are 9 false-positives. For each of those false-positives there were 19 true-negatives. So about 1 out of 180 research attempts are reproducible truly-meaningful novel results. Does that sound reasonable? It's hard, right, especially to keep finding new things on top of everyone else's continued progress too? But it isn't necessarily malicious or fraudulent every step of the way - we just have our system structured in a way that biases in favor of creating false positives. (Put in other numbers, say 10% of all research has significant results, then 90% of it doesn't, and at the 0.05 level, that's still (up to?) 4.5 false positives for every 10 truly statistically significant things, meaning close to 1 out of every 3 papers would still be expected to be a false positive.) Publishing negative results would require a bigger publication industry, and more people spending more time filtering through it all, but you'd at least be able to compare to the literature to say "this looks like it worked, but ... it hasn't ever worked for anyone else."
screye 5y ago
IMO, it is another form of compensation. People look to get 3 things in a job: Compensation, Interest and Reputation. The professor is a researcher who teaches. However, they do not want to merely be a teacher. Especially when it comes with a substantial pay cut in comparison to the industry. So, you offer them repute and interest by giving tenure and something to research. The university gets a better teacher for cheap and the professor gets an inflated sense of ego and self-actualization. From the student's point of view, it opens up alternate avenues for promotion. If only the top 5 labs get funded enough to do good work, then entering those top 5 labs early is absolutely essential. Anyone who doesn't get on the bandwagon early, will be left to the wolves like medical residencies. It will greatly exacerbate the competition in standardized exams and sideline students who don't abide by a rigid template early in life. (eg: a lot of friends went to tier 2 universities for their masters, did excellent work and got into the top 5 labs purely because the top 50 lab was funded enough for them to make up for lost time early in life) I have also seen lab impact scores jump around substantially enough, that while the 90% hypothesis may be true, we have no way to picking out who will create the 10% that's gold. While half of the 10% might get reliably produced by the top 5 labs. The other half can be notoriously hard to pin down. You will see this in top industry groups (MSR, Brain) that hire exclusively out of the top 5 labs. Even there, the 90% number holds, despite there being a hiring system specifically built for capitalistic impact. On the flip side, any academic field whose employability exists only within academia is a dictionary-definition pyramid scheme. It is a common accusation that's correctly levied on the liberal arts, but STEM should not be immune to it.
908B64B197 5y ago
> So his analysis was, anything you get in the field of medical experimentation, biomedical development, and this is going to be true of any field, there’s like five labs total in the world that are really good at what they’re doing and doing really cutting edge work. And this is true of quantum computing; pick any field for advanced technology you want. That's also true for software engineers. A handful of 10x can carry a company a long way.
sjg007 5y ago
It is also difficult to identify who is doing good work a priori and there’s also the out of left field work which proves successful despite being a no name. You also train new scientists etc…. I think most noble prizes are from people whose research was initially discounted.
cscurmudgeon 5y ago
> So those five labs have a pretty good shot at doing interesting work, even some of that is going to reproduce and some of it isn’t. But once you get out of those top five labs, it’s pretty much make-work, incremental, marginal improvements at best, and a complete waste of time otherwise. And I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Anyway, a longwinded way of saying that we have pretty serious structural and incentive problems in the research complex. " You need to fund avenues that don't look promising now. Some new developments may make them good. E.g. deep learning before 2011.

satellite2 5y ago

As if we are capable of determining who is doing the top 10% best science and are not glued to the 10% most shining project according to the fad of the day.

dotcommand 5y ago

> He’s like “well look, 90% of everything is shit” I think there is a difference between "being shit" and "being fake" though. 'do you really think it’s true that 50-70% of biomedical research is fake?' Fake seems more insidious and problematic. Also this brings up a very troubling issue. If 90% of the "research is fake/shit" and lets say that means that 90% of the researchers are "fake/shit", then it means that 'scientific' consensus is also likely "fake/shit". This is why I'm always skeptical of 'scientific' consensus. Science is about evidence, testing, etc. Not consensus - which is the realm of politics, law, etc. And I'd suspect 99.99% of social 'sciences' is probably 'fake/shit' and that is used to push/change society/government/etc.

techfoolery 5y ago
I think it's irrelevant to what was being said in their conversation. In Marc's context, being fake/shit equated to non-replicable simply. Before that quote, I believe he was talking about the replication crisis briefly.

petermcneeley 5y ago

https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law This law seems to lack any kind of rigor or theoretical basis and seems like a purely rhetorical device.

ysavir 5y ago
As are most of these "laws". The idea isn't to do a conclusive, thorough examination of all things under the sun and render a statement that reflects that particular moment in time. The idea is to make an observation--perhaps true, perhaps false--that helps us maintain a healthy mindset when examining and comprehending things. It's not meant to be insightful, it's meant to encourage us to strive for better insight.
- vickspicks 5y ago
  90% of laws are bad
  - capiki 5y ago
    What law is that?
    - ysavir 5y ago
      Technically, Sturgeon's law.
  - breck 5y ago
    84.3% of statistics are made up on the spot
- blueboo 5y ago
  OK, but let’s be clear: it’s not rational. It’s “thoughts and prayers” for the Star Slate Codex set.
  - ysavir 5y ago

nonameiguess 5y ago

Not only this, but it appears to me that diagnoses of the state of science as a problem are making the classic problem a person like Andrew Gelman would usually quickly point out, which is ignoring base rates. Saying 70% of published research can't replicate sounds bad in a vacuum, but how does that compare to 50 years ago? We have no idea if this number that sounds terrible outside of any context represents an improvement, degradation, or no difference at all from what we should expect. That isn't by any means to suggest we shouldn't try to make science better by being more replicable and reducing fraud, but if it is an improvement, we're already doing something right and there is far less cause for despair and cynicism. It also, of course, doesn't do any base rate comparison to non-scientific ways of generating facts. If 90% of published psychology research turns out to be wrong, but 99% of folk psychology turns out to be wrong, then science is still winning there.

jollybean 5y ago
"Saying 70% of published research can't replicate sounds bad in a vacuum, but how does that compare to 50 years ago?" It doesn't matter a whole lot with respect to 'how it compares to 50 years ago' if the assumption that science is supposed to be objective and credible. It's supposed to be 'approaching 100%' i.e. they are publishing findings, not fake findings. i.e. it's not a game of 'getting a higher percentage' it's a game of integrity. The implications that 'most science is rubbish' is pretty scary actually and it has profound consequences for the future both in practical terms (i.e. we can't trust a lot of it), but especially in populist terms (i.e. people not trusting climate change, vaccine research etc.). "If 90% of published psychology research turns out to be wrong, but 99% of folk psychology turns out to be wrong, then science is still winning there. " Is it though? We spend bazillions on psychology and nothing on pop psychology, so we'd have to contemplate how valuable that '10%' is to society - especially in the context that we may not even know 'which 10%'. This is kind of a big problem.
- majormajor 5y ago
  There's a big, important difference between "fake fraudulent findings" and "not-replicable findings." The latter suggests problems stemming from a big nasty complicated poorly-understood world, and can be improved with introspection and change over time. The first step is the greatly increased discussion over it now than in the past - we're discovering the size of the problem. The next step will be to do things with that. Publishing negative results, getting everyone to have a better understanding of prob/stat... I think those are the directions to go in. If, on the other hand, you just dumped all those psychology researchers out of the academic environment and sent them to marketing and advertising companies... are we better off?

ren_engineer 5y ago

it's all about incentives, the modern academic system has the incentives all wrong for creating innovation and quality work. Replication crisis is just a symptom of the real problem

u385639 5y ago

Interesting. Marc is a nuanced guy of course but I sense he may have shifted in the last while. I recall in his 1:1 w/ Peter Thiel he took the opposing side of this argument. Peter does not mince words re: the progress of "science". Perhaps Marc was just doing it for the sake of discussion.

j7ake 5y ago

Yes, but this sturgeons law (maybe less extreme extent) often applies to within labs or within scientists as well. Newton did first rate work, but he also spent a lot of time working on alchemy and theology which didn’t get anywhere. Research grants are generally competitive, if you haven’t shown you’ve ever done something my significant or haven’t done something significant in a long time, it is difficult to survive. But indeed even the top scientists spend a lot of time on dead ends. It is true that often only top 5 labs or so doing good work, but those top 5 labs are not constant over time, they wax and wane like fashion trends. The lesson is just because over last few years the top work came from only a few labs, it’s premature to eliminate all other labs because those top labs could be lower in the next cycle.

lisper 5y ago
It's also important to distinguish between things that turn out to be crap in retrospect and things that can be known to be crap given the information available at the time. Those are two very different things. Yes, alchemy is crap, but there's no way Newton could have known that given the data he had at his disposal at the time.
- WalterBright 5y ago
  > things that can be known to be crap given the information available at the time Sometimes people do investigate the crap, and discover it's actually fertilizer. The trouble is, one cannot tell in advance.
- lambdatronics 5y ago
  You understate your point. Many advances were 'known' in advance to be crap.
tmn 5y ago
It’s somewhat laughable to say he worked on alchemy and theology and that got no where, while we are discussing him and his accomplishments 400 years later. Alchemy is an esoteric inner journey. Ramanujan was by all accounts a one of a kind genius. Just from wiki ‘Ramanujan credited his substantial mathematical capacities to divinity, and said the mathematical knowledge he displayed was revealed to him by his family goddess Namagiri Thayer’. Belief systems can be very powerful or destructive to ones own accomplishments

version_five 5y ago

I don't see a better system. Maybe I'm misunderstanding his or your point. To me this is essentially capitalism. 90% or more of businesses fail, and of those that don't, 90% or more suck. But if you tried to call that 99% "waste" that could be avoided by only running the good business, you would fail miserably. Research is exactly the same. We can't just identify and fund the good research. (But this should be obvious to a VC so I may be reading it wrong)

yuy910616 5y ago
Well said - Perhaps a way to look at it is that 'good' is a subjective target that should be define as 'distance from average/medium'. So let's be thankful of the crappy medium haha
jollybean 5y ago
This is definitely not the right analogy because one is a matter of risk, the other a matter of integrity. New business ventures are inherently risky. Research is inherently risky, and we would expect 90% to 'fail' but for 90% of papers to be rubbish would be a matter of 'extreme corruption' along the lines of 90% of judges taking bribes etc.. "I don't see a better system." Well, one thing we could do is punish people for knowingly publishing crap results, i.e. have a 'secondary review system' that investigates 'why' results were rubbish, and if there was malfeasance, well, put in place some kind of retribution. We could also change our tune with respect to expectations, and come to 'expect' that the majority of research should publish then 'null' result and that should just be it.
- yuy910616 5y ago
  I think this is a good point too - venture companies and research SHOULD have two very different distributions
- nostrademons 5y ago
  Be careful about assuming malice where ignorance exists. It's possible for 90% of papers to be rubbish without any ill intent or dishonesty on the part of the experimenter. They might simply suffer from "works for me" syndrome, as it afflicts software developers. In science, this equates to failing to control for all the possible confounding factors that might affect the result of the experiment. They might have carefully controlled for everything they thought of and gotten methodologically valid results, but then when another experimenter tries to replicate it, they find their experimental environment is different in a way that neither considered important but alters the results. Finding all these conflaters is the point of science. If you can't replicate the experiment, it means the results don't hold, but it doesn't mean the investigators were bad people - and assuming they are is unlikely to get better science.

majormajor 5y ago

> So those five labs have a pretty good shot at doing interesting work, even some of that is going to reproduce and some of it isn’t. But once you get out of those top five labs, it’s pretty much make-work, incremental, marginal improvements at best, and a complete waste of time otherwise. And I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.” Anyway, a longwinded way of saying that we have pretty serious structural and incentive problems in the research complex. " This seems like an non-thorough and fairly uninteresting answer to the question. I would want to know how much useful-but-not-groundbreaking stuff still comes out of the "make work" crowd, how wide you need the funnel to be at the top to make sure you catch the people who can end up in the top labs and who make breakthroughs, how much of a "safety net" that you need to make sure people are motivated to jump into the funnel in the first place rather than going into something safer, etc. A world where we have fewer "wasteful" make-work research jobs but also have fewer breakthroughs isn't great. Hell, you could call the majority of today's service- and information-oriented non-academic jobs "make-work" too (let's up those clickthrough rates! let's stream this video in 8k instead of 4k! let's move this money around slightly faster!). So is there necessarily much utility in moving "make-work" researchers into other fields?

deleted 5y ago

[deleted]

yuy910616 5y ago

It should be point out that Marc Andreessen of all people should deeply believe Sturgeon's law - after all, return on investment for venture firms are dominated by few 'hits' and lots of 'duds'. Of course - I think if anyone act on the belief that 90% of their bets are duds, they would not be in the venture business. It's really interesting that a process could lead to a distribution - but knowing the outcome distribution is actually...useless?

helixc 5y ago

I believe, this is how scaling works, and exactly the price we pay to afford the top-notch. In any given field, company, pro sport team, there are top 10% labs/groups/players that matters the most. However, to make the 10% emerge, there has to be a pyramid where the rest 90% is responsible for forming a stable infrastructure. Then it comes down to how well we can design the pyramid. Our result should be evaluated not by how thin we cut down the infrastructure, but how easy we make top-players emerge. On another note, I have seen small no-name labs published impactful scientific results, and then went back to stealth mode again. It's super cool. Science labs, unlike a business entity, should not be measured by recurrent revenue and growth.

908B64B197 5y ago
> On another note, I have seen small no-name labs published impactful scientific results, and then went back to stealth mode again. It's super cool. Science labs, unlike a business entity, should not be measured by recurrent revenue and growth. Did the individual contributors stay at these labs?
habitue 5y ago
> the price we pay to afford the top-notch. So, usually when people say this about sturgeon's law type stuff, it's because we genuinely don't know how to determine what's crap and what's not overall. For example, if we're talking about 90% of academic papers are crap, ok, it's true, but we pay the cost because we don't apriori which 10% will be the good ones. The issue is that in academics, we actually have a pretty good idea which programs produce the best and most impactful work. Now it doesn't seem like a cost we have to pay, but rather a conscious decision not to cut out the crap (or to keep funding the crap). I think the ideal outcome is that the culture and resources of all programs are approximately the same, and each produces 90% crap, and 10% good stuff.
- lambdatronics 5y ago
  There's a difference between high-impact and high quality. You'd hope that the peer review process selects for high quality and then the impact is TBD by history, but it's usually the other way around.

deleted 5y ago

[deleted]

majormajor 5y ago

Playing with some numbers... here's an interesting way setting a bar for publication can backfire: if you don't publish negative results, and you define positive results as 0.05-level, you're going to greatly inflate the amount of false-positives you publish, in fields that rely on statistical significance. (Fudging here around the 0.05 threshold threshold, since stuff on that boundary is the most likely to produce false positives, not sure what how to use a more realistic range... I think this would just be upper-bound on false positives? Please correct/clarify if your probability/statistics knowledge here is deeper than mine!) Let's say twenty different researchers have a similar idea, they all pursue it, and one of them gets (un)lucky and gets apparently statistically significant results. They're the only ones that try to publish, and nobody ever has the knowledge that it only worked 1 time out of 20 (or more?) - even the people who's experiments failed who see it are only aware of their own attempt, and maybe they just messed something up. Let's say 90% of all published results are non-reproducible. What sort of pipeline would be required to create that? For every reproducible, significant finding, there are 9 false-positives. For each of those false-positives there were 19 true-negatives. So about 1 out of 180 research attempts are reproducible truly-meaningful novel results. Does that sound reasonable? It's hard, right, especially to keep finding new things on top of everyone else's continued progress too? But it isn't necessarily malicious or fraudulent every step of the way - we just have our system structured in a way that biases in favor of creating false positives. (Put in other numbers, say 10% of all research has significant results, then 90% of it doesn't, and at the 0.05 level, that's still (up to?) 4.5 false positives for every 10 truly statistically significant things, meaning close to 1 out of every 3 papers would still be expected to be a false positive.) Publishing negative results would require a bigger publication industry, and more people spending more time filtering through it all, but you'd at least be able to compare to the literature to say "this looks like it worked, but ... it hasn't ever worked for anyone else."

screye 5y ago

IMO, it is another form of compensation. People look to get 3 things in a job: Compensation, Interest and Reputation. The professor is a researcher who teaches. However, they do not want to merely be a teacher. Especially when it comes with a substantial pay cut in comparison to the industry. So, you offer them repute and interest by giving tenure and something to research. The university gets a better teacher for cheap and the professor gets an inflated sense of ego and self-actualization. From the student's point of view, it opens up alternate avenues for promotion. If only the top 5 labs get funded enough to do good work, then entering those top 5 labs early is absolutely essential. Anyone who doesn't get on the bandwagon early, will be left to the wolves like medical residencies. It will greatly exacerbate the competition in standardized exams and sideline students who don't abide by a rigid template early in life. (eg: a lot of friends went to tier 2 universities for their masters, did excellent work and got into the top 5 labs purely because the top 50 lab was funded enough for them to make up for lost time early in life) I have also seen lab impact scores jump around substantially enough, that while the 90% hypothesis may be true, we have no way to picking out who will create the 10% that's gold. While half of the 10% might get reliably produced by the top 5 labs. The other half can be notoriously hard to pin down. You will see this in top industry groups (MSR, Brain) that hire exclusively out of the top 5 labs. Even there, the 90% number holds, despite there being a hiring system specifically built for capitalistic impact. On the flip side, any academic field whose employability exists only within academia is a dictionary-definition pyramid scheme. It is a common accusation that's correctly levied on the liberal arts, but STEM should not be immune to it.

908B64B197 5y ago

> So his analysis was, anything you get in the field of medical experimentation, biomedical development, and this is going to be true of any field, there’s like five labs total in the world that are really good at what they’re doing and doing really cutting edge work. And this is true of quantum computing; pick any field for advanced technology you want. That's also true for software engineers. A handful of 10x can carry a company a long way.

sjg007 5y ago

It is also difficult to identify who is doing good work a priori and there’s also the out of left field work which proves successful despite being a no name. You also train new scientists etc…. I think most noble prizes are from people whose research was initially discounted.

cscurmudgeon 5y ago