10 ms·

The reality is that genuinely interesting, productive and novel results are hard to produce. Even for teams of highly intelligent researchers. The scientific in

The reality is that genuinely interesting, productive and novel results are hard to produce. Even for teams of highly intelligent researchers. The scientific industry is built on the assumption that such results are in greater supply - or more accessible - than they actually are.

icelancer 9y ago
I don't think I could have put it better than that. Very well done. I am reminded of a parallel quote by Warren Buffett regarding the lack of truly good investments: "I could improve your ultimate financial welfare by giving you a ticket with only twenty slots in it so that you had twenty punches - representing all the investments that you got to make in a lifetime. And once you'd punched through the card, you couldn't make any more investments at all. Under those rules, you'd really think carefully about what you did, and you'd be forced to load up on what you'd really thought about. So you'd do so much better."
- jon_richards 9y ago
  I'm really surprised he said that. It flies in the face of diversified risk and everything it affects, like Sharpe ratios.
  - davidgrenier 9y ago
    I think it's meant to be taken as the number of times you get to change your portfolio.
    - njarboe 9y ago
      I don't think so. Buffett is more of a buy to hold forever kind of investor.
  - Lon7 9y ago
    The common numbers thrown around are that with 20 stocks your portfolio risk is reduced by 70%. The corresponding ratios follow. So in that sense it's not that bad. Of course that's just one form of risk. The less attractive aspect of a 20 stock strategy comes from the fact that the majority of the market's returns come from very few stocks - The 80/20 rule applies pretty well here. With only 20 stocks you'll probably miss out on the few winners that contribute all the market's gains. It's very easy to end up with a 20 stock portfolio with low risk/variance and low returns. Of course if you are Buffet, then your goal is to pick 20 stocks that all outperform.
  - mcguire 9y ago
    Buffet is not widely recognized as a fan of any form of the efficient market hypothesis in general or of diversification-reduces-risk specifically. https://www8.gsb.columbia.edu/articles/columbia-business/superinvestors https://www8.gsb.columbia.edu/articles/columbia-business/sup...
  - sobani 9y ago
    "The way to become rich is to put all your eggs in one basket and then watch that basket." - Andrew Carnegie The way Buffett invests is by reading all day, everyday and once in a while making a very informed investment. Most people haven't even reached the level of staying away from 'sure things' and panic selling during bad news. How many people would've been able to stay away from IT stocks during the dotcom boom, to even not invest in the company of one of your friends. (Bill Gates and Buffett have known each other since '91) How often do you, as a 'master', break rules that you would tell a 'apprentice' are immutable?
  - pandem 9y ago
    It doesn't mean you can't buy indexes or other diversified assets but rather that you should stick with what you choose and not try to time the market.
  - tome 9y ago
    One way of thinking about this: Every time you add a new instrument to your portfolio you risk goes down. But return also goes down because there are only a very small number of excellent investments available.
mulmen 9y ago
"Everything that can be invented has already been invented" - Charles H. Duell, 1899. We don't know anywhere near "most" things. The supply of novel discovery is not the limitation.
- AdeptusAquinas 9y ago
  As an aside, that quote is apocryphal, by the way. The actual Charles H Duell said almost the exact opposite
- icelancer 9y ago
  The average age of a Nobel laureate continues to climb at a steady rate. We are reaching asymptotic knowledge gains in all the major fields. This fact combined with the tenure track / novelty factor leads to what we see today.
  - deleted 9y ago
    [deleted]
  - indubitable 9y ago
    There is, of course, an alternative explanation. Our society has changed, dramatically, in the past 30 years. We've gone from a completely disconnected society with connections spanning arm's reach to a globally connected society with connections spanning just as far. Baby sitters have been replaced with digital tablets providing endless entertainment and ensuring no person, adult or child, ever need sit with their own recreation being solely within their mind. One needs only look at the slew of digital devices whipped out at any eatery, lest individuals be left to their own thoughts for an agonizing 10 minutes. The internet and our technology has shattered nearly all educational barriers. Getting an MIT caliber education for free is now something any child in a developing country who has access to the internet can do, if so inclined. On the other hand they can find countless entertaining drama to read or participate in, find an endless supply of porn, engage in social media and other online sites that are specifically designed for addiction, or find an array of games and other forms of fast response stimuli all designed to make him or her feel good. Einstein constantly related his tales of meandering strolls as he pondered his work. Another individual that has managed to achieve some great things in modern science is Stephen Hawking. He's mentioned spending decades thinking intently, even if not exclusively, on singular topics. And there is absolutely every reason to believe that is in no way bluster. Given his condition, his mind is his recreation. Is the breaking down of information barriers in our society helping more than the massive commercialization and directed addiction towards all sorts of mediums and vices along with the rapid reward systems they offer, is hurting it? It's not just high level academia that's failing. Education systems to be deteriorating at all levels with US (which is the center of much of this change) scores in math and science rapidly falling behind much of the rest of the world. Healthfulness has fallen to dangerously low levels. And more. Regression is not impossible.
lisper 9y ago
This is a cultural holdover from the late 19th and earlier 20th century when novel results were much more plentiful. But now nearly all of the low-lying scientific fruit has been picked so the scientific endeavor needs to adjust its expectations. Unfortunately, culture changes more slowly than science. BTW, the lack of new results is actually a good thing. It means we've got a lot of stuff figured out.
- dekhn 9y ago
  or it could mean we're stuck in a local minima due to structural barriers
- jansho 9y ago
  > BTW, the lack of new results is actually a good thing. It means we've got a lot of stuff figured out. Not necessarily. If anything, the more we find out, the more we realise how much we don't know. Take quantum physics, that stuff slowed down since the 1930s. It's not a bad thing, the journey's always fun too. But let's not kid ourselves that we're close to figuring everything out.
  - smallnamespace 9y ago
    Personally, I interpret GP as we have already proven many of the true hypotheses that can be simply and easily formulated, based on the categories and concepts that we currently find intuitive. And this shouldn't really be a surprise. Our cultures and societies have had thousands of years of evolution; our human instincts have had at least hundreds of thousands of years. In that time, both culture and instinct have picked up many true facts about the world that have helped us survive. A lot of science has been taking those facts and systematically investigating how far they hold. So these both can be true: the more questions we ask, the more questions we have to answer, but also that many of the 'easy' hypotheses have been mined out, and any true advance in science will by necessity pull us further and further from the realm where our basic instincts and intuitions can usefully guide us. Quantum mechanics being the prototypical example, of course.
    - 9y ago
BurningFrog 9y ago
Meanwhile, false positives are as easy as ever to come across. Easier, if you factor in how much easier P hacking is with modern software.
jansho 9y ago
Then all the more reason for replication. Shouldn't we focus on making the foundation as robust as possible, before building more on it? It's possible to discover something exciting, or completely novel, but "exciting" and "novel" shouldn't really be the sole pursuit of a scientist. Rather, all outcomes including 'boring' ones should be considered with equal attention.
adrianratnapala 9y ago
Hard to produce is not the same as hard to reproduce. Genuinely new results require developing new insights, new techniques or just making new guesses that happen to be right. All are rare enough. But once that is done, the information gets out and the experiments should be easily reproducible. Some dispciplines get a lot of reproduction as a side effect of using last year's breakthrough research as "mere" technology for implementing next year's expriment. But I guess that sort of thing is less natural pyschology and sociology.
- PeterisP 9y ago
  For many experiments, the cost (required effort) of reproducing them is the same as producing them, simply because you need to do all the same major activities once more. If they needed to spend an hour each with a hundred subjects, you need to do that too. If they needed to carefully run a long process on some materials, you need to do that to. If they needed to build a particular apparatus specific to that experiment, then you need to build or buy one. And it's possibly more costly than the original since the initial lab likely was able to re-use some of their existing resources, trained people, infrastructure, lab animals, equipment (whatever matters for the discipline) which you don't have; they likely chose this particular experiment because it fits well with the stuff that they have that's not available everywhere. If they developed a particular experimental setup / gear / network of volunteers doing interviews in the subpopulation you're studying / cell or animal line / etc, then they're using that for many publications over the years, but you need to spend the whole cost to reproduce any one of them.
  - adrianratnapala 9y ago
    Good, point. I was once an experimental physicist and never psychologist. In physics there is a lot of trial and error before you can get the experiment to do anything at all. Anyone who reads your paper needs fewer trials. So I was going to agree with you that in fields where experiments work differently, reproduction can cost as much as the original study. But on second thought, that's wrong: and indeed it is the whole point of the thread. There is always trial-and-error. You simply don't know if your theory is correct. And there is always risk of a methodolical weakness that you only understand after seeing the data. The trouble is after you have already spent all your time and grant money on your study, you have every incentive to paper over these issues. Indeed even if you are scrupulously honest, you will have to shove study in the proverbial file drawer - because you don't have the resources to try another iteration. So I think it's still true that a new experiment costs more than reproduction, and many of the problems with studies arise from trying to pretend that cost doesn't exist.

icelancer 9y ago

I don't think I could have put it better than that. Very well done. I am reminded of a parallel quote by Warren Buffett regarding the lack of truly good investments: "I could improve your ultimate financial welfare by giving you a ticket with only twenty slots in it so that you had twenty punches - representing all the investments that you got to make in a lifetime. And once you'd punched through the card, you couldn't make any more investments at all. Under those rules, you'd really think carefully about what you did, and you'd be forced to load up on what you'd really thought about. So you'd do so much better."

jon_richards 9y ago
I'm really surprised he said that. It flies in the face of diversified risk and everything it affects, like Sharpe ratios.
- davidgrenier 9y ago
  I think it's meant to be taken as the number of times you get to change your portfolio.
  - njarboe 9y ago
    I don't think so. Buffett is more of a buy to hold forever kind of investor.
- Lon7 9y ago
  The common numbers thrown around are that with 20 stocks your portfolio risk is reduced by 70%. The corresponding ratios follow. So in that sense it's not that bad. Of course that's just one form of risk. The less attractive aspect of a 20 stock strategy comes from the fact that the majority of the market's returns come from very few stocks - The 80/20 rule applies pretty well here. With only 20 stocks you'll probably miss out on the few winners that contribute all the market's gains. It's very easy to end up with a 20 stock portfolio with low risk/variance and low returns. Of course if you are Buffet, then your goal is to pick 20 stocks that all outperform.
- mcguire 9y ago
  Buffet is not widely recognized as a fan of any form of the efficient market hypothesis in general or of diversification-reduces-risk specifically. https://www8.gsb.columbia.edu/articles/columbia-business/superinvestors https://www8.gsb.columbia.edu/articles/columbia-business/sup...
- sobani 9y ago
  "The way to become rich is to put all your eggs in one basket and then watch that basket." - Andrew Carnegie The way Buffett invests is by reading all day, everyday and once in a while making a very informed investment. Most people haven't even reached the level of staying away from 'sure things' and panic selling during bad news. How many people would've been able to stay away from IT stocks during the dotcom boom, to even not invest in the company of one of your friends. (Bill Gates and Buffett have known each other since '91) How often do you, as a 'master', break rules that you would tell a 'apprentice' are immutable?
- pandem 9y ago
  It doesn't mean you can't buy indexes or other diversified assets but rather that you should stick with what you choose and not try to time the market.
- tome 9y ago
  One way of thinking about this: Every time you add a new instrument to your portfolio you risk goes down. But return also goes down because there are only a very small number of excellent investments available.

mulmen 9y ago

"Everything that can be invented has already been invented" - Charles H. Duell, 1899. We don't know anywhere near "most" things. The supply of novel discovery is not the limitation.

AdeptusAquinas 9y ago
As an aside, that quote is apocryphal, by the way. The actual Charles H Duell said almost the exact opposite
icelancer 9y ago
The average age of a Nobel laureate continues to climb at a steady rate. We are reaching asymptotic knowledge gains in all the major fields. This fact combined with the tenure track / novelty factor leads to what we see today.
- deleted 9y ago
  [deleted]
- indubitable 9y ago
  There is, of course, an alternative explanation. Our society has changed, dramatically, in the past 30 years. We've gone from a completely disconnected society with connections spanning arm's reach to a globally connected society with connections spanning just as far. Baby sitters have been replaced with digital tablets providing endless entertainment and ensuring no person, adult or child, ever need sit with their own recreation being solely within their mind. One needs only look at the slew of digital devices whipped out at any eatery, lest individuals be left to their own thoughts for an agonizing 10 minutes. The internet and our technology has shattered nearly all educational barriers. Getting an MIT caliber education for free is now something any child in a developing country who has access to the internet can do, if so inclined. On the other hand they can find countless entertaining drama to read or participate in, find an endless supply of porn, engage in social media and other online sites that are specifically designed for addiction, or find an array of games and other forms of fast response stimuli all designed to make him or her feel good. Einstein constantly related his tales of meandering strolls as he pondered his work. Another individual that has managed to achieve some great things in modern science is Stephen Hawking. He's mentioned spending decades thinking intently, even if not exclusively, on singular topics. And there is absolutely every reason to believe that is in no way bluster. Given his condition, his mind is his recreation. Is the breaking down of information barriers in our society helping more than the massive commercialization and directed addiction towards all sorts of mediums and vices along with the rapid reward systems they offer, is hurting it? It's not just high level academia that's failing. Education systems to be deteriorating at all levels with US (which is the center of much of this change) scores in math and science rapidly falling behind much of the rest of the world. Healthfulness has fallen to dangerously low levels. And more. Regression is not impossible.

lisper 9y ago

This is a cultural holdover from the late 19th and earlier 20th century when novel results were much more plentiful. But now nearly all of the low-lying scientific fruit has been picked so the scientific endeavor needs to adjust its expectations. Unfortunately, culture changes more slowly than science. BTW, the lack of new results is actually a good thing. It means we've got a lot of stuff figured out.

dekhn 9y ago
or it could mean we're stuck in a local minima due to structural barriers
jansho 9y ago
> BTW, the lack of new results is actually a good thing. It means we've got a lot of stuff figured out. Not necessarily. If anything, the more we find out, the more we realise how much we don't know. Take quantum physics, that stuff slowed down since the 1930s. It's not a bad thing, the journey's always fun too. But let's not kid ourselves that we're close to figuring everything out.
- smallnamespace 9y ago
  Personally, I interpret GP as we have already proven many of the true hypotheses that can be simply and easily formulated, based on the categories and concepts that we currently find intuitive. And this shouldn't really be a surprise. Our cultures and societies have had thousands of years of evolution; our human instincts have had at least hundreds of thousands of years. In that time, both culture and instinct have picked up many true facts about the world that have helped us survive. A lot of science has been taking those facts and systematically investigating how far they hold. So these both can be true: the more questions we ask, the more questions we have to answer, but also that many of the 'easy' hypotheses have been mined out, and any true advance in science will by necessity pull us further and further from the realm where our basic instincts and intuitions can usefully guide us. Quantum mechanics being the prototypical example, of course.
  - 9y ago

BurningFrog 9y ago

Meanwhile, false positives are as easy as ever to come across. Easier, if you factor in how much easier P hacking is with modern software.

jansho 9y ago

Then all the more reason for replication. Shouldn't we focus on making the foundation as robust as possible, before building more on it? It's possible to discover something exciting, or completely novel, but "exciting" and "novel" shouldn't really be the sole pursuit of a scientist. Rather, all outcomes including 'boring' ones should be considered with equal attention.

adrianratnapala 9y ago

Hard to produce is not the same as hard to reproduce. Genuinely new results require developing new insights, new techniques or just making new guesses that happen to be right. All are rare enough. But once that is done, the information gets out and the experiments should be easily reproducible. Some dispciplines get a lot of reproduction as a side effect of using last year's breakthrough research as "mere" technology for implementing next year's expriment. But I guess that sort of thing is less natural pyschology and sociology.

PeterisP 9y ago
For many experiments, the cost (required effort) of reproducing them is the same as producing them, simply because you need to do all the same major activities once more. If they needed to spend an hour each with a hundred subjects, you need to do that too. If they needed to carefully run a long process on some materials, you need to do that to. If they needed to build a particular apparatus specific to that experiment, then you need to build or buy one. And it's possibly more costly than the original since the initial lab likely was able to re-use some of their existing resources, trained people, infrastructure, lab animals, equipment (whatever matters for the discipline) which you don't have; they likely chose this particular experiment because it fits well with the stuff that they have that's not available everywhere. If they developed a particular experimental setup / gear / network of volunteers doing interviews in the subpopulation you're studying / cell or animal line / etc, then they're using that for many publications over the years, but you need to spend the whole cost to reproduce any one of them.
- adrianratnapala 9y ago
  Good, point. I was once an experimental physicist and never psychologist. In physics there is a lot of trial and error before you can get the experiment to do anything at all. Anyone who reads your paper needs fewer trials. So I was going to agree with you that in fields where experiments work differently, reproduction can cost as much as the original study. But on second thought, that's wrong: and indeed it is the whole point of the thread. There is always trial-and-error. You simply don't know if your theory is correct. And there is always risk of a methodolical weakness that you only understand after seeing the data. The trouble is after you have already spent all your time and grant money on your study, you have every incentive to paper over these issues. Indeed even if you are scrupulously honest, you will have to shove study in the proverbial file drawer - because you don't have the resources to try another iteration. So I think it's still true that a new experiment costs more than reproduction, and many of the problems with studies arise from trying to pretend that cost doesn't exist.