3 ms·

You misrepresent the linked article which doesn't report replication rates but forecasted replication rates from participants "recruited through a number of mai

You misrepresent the linked article which doesn't report replication rates but forecasted replication rates from participants "recruited through a number of mailing lists, Twitter and blog posts".

qsort 3y ago
They used a prediction market, which at the time the article was published was still to resolve. This is standard practice and methodologically sound. They didn't go around asking people on Twitter.
- _Nat_ 3y ago
  Just skimmed it quickly, but from [their paper](https://royalsocietypublishing.org/doi/10.1098/rsos.200566 https://royalsocietypublishing.org/doi/10.1098/rsos.200566 ): > Participants were recruited through a number of mailing lists, Twitter and blog posts. [...] For all analyses, we use survey responses; for analysis (iii), we additionally use responses to a demographic survey that included a question on academic interests; and for analysis (iv), we additionally use the prediction market data. So it sounds like they only used prediction-market data "for analysis (iv)", which they describe as: > [...] (iv) whether survey-based aggregated forecasts and market-based forecasts are correlated. --- The last line of their paper: > If our forecasts hold up, it will be interesting to investigate if specific factors (such as different methodologies and policies) can be identified that influence replication rates. So, it doesn't sound like they're claiming that their forecasts should be assumed reliable.
- tgv 3y ago
  No, it isn't. A prediction market is not proof of anything.
- begemotz 3y ago
  I didn't say that the methodology was not standard practice or methodologically sound. I said you misrepresented the study, which you did. You said "Most social sciences have numbers that are far better than psychology's." in the context of a conversation about *replication numbers*. You did not say "in a prediction market, participants ("recruited through a number of mailing lists, Twitter and blog posts") have lower replication expectation rates for Psychology than other social sciences" -- if you had then, well, you wouldn't have misrepresented the study.
  - qsort 3y ago
    Any meta-study trying to get replication numbers would likely use a combination of methods, including prediction markets, with techniques like having a small fraction of the contracts resolve. So no, implying this is a Twitter poll is a far worse misrepresentation than even the most uncharitable interpretation of what I said.
    - begemotz 3y ago
      > Any meta-study trying to get replication numbers would likely use a combination of methods, including prediction markets, with techniques like having a small fraction of the contracts resolve." this is non-nonsensical obfuscation. poor attempt at moving the goal posts -- for the record I did not imply, I DIRECTLY QUOTED from the article.

You misrepresent the linked article which doesn't report replication rates but forecasted replication rates from participants "recruited through a number of mailing lists, Twitter and blog posts".

qsort 3y ago

They used a prediction market, which at the time the article was published was still to resolve. This is standard practice and methodologically sound. They didn't go around asking people on Twitter.

_Nat_ 3y ago
Just skimmed it quickly, but from [their paper](https://royalsocietypublishing.org/doi/10.1098/rsos.200566 https://royalsocietypublishing.org/doi/10.1098/rsos.200566 ): > Participants were recruited through a number of mailing lists, Twitter and blog posts. [...] For all analyses, we use survey responses; for analysis (iii), we additionally use responses to a demographic survey that included a question on academic interests; and for analysis (iv), we additionally use the prediction market data. So it sounds like they only used prediction-market data "for analysis (iv)", which they describe as: > [...] (iv) whether survey-based aggregated forecasts and market-based forecasts are correlated. --- The last line of their paper: > If our forecasts hold up, it will be interesting to investigate if specific factors (such as different methodologies and policies) can be identified that influence replication rates. So, it doesn't sound like they're claiming that their forecasts should be assumed reliable.
tgv 3y ago
No, it isn't. A prediction market is not proof of anything.
begemotz 3y ago
I didn't say that the methodology was not standard practice or methodologically sound. I said you misrepresented the study, which you did. You said "Most social sciences have numbers that are far better than psychology's." in the context of a conversation about *replication numbers*. You did not say "in a prediction market, participants ("recruited through a number of mailing lists, Twitter and blog posts") have lower replication expectation rates for Psychology than other social sciences" -- if you had then, well, you wouldn't have misrepresented the study.
- qsort 3y ago
  Any meta-study trying to get replication numbers would likely use a combination of methods, including prediction markets, with techniques like having a small fraction of the contracts resolve. So no, implying this is a Twitter poll is a far worse misrepresentation than even the most uncharitable interpretation of what I said.
  - begemotz 3y ago
    > Any meta-study trying to get replication numbers would likely use a combination of methods, including prediction markets, with techniques like having a small fraction of the contracts resolve." this is non-nonsensical obfuscation. poor attempt at moving the goal posts -- for the record I did not imply, I DIRECTLY QUOTED from the article.