3 ms·

Serious question - how do you engage with new studies and find value? Or do you? I'm trying to rejigger what my info pipelines look like and I'm looking for i

by wsinks 4y ago

Serious question - how do you engage with new studies and find value? Or do you? I'm trying to rejigger what my info pipelines look like and I'm looking for ideas. I like hearing about new stuff, but you're right that I just can't trust a science headline.

colechristensen 4y ago
Read them. Abstract and conclusions. The middle if it seems interesting, followed by looking for related research to confirm the conclusions. Most people don't have the skills to do this, we need better science journalism.
- biomcgary 4y ago
  We need better incentives for science journalism than clicks and ad dollars. The problem is obvious, but the solution is not. (We also have an incentive problem for science, but that is another problem.)
gus_massa 4y ago
Use the press article to get a general understanding. Ignore the part about how it will fix all the problems in the world. Take a look at Wikipedia. Read the abstract of the research article and look for the most important graphic / table. Does it support the abstract of the article? Does it even support the caption of the figure? Ctr+F for some keywords, like "control group", "exclusion criteria", "almost statistically significant". Try to find out if they are using some "interesting" method to filter the data. Use some back of the envelope statistic, like comparing the difference, variance and Sqrt(N). Count how many results are statistically significant and how many are reported. How many implicit results are not reported? Also, are there some results with an incredible small error? There are just too many predatory journals and bad journals with a peer review that is a joke. Unless it's a journal in your area, just ignore the journal and look at the data. And even some serious journals publish bad articles. I use a modified version of the crackpot index were the article starts with +5 points https://math.ucr.edu/home/baez/crackpot.html https://math.ucr.edu/home/baez/crackpot.html In god we trust, all others must bring data.
swayvil 4y ago
Unless you made the observation personally, or personally know the fellow who made the observation, you should take such stories/studies with a grain of salt. It's the sane approach.

by wsinks 4y ago

colechristensen 4y ago

Read them. Abstract and conclusions. The middle if it seems interesting, followed by looking for related research to confirm the conclusions. Most people don't have the skills to do this, we need better science journalism.

biomcgary 4y ago
We need better incentives for science journalism than clicks and ad dollars. The problem is obvious, but the solution is not. (We also have an incentive problem for science, but that is another problem.)

gus_massa 4y ago

Use the press article to get a general understanding. Ignore the part about how it will fix all the problems in the world. Take a look at Wikipedia. Read the abstract of the research article and look for the most important graphic / table. Does it support the abstract of the article? Does it even support the caption of the figure? Ctr+F for some keywords, like "control group", "exclusion criteria", "almost statistically significant". Try to find out if they are using some "interesting" method to filter the data. Use some back of the envelope statistic, like comparing the difference, variance and Sqrt(N). Count how many results are statistically significant and how many are reported. How many implicit results are not reported? Also, are there some results with an incredible small error? There are just too many predatory journals and bad journals with a peer review that is a joke. Unless it's a journal in your area, just ignore the journal and look at the data. And even some serious journals publish bad articles. I use a modified version of the crackpot index were the article starts with +5 points https://math.ucr.edu/home/baez/crackpot.html https://math.ucr.edu/home/baez/crackpot.html In god we trust, all others must bring data.

swayvil 4y ago

Unless you made the observation personally, or personally know the fellow who made the observation, you should take such stories/studies with a grain of salt. It's the sane approach.