4 ms·

As a software engineer with no research experience, I wonder if one answer may lie in more foolproofing of the most commonly statistical analysis software, like

As a software engineer with no research experience, I wonder if one answer may lie in more foolproofing of the most commonly statistical analysis software, like matlab and stata. They are hugely powerful tools but obviously require quite a bit of domain knowledge to use properly. Could many of the most common types of analyses be made into more easily runnable happy paths that also warn against common invalid conclusions and methods? I don’t mean to imply any of this is easy, but I’ve seen other pro-level software successfully integrate happy paths for novices.

Fomite 3y ago
There are a number of statistical software packages that have, as you refer to them, "easily runnable happy paths". I'd assert that these are the source of many of the problems, because statistical reasoning is not purely algorithmic.
- beerandt 3y ago
  No bluescreen level error when I click run? Resulting answer must be be correct.
tnecniv 3y ago
I think better tooling may help, but part of the issue is that the tools are asked to do the wrong thing because you’re thinking about the problem wrong. In the article, they cite the hot hand research: for a long time, the understanding was that the hot hand is a myth, but recent work identified a source of significant statistical bias in the prevailing methodology, and identified that the hot hand is real when you correct for that bias. Better tools definitely will help practitioners, but no tool will save you if you call upon it to do something that isn’t rigorous. As an aside, I’ve wondered about a system that augments data with units dynamically. You tell it what units your data is in when each variable is created, and it carries that through basic computations. Thus if you divide a “meters” variable by a “seconds” variable and print the result, the result will be in meters per second. The value being that it is instantly obvious if you are doing something wrong because you’ll get nonsense units.
- ahartman00 3y ago
  "I’ve wondered about a system that augments data with units dynamically" F#'s units of measure do this! Though the compiler checks it, so it's static, not dynamic. https://fsharpforfunandprofit.com/posts/units-of-measure/ https://fsharpforfunandprofit.com/posts/units-of-measure/

Fomite 3y ago

There are a number of statistical software packages that have, as you refer to them, "easily runnable happy paths". I'd assert that these are the source of many of the problems, because statistical reasoning is not purely algorithmic.

beerandt 3y ago
No bluescreen level error when I click run? Resulting answer must be be correct.

tnecniv 3y ago

I think better tooling may help, but part of the issue is that the tools are asked to do the wrong thing because you’re thinking about the problem wrong. In the article, they cite the hot hand research: for a long time, the understanding was that the hot hand is a myth, but recent work identified a source of significant statistical bias in the prevailing methodology, and identified that the hot hand is real when you correct for that bias. Better tools definitely will help practitioners, but no tool will save you if you call upon it to do something that isn’t rigorous. As an aside, I’ve wondered about a system that augments data with units dynamically. You tell it what units your data is in when each variable is created, and it carries that through basic computations. Thus if you divide a “meters” variable by a “seconds” variable and print the result, the result will be in meters per second. The value being that it is instantly obvious if you are doing something wrong because you’ll get nonsense units.

ahartman00 3y ago
"I’ve wondered about a system that augments data with units dynamically" F#'s units of measure do this! Though the compiler checks it, so it's static, not dynamic. https://fsharpforfunandprofit.com/posts/units-of-measure/ https://fsharpforfunandprofit.com/posts/units-of-measure/