4 ms·

IanCal - I upvoted you but let me comment for extra emphasis. I remember coming out of PhD level quantitative social science studies in academia back in the lat

by mswen 10y ago

IanCal - I upvoted you but let me comment for extra emphasis. I remember coming out of PhD level quantitative social science studies in academia back in the late 90s where I was using K-means clustering, Factor analysis, multiple linear regression, ANOVA and more. When I moved into marketing research and dealing with company data it was shocking and disheartening how little of those skills I could actually deploy. Data quality, data management and just the cost of capturing relevant data was so high that we were reduced to much simpler analyses. Over time I came to appreciate exactly what you were saying. Many companies can be helped by fairly simple analyses. Fast forward - data is getting much cheaper and now all these years later my more advanced stats skills seem to actually matter. But even in this apparent abundance ... data management and data quality assurance is often lacking or significantly underfunded and simple aggregations and analyses still make the most difference in many organizations.

Declanomous 10y ago
I went into marketing from a biology background, and I felt the transition was very easy. Collecting data in biology, especially on an ecosystem scale, is extremely difficult, and your data has a lot of noise in it. How the data is collected and processed is almost more important than how the data was analyzed. Collecting and processing data is going to be extremely time consuming and expensive regardless of what you do, but you can make it easier on yourself through smart experimental design. P-values are still widely used for analysis in biology, to the chagrin of most statisticians. I don't think it is as much of a problem as it comes across as though. Bayesian reasoning is inherently included in the scientific process as part of the experimental design stage. The reliance on p-values becomes a problem when journalists report on research findings in a single paper that is "significant" because of p-values. The professionals in the field are capable of balanced analysis of p-value based research, but we end up with issues like anti-vaxxers when research is presented outside the field of professionals. Of course there are still ways the process can allow incorrect research to present itself. For one thing, journals are not interested in publishing studies that do not have "significant" results, which means that bad research can stick around for far too long. I personally think that the solution to these problems is open data, since so much of the research depends on how the data was manipulated and 'cleaned' prior to analysis.

by mswen 10y ago

Declanomous 10y ago

I went into marketing from a biology background, and I felt the transition was very easy. Collecting data in biology, especially on an ecosystem scale, is extremely difficult, and your data has a lot of noise in it. How the data is collected and processed is almost more important than how the data was analyzed. Collecting and processing data is going to be extremely time consuming and expensive regardless of what you do, but you can make it easier on yourself through smart experimental design. P-values are still widely used for analysis in biology, to the chagrin of most statisticians. I don't think it is as much of a problem as it comes across as though. Bayesian reasoning is inherently included in the scientific process as part of the experimental design stage. The reliance on p-values becomes a problem when journalists report on research findings in a single paper that is "significant" because of p-values. The professionals in the field are capable of balanced analysis of p-value based research, but we end up with issues like anti-vaxxers when research is presented outside the field of professionals. Of course there are still ways the process can allow incorrect research to present itself. For one thing, journals are not interested in publishing studies that do not have "significant" results, which means that bad research can stick around for far too long. I personally think that the solution to these problems is open data, since so much of the research depends on how the data was manipulated and 'cleaned' prior to analysis.