5 ms·

So is it really science? These are concepts from stats 101. And the reasons and need, and the risks of not having them are very clear. But you have millions bei

So is it really science? These are concepts from stats 101. And the reasons and need, and the risks of not having them are very clear. But you have millions being put into models without these pre-requisites, and being sold to people as solutions, and waved away as "if people buy is it's bc it has value". People also pay fraudsters.

nradov 3y ago
Mostly not. Very few data "scientists" working in industry actually follow the scientific method. Instead they just mess around with various statistical techniques (including AI/ML) until they get a result that management likes.
- marcinzm 3y ago
  Most decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistically valid.
  - staunton 3y ago
    The magic words here are make sure.
    - marcinzm 3y ago
      In my experience they make sure a ton more in industry than in academia.
      - staunton 3y ago
        Any anecdotes you can share? Also, I meant "make sure" in a negative way. As in you "make sure things are statistically significant" by e.g. p-hacking. Not that this isn't done in science but I think you're more in danger of being embarrassed during peer review than by the C-suite reading your executive summary...
        marcinzm 3y ago
        Most companies that care will run everything though an AB test, the number of AB tests is physically limited by traffic volume and the team in charge of measuring the results of the AB tests is not the team that created the experiment. That makes it much harder to p-hack since you cannot re-run experiments infinitely on a laptop, and the measurement team is judged on the accuracy of their forecasts versus the revenue impact.
        figassis 3y ago
        The worse things can and often do generate more revenue. So what are the goals of the AB test?
        marcinzm 3y ago
        >The worse things can and often do generate more revenue. The goal of a business is usually to generate revenue so I'm confused about your question.
- jmalicki 3y ago
  Science is just fooling around with data until you get a result a journal reviewer likes.
borroka 3y ago
But even in academia, where supposedly "true science" is, if not done, at least pursued, uncertainty intervals are rarely, with respect to the times they would be needed, understood and used. When I used to publish stats- and math-heavy papers in the biological sciences, very rarely the reviewers--and I used to publish in intermediate and up journals--were paying any attention to the quality of the predictions, beyond a casual look at the R2 or R2-equivalents and mean absolute errors.

nradov 3y ago

Mostly not. Very few data "scientists" working in industry actually follow the scientific method. Instead they just mess around with various statistical techniques (including AI/ML) until they get a result that management likes.

marcinzm 3y ago
Most decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistically valid.
- staunton 3y ago
  The magic words here are make sure.
  - marcinzm 3y ago
    In my experience they make sure a ton more in industry than in academia.
    - staunton 3y ago
      Any anecdotes you can share? Also, I meant "make sure" in a negative way. As in you "make sure things are statistically significant" by e.g. p-hacking. Not that this isn't done in science but I think you're more in danger of being embarrassed during peer review than by the C-suite reading your executive summary...
      - marcinzm 3y ago
        Most companies that care will run everything though an AB test, the number of AB tests is physically limited by traffic volume and the team in charge of measuring the results of the AB tests is not the team that created the experiment. That makes it much harder to p-hack since you cannot re-run experiments infinitely on a laptop, and the measurement team is judged on the accuracy of their forecasts versus the revenue impact.
        figassis 3y ago
        The worse things can and often do generate more revenue. So what are the goals of the AB test?
        marcinzm 3y ago
        >The worse things can and often do generate more revenue. The goal of a business is usually to generate revenue so I'm confused about your question.
jmalicki 3y ago
Science is just fooling around with data until you get a result a journal reviewer likes.

borroka 3y ago

But even in academia, where supposedly "true science" is, if not done, at least pursued, uncertainty intervals are rarely, with respect to the times they would be needed, understood and used. When I used to publish stats- and math-heavy papers in the biological sciences, very rarely the reviewers--and I used to publish in intermediate and up journals--were paying any attention to the quality of the predictions, beyond a casual look at the R2 or R2-equivalents and mean absolute errors.