5 ms·

really? I think you are being extremely unfavorable to the authors here. They are simply stating the results on blood pressure went down but in a non-significan

really? I think you are being extremely unfavorable to the authors here. They are simply stating the results on blood pressure went down but in a non-significant way. There is nothing wrong with that and also they are not making claims either way they are just reporting the results of the study and literally describing what happened. Please enlighten with what claims they are allegedly making? that the blood pressure decreased in a non-significant way?

robotresearcher 5y ago
"Not significant" means that the probability is >=5% their result was obtained by chance. We've settled as a community on a convention that we don't claim an effect is real until it is supported by data ("statistically significant") ie. <5% likely to be explained by chance in your results. "Significant" does not mean big or important in this context. It means better than 5% unlikely to be (un)lucky data.
- jjcon 5y ago
  They didn't claim it was real did they? Just read out the result which was lower but not in a significant way. I've read hundreds of papers do the same.
  - jhgb 5y ago
    > which was lower but not in a significant way But if "result [was] lower but not in a significant way" means "result was not proven to be lower", how does saying "lower but not really lower" make ever any sense? It seems to me that such nonsensical formulation ought not to be ever used by anyone.
    - jjcon 5y ago
      Because significance thresholds can vary pretty dramatically. Plenty of experiments done in physics for instance have reported results even though they didn't yet reach a 5 sigma threshold (3x10e-7). In physics something can be highly highly likely but still not 'significant' enough to warrant a discovery. They simply couch it as, hey this was the result and even though it isn't 'significant' the high likelihood may warrant additional research here. Reporting a binary significant/not significant is far less useful.
  - robotresearcher 5y ago
    "was lower but not in a significant way" By convention, this means "indistinguishable from", so reporting that it is lower is an unsupported claim. They would be equally justified in reporting that it was higher, ie. not at all.
    - jjcon 5y ago
      It was lower though, just not significantly so (depending on your threshold for significance) - that's the standard way of reporting it. You can't just chunk part of the sentence and take issue with it, the sentence in it's entirety is accurate.
      - robotresearcher 5y ago
        Yes I agree that the sentence is accurate. Perhaps I’m too zealous about it, and maybe the conventions vary, but I was trained to avoid using words like ‘lower’ here.
- kqr 5y ago
  The threshold for significance lies in the eye of the beholder. A particle physicist might not be satisfied with anything over 0.01 %. A social scientist might be happy to see 10 %. The 5 % number you mention is completely arbitrary and often woefully inappropriate. Look at it from a betting perspective. Can you earn more than 10 × your investment if the null hypothesis is false? Then anything less likely than 10 % is significant.
  - robotresearcher 5y ago
    It's a convention for scientific reporting. Your trades are not bound by this convention. The parameter value is not arbitrary. It's a convention arrived at after hundreds of years. If it were arbitrary, p=0.999 or p=0.00001 would be just as good. We've settled on p=0.05 being usefully convincing but not crazy demanding to obtain by experiment with noisy measurements.
    - aaplok 5y ago
      Null hypothesis testing was invented less than 100 years ago by Fisher, who completely arbitrarily picked 0.05 [0]. That value was not arrived at through wisdom of experience, and certainly not after hundreds of years of practice. Though it has now indeed become conventional to test with p=0.05, there is nothing wrong with reporting an effect that fails the null hypothesis test. At least that is the position of the American Statistical Association [1]. [0] https://www.cantorsparadise.com/what-is-the-meaning-of-p-value-and-why-is-it-0-05-5ae45644e7f6 https://www.cantorsparadise.com/what-is-the-meaning-of-p-val... [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5017929/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5017929/
      - robotresearcher 5y ago
        Thanks for these refs. I read [1] carefully and I take your point that it’s ok to strictly report whatever the data says. On the value itself, we are quibbling about the meaning of ‘arbitrary’: Fisher certainly could have chosen another value, but not all values would be considered useful. Some expertise about the nature of real world data and the minds of statisticians is encoded in the chosen value. If I propose that we change the convention to use 1e-12 instead and you think ‘that’s too small, I prefer it the way it is’, then it’s not arbitrary in the sense I mean.
- david-gpu 5y ago
  > "Not significant" means that the probability is >=5% their result was obtained by chance. Ackchually... p-value represents the probability that results like these would be observed even if there was no difference between the two choices, simply due to chance.
mlindner 5y ago
"Non-significant" in scientific means "within the error bars". It means that it is indistinguishable from randomness.
- kqr 5y ago
  That's a bit of an oversimplification that leaves out important qualifiers. "Non-significant" usually means it's indistinguishable from a null hypothesis when a certain level of randomness is allowed in a single, isolated trial. Who picks the null hypothesis? How is it picked, i.e. why does that specific hypothesis get favourable treatment? What level of randomness should one allow? What does it even mean for an experiment to be a single, isolated trial? How can anything be? Those are critical questions to understand the concept, and your explanation just pretends they don't exist.
  - tragictrash 5y ago
    You may be technically correct, but in real life where the rest of us live, it means this study can't be used to draw any conclusions.
    - kqr 5y ago
      That's another fallacy of frequentist reasoning, that we have to draw definitive conclusions from evidence. That something is definitely false until we have "statistical significance" where it all of a sudden becomes definitely true. In real life, to borrow your description, we can hold varying levels of belief in statements depending on how strong the evidence is, and the magnitude of the payoff in the various cases. Maybe the probability of the result in the study in question is 51 %. That's still more than 50 %. Whether that difference is meaningful to you is not something someone else can decide.
      - JohnHaugeland 5y ago
        This is not about fallacy or frequentism. You badly misunderstand
      - 5y ago
JohnHaugeland 5y ago
The entire point of non-significant is that you can not say that anything happened Authors are making an extremely serious mistake here and you shouldn't be pushing back
robotresearcher 5y ago
(Next day reply, after reading this discussion a couple of times) > "432 Hz tuned music was associated with a slight decrease of mean (systolic and diastolic) blood pressure values (although not significant)" I can see why this line is fine to some and bothersome to others. Strictly, it's just describing their results. No problem. The numbers are what they are. On the other hand, why draw attention to the difference in means, when they are about to tell you not to take it very seriously? This version avoids that: "432 Hz tuned music was not associated with a significant difference in mean (systolic and diastolic) blood pressure values" Maybe it's my age, origin, or personality, but I prefer this version.

robotresearcher 5y ago

"Not significant" means that the probability is >=5% their result was obtained by chance. We've settled as a community on a convention that we don't claim an effect is real until it is supported by data ("statistically significant") ie. <5% likely to be explained by chance in your results. "Significant" does not mean big or important in this context. It means better than 5% unlikely to be (un)lucky data.

jjcon 5y ago
They didn't claim it was real did they? Just read out the result which was lower but not in a significant way. I've read hundreds of papers do the same.
- jhgb 5y ago
  > which was lower but not in a significant way But if "result [was] lower but not in a significant way" means "result was not proven to be lower", how does saying "lower but not really lower" make ever any sense? It seems to me that such nonsensical formulation ought not to be ever used by anyone.
  - jjcon 5y ago
    Because significance thresholds can vary pretty dramatically. Plenty of experiments done in physics for instance have reported results even though they didn't yet reach a 5 sigma threshold (3x10e-7). In physics something can be highly highly likely but still not 'significant' enough to warrant a discovery. They simply couch it as, hey this was the result and even though it isn't 'significant' the high likelihood may warrant additional research here. Reporting a binary significant/not significant is far less useful.
- robotresearcher 5y ago
  "was lower but not in a significant way" By convention, this means "indistinguishable from", so reporting that it is lower is an unsupported claim. They would be equally justified in reporting that it was higher, ie. not at all.
  - jjcon 5y ago
    It was lower though, just not significantly so (depending on your threshold for significance) - that's the standard way of reporting it. You can't just chunk part of the sentence and take issue with it, the sentence in it's entirety is accurate.
    - robotresearcher 5y ago
      Yes I agree that the sentence is accurate. Perhaps I’m too zealous about it, and maybe the conventions vary, but I was trained to avoid using words like ‘lower’ here.
kqr 5y ago
The threshold for significance lies in the eye of the beholder. A particle physicist might not be satisfied with anything over 0.01 %. A social scientist might be happy to see 10 %. The 5 % number you mention is completely arbitrary and often woefully inappropriate. Look at it from a betting perspective. Can you earn more than 10 × your investment if the null hypothesis is false? Then anything less likely than 10 % is significant.
- robotresearcher 5y ago
  It's a convention for scientific reporting. Your trades are not bound by this convention. The parameter value is not arbitrary. It's a convention arrived at after hundreds of years. If it were arbitrary, p=0.999 or p=0.00001 would be just as good. We've settled on p=0.05 being usefully convincing but not crazy demanding to obtain by experiment with noisy measurements.
  - aaplok 5y ago
    Null hypothesis testing was invented less than 100 years ago by Fisher, who completely arbitrarily picked 0.05 [0]. That value was not arrived at through wisdom of experience, and certainly not after hundreds of years of practice. Though it has now indeed become conventional to test with p=0.05, there is nothing wrong with reporting an effect that fails the null hypothesis test. At least that is the position of the American Statistical Association [1]. [0] https://www.cantorsparadise.com/what-is-the-meaning-of-p-value-and-why-is-it-0-05-5ae45644e7f6 https://www.cantorsparadise.com/what-is-the-meaning-of-p-val... [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5017929/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5017929/
    - robotresearcher 5y ago
      Thanks for these refs. I read [1] carefully and I take your point that it’s ok to strictly report whatever the data says. On the value itself, we are quibbling about the meaning of ‘arbitrary’: Fisher certainly could have chosen another value, but not all values would be considered useful. Some expertise about the nature of real world data and the minds of statisticians is encoded in the chosen value. If I propose that we change the convention to use 1e-12 instead and you think ‘that’s too small, I prefer it the way it is’, then it’s not arbitrary in the sense I mean.
david-gpu 5y ago
> "Not significant" means that the probability is >=5% their result was obtained by chance. Ackchually... p-value represents the probability that results like these would be observed even if there was no difference between the two choices, simply due to chance.

mlindner 5y ago

"Non-significant" in scientific means "within the error bars". It means that it is indistinguishable from randomness.

kqr 5y ago
That's a bit of an oversimplification that leaves out important qualifiers. "Non-significant" usually means it's indistinguishable from a null hypothesis when a certain level of randomness is allowed in a single, isolated trial. Who picks the null hypothesis? How is it picked, i.e. why does that specific hypothesis get favourable treatment? What level of randomness should one allow? What does it even mean for an experiment to be a single, isolated trial? How can anything be? Those are critical questions to understand the concept, and your explanation just pretends they don't exist.
- tragictrash 5y ago
  You may be technically correct, but in real life where the rest of us live, it means this study can't be used to draw any conclusions.
  - kqr 5y ago
    That's another fallacy of frequentist reasoning, that we have to draw definitive conclusions from evidence. That something is definitely false until we have "statistical significance" where it all of a sudden becomes definitely true. In real life, to borrow your description, we can hold varying levels of belief in statements depending on how strong the evidence is, and the magnitude of the payoff in the various cases. Maybe the probability of the result in the study in question is 51 %. That's still more than 50 %. Whether that difference is meaningful to you is not something someone else can decide.
    - JohnHaugeland 5y ago
      This is not about fallacy or frequentism. You badly misunderstand
    - 5y ago

JohnHaugeland 5y ago

The entire point of non-significant is that you can not say that anything happened Authors are making an extremely serious mistake here and you shouldn't be pushing back

robotresearcher 5y ago

(Next day reply, after reading this discussion a couple of times) > "432 Hz tuned music was associated with a slight decrease of mean (systolic and diastolic) blood pressure values (although not significant)" I can see why this line is fine to some and bothersome to others. Strictly, it's just describing their results. No problem. The numbers are what they are. On the other hand, why draw attention to the difference in means, when they are about to tell you not to take it very seriously? This version avoids that: "432 Hz tuned music was not associated with a significant difference in mean (systolic and diastolic) blood pressure values" Maybe it's my age, origin, or personality, but I prefer this version.