8 ms·

Not all null results are created equal. There are interesting null results that get published and are well known. For example, Card & Kruger (1994) was a null

Not all null results are created equal. There are interesting null results that get published and are well known. For example, Card & Kruger (1994) was a null result paper showing that increasing the minimum wage has a null effect on employment rates. This result went against the common assumption that increasing wages will decrease employment at the time. Other null results are either dirty (e.g., big standard errors) or due to process problems (e.g., experimental failure). These are more difficult to publish because it's difficult to learn anything new from these results. The challenge is that researchers do not know if they are going to get a "good" null or a "bad" one. Most of the time, you have to invest significant effort and time into a project, only to get a null result at the end. These results are difficult to publish in most cases and can lead to the end of careers if someone is pre-tenure or lead to funding problems for anyone.

kazinator 1y ago
What matters for publication is a surprising result, not whether it confirms the main hypothesis or the null one. The "psychological null hypothesis" is that which follows the common assumption, whether that assumption states that there is a relationship between the variables or that there is not.
MITSardine 1y ago
Is that actually a null result though? That sounds like a standard positive result: "We managed to show that minimum wage has no effect on employment rate". A null result would have been: "We tried to apply Famous Theory to showing that minimum wage has no effect on employment rate but we failed because this and that".
- Analemma_ 1y ago
  No, because in theory a minimum wage increase could decrease the unemployment rate. If it does neither, that’s a null result.
  - mitthrowaway2 1y ago
    That's sort of like saying "I measured acceleration due to gravity by dropping a bowling ball off the Tower of Pisa, and got a null result: the ball simply hovered in midair." If the result is very surprising and contradicts established theory, I wouldn't consider it a null result, even if some parameter numerically measures as zero.
    - Analemma_ 1y ago
      The question being asked is, "what is the correlation between these two variables": is it positive, negative, or zero (null). The null hypothesis is the baseline you start from because the overwhelming majority of variables from physical observations are uncorrelated (e.g. what is the correlation between "how many people in the treatment group of this clinical trial made a full recovery" and "the price of eggs in Slovakia"). Measurements of some physical quantity are a different kind of experiment, you cannot phrase it as a question about the correlation between two variables. Instead you take measurements and put error bars on them (unless what you're measuring is an indirect proxy for the actual quantity, in which case the null hypothesis and p-value testing does become relevant again).
      - mitthrowaway2 1y ago
        In fact, you can express measurement of g as a linear correlation between y-y0 and (t-t0)^0.5.
- iamjfu 1y ago
  From a related nature article (https://www.nature.com/articles/d41586-024-02383-9 https://www.nature.com/articles/d41586-024-02383-9), "null or negative results — those that fail to find a relationship between variables or groups, or that go against the preconceived hypothesis." According to this definition, I think both examples you provided are null results. Particularly here, where the context is the file drawer problem.
- jruohonen 1y ago
  > Is that actually a null result though? The above is a good point but I would extend it further. I mean, philosophically, you get a positive result from a negative (null) result by merely changing your hypotheses (e.g., something should not cause something else).
  - tempusalaria 1y ago
    if you are p testing this isn’t the case. A positive result is a much stronger assertion
    - jruohonen 1y ago
      Sure and of course you shouldn't change your hypotheses. That's kind of the whole purpose of pre-registrations at a meta-level of things.
- roenxi 1y ago
  Looking at the definition in the article, it definitely is a null result. However the example does illustrate that 'a null result' probably isn't a very interesting thing to talk about because it covers too many types of result. I think what people on HN actually want to track is something more like 'a boring null result'. The real question is whether there a process that is reliably being followed and leading to research that matches reality (where a statistically significant result suggests something is real) or is scientific publishing highly biased towards odd results (where studies that muck up or get lucky with statistical noise are over-represented). In this case we would expect some studies of the minimum wage to show it increases employment regardless of what the effect of wage rises is in the general case - eg, some official raised the minimum wage while a sector went into a boom for unrelated and coincidental reasons.
- im3w1l 1y ago
  Null means nothing, zero. In the context of scientific articles a null result means that the difference is zero. What difference? Well it depends. It could be difference between doing something and not doing it. The difference between before and after some intervention. Or perhaps the difference between two different actions. In this case the difference between before and after raising the minimum wage. Furthermore, the thing with a null result is that it's always dependant on how sensitive your study is. A null result is always of the form "we can't rule out that it's 0". If the study is powerful then a null result will rule out a large difference, but there is always the possibility that there is a difference too small to detect.
  - ktallett 1y ago
    Null means something doesn't exist and proving something doesn't exist can at times especially in certain sciences be as valuable as proving something does exist. Especially if it stops the rest of the field chasing the pot of gold at the end of the rainbow.
hatthew 1y ago
Also, doesn't a null result mean "we weren't able to measure an effect" rather than "there is no effect"? That's a pretty big difference in how important something is.
- reliabilityguy 1y ago
  Sure. But publishing this result with your experimental methodology may help others to refine the experiment and get a better answer. It is absolutely shameful that negative results are almost never published. I sure that a lot of money and effort wasted by repeating the same dead-end experiments by many research groups just because there is no paper that said: “we tried it this way, it didn’t work”.
  - deleted 1y ago
    [deleted]
  - 9rx 1y ago
    1. Papers are written to invite replication — the most important part of the scientific process. It is already difficult to compel replication even when you only put the most promising research in people's faces. Now you want them to have to also sift through thousands of entrants that have almost no reason for replication attempts? 2. It's easy to call it shameful when it isn't you who has to do the work. If you are like most other normally functioning people, you no doubt perform little experiments every day that end up going nowhere. How many have you written papers for? I can confidently say zero. I've got better things to do.
    - jraph 1y ago
      The lack of null result publication is a real issue for research and scientific knowledge in general, including for accurate positive results (what if once you see a positive result by statistical chance but the experiment was actually conducted before you by 10 other people but that wasn't published because the results were null?) > Papers are written to invite replication They are written to share knowledge, new discoveries. We hope that they are replicated. > It's easy to call it shameful when it isn't you who has to do the work. We are not judging anyone, we are qualifying the situation. And we are speaking of publishing results for experiments that have already conducted, not voluntarily making up null cases and doing the related experiments. It wouldn't make sense and would be harmful. But if you did an experiment that produced a null result, not publishing is a loss of knowledge. Again, we are not judging anybody, it's a failure of the publishing system. But this cannot change if nobody points it out. I'd have troubles understanding a researcher not acknowledging today that the lack of null result publication is an issue. It would show a lack of perspective IMHO. And for someone acknowledging that there's an issue, pushing back is not the right stance.
Natsu 1y ago
> Card & Kruger (1994) was a null result paper showing that increasing the minimum wage has a null effect on employment rates. This result went against the common assumption that increasing wages will decrease employment at the time. I had to look that up, because more precisely, it showed that a particular minimum wage increase in NJ from $4.25 to $5.05 didn't increase unemployment in 410 particular fast food joints in 1989-1990 - https://davidcard.berkeley.edu/papers/njmin-aer.pdf https://davidcard.berkeley.edu/papers/njmin-aer.pdf - not that "increasing the minimum wage has a null effect on employment rates" at all, ever, no matter what. It's not as if increasing the minimum wage to something impossible to afford like $100 trillion wouldn't force everyone to get laid off, but nobody generally cares about the limiting case like that, as that's relatively unlikely. The interesting part is non-linearity in the response, seeing where and how much employment rates might change given the magnitude of a particular minimum wage increase, what sort of profit margins the affected industries have, elasticity of the demand for labor and other adaptations by businesses in response to increases, not whether it's a thing that can happen at all. And we're seeing a lot more such adaptions these days. There are McDonalds around here that have computers in the lobby where you input your order yourself and I've gotten drones to deliver my food instead of DoorDash drivers. That kind of thing was not yet practical back in 1989, when I remember using an Apple ][ GS and it's not clear that findings like this should be relied upon too heavily given that some of the adaptations available to businesses now were not practical back then, especially when technology may change that even more in the future.
deleted 1y ago
[deleted]
SubiculumCode 1y ago
One of my most celebrated recent "null result" was published in Biological Psychiatry, a journal with an impact Factor about 13 last I checked. Even though it was a null result, it was decently powered longitudinal study that provided evidence inconsistent with the prevailing view in autism research. Even if there exists a difference that we failed to detect, it probably much smaller than what others had expected on the basis of second-grade evidence (cross-sectional group differences).
aydyn 1y ago
I agree. More succinctly, it confounds two types of results: a null result due to statistical noise (big error bars, experiment failure) and a null result where the null model is more likely (actually, the effect doesnt exist). Like many things in statistics, this is solved by Bayesian analysis: instead of asking if we can reject the null hypothesis, the question should be which model is more likely, the null model or the alternate model.

kazinator 1y ago

What matters for publication is a surprising result, not whether it confirms the main hypothesis or the null one. The "psychological null hypothesis" is that which follows the common assumption, whether that assumption states that there is a relationship between the variables or that there is not.

MITSardine 1y ago

Is that actually a null result though? That sounds like a standard positive result: "We managed to show that minimum wage has no effect on employment rate". A null result would have been: "We tried to apply Famous Theory to showing that minimum wage has no effect on employment rate but we failed because this and that".

Analemma_ 1y ago
No, because in theory a minimum wage increase could decrease the unemployment rate. If it does neither, that’s a null result.
- mitthrowaway2 1y ago
  That's sort of like saying "I measured acceleration due to gravity by dropping a bowling ball off the Tower of Pisa, and got a null result: the ball simply hovered in midair." If the result is very surprising and contradicts established theory, I wouldn't consider it a null result, even if some parameter numerically measures as zero.
  - Analemma_ 1y ago
    The question being asked is, "what is the correlation between these two variables": is it positive, negative, or zero (null). The null hypothesis is the baseline you start from because the overwhelming majority of variables from physical observations are uncorrelated (e.g. what is the correlation between "how many people in the treatment group of this clinical trial made a full recovery" and "the price of eggs in Slovakia"). Measurements of some physical quantity are a different kind of experiment, you cannot phrase it as a question about the correlation between two variables. Instead you take measurements and put error bars on them (unless what you're measuring is an indirect proxy for the actual quantity, in which case the null hypothesis and p-value testing does become relevant again).
    - mitthrowaway2 1y ago
      In fact, you can express measurement of g as a linear correlation between y-y0 and (t-t0)^0.5.
iamjfu 1y ago
From a related nature article (https://www.nature.com/articles/d41586-024-02383-9 https://www.nature.com/articles/d41586-024-02383-9), "null or negative results — those that fail to find a relationship between variables or groups, or that go against the preconceived hypothesis." According to this definition, I think both examples you provided are null results. Particularly here, where the context is the file drawer problem.
jruohonen 1y ago
> Is that actually a null result though? The above is a good point but I would extend it further. I mean, philosophically, you get a positive result from a negative (null) result by merely changing your hypotheses (e.g., something should not cause something else).
- tempusalaria 1y ago
  if you are p testing this isn’t the case. A positive result is a much stronger assertion
  - jruohonen 1y ago
    Sure and of course you shouldn't change your hypotheses. That's kind of the whole purpose of pre-registrations at a meta-level of things.
roenxi 1y ago
Looking at the definition in the article, it definitely is a null result. However the example does illustrate that 'a null result' probably isn't a very interesting thing to talk about because it covers too many types of result. I think what people on HN actually want to track is something more like 'a boring null result'. The real question is whether there a process that is reliably being followed and leading to research that matches reality (where a statistically significant result suggests something is real) or is scientific publishing highly biased towards odd results (where studies that muck up or get lucky with statistical noise are over-represented). In this case we would expect some studies of the minimum wage to show it increases employment regardless of what the effect of wage rises is in the general case - eg, some official raised the minimum wage while a sector went into a boom for unrelated and coincidental reasons.
im3w1l 1y ago
Null means nothing, zero. In the context of scientific articles a null result means that the difference is zero. What difference? Well it depends. It could be difference between doing something and not doing it. The difference between before and after some intervention. Or perhaps the difference between two different actions. In this case the difference between before and after raising the minimum wage. Furthermore, the thing with a null result is that it's always dependant on how sensitive your study is. A null result is always of the form "we can't rule out that it's 0". If the study is powerful then a null result will rule out a large difference, but there is always the possibility that there is a difference too small to detect.
- ktallett 1y ago
  Null means something doesn't exist and proving something doesn't exist can at times especially in certain sciences be as valuable as proving something does exist. Especially if it stops the rest of the field chasing the pot of gold at the end of the rainbow.

hatthew 1y ago

Also, doesn't a null result mean "we weren't able to measure an effect" rather than "there is no effect"? That's a pretty big difference in how important something is.

reliabilityguy 1y ago
Sure. But publishing this result with your experimental methodology may help others to refine the experiment and get a better answer. It is absolutely shameful that negative results are almost never published. I sure that a lot of money and effort wasted by repeating the same dead-end experiments by many research groups just because there is no paper that said: “we tried it this way, it didn’t work”.
- deleted 1y ago
  [deleted]
- 9rx 1y ago
  1. Papers are written to invite replication — the most important part of the scientific process. It is already difficult to compel replication even when you only put the most promising research in people's faces. Now you want them to have to also sift through thousands of entrants that have almost no reason for replication attempts? 2. It's easy to call it shameful when it isn't you who has to do the work. If you are like most other normally functioning people, you no doubt perform little experiments every day that end up going nowhere. How many have you written papers for? I can confidently say zero. I've got better things to do.
  - jraph 1y ago
    The lack of null result publication is a real issue for research and scientific knowledge in general, including for accurate positive results (what if once you see a positive result by statistical chance but the experiment was actually conducted before you by 10 other people but that wasn't published because the results were null?) > Papers are written to invite replication They are written to share knowledge, new discoveries. We hope that they are replicated. > It's easy to call it shameful when it isn't you who has to do the work. We are not judging anyone, we are qualifying the situation. And we are speaking of publishing results for experiments that have already conducted, not voluntarily making up null cases and doing the related experiments. It wouldn't make sense and would be harmful. But if you did an experiment that produced a null result, not publishing is a loss of knowledge. Again, we are not judging anybody, it's a failure of the publishing system. But this cannot change if nobody points it out. I'd have troubles understanding a researcher not acknowledging today that the lack of null result publication is an issue. It would show a lack of perspective IMHO. And for someone acknowledging that there's an issue, pushing back is not the right stance.

Natsu 1y ago

> Card & Kruger (1994) was a null result paper showing that increasing the minimum wage has a null effect on employment rates. This result went against the common assumption that increasing wages will decrease employment at the time. I had to look that up, because more precisely, it showed that a particular minimum wage increase in NJ from $4.25 to $5.05 didn't increase unemployment in 410 particular fast food joints in 1989-1990 - https://davidcard.berkeley.edu/papers/njmin-aer.pdf https://davidcard.berkeley.edu/papers/njmin-aer.pdf - not that "increasing the minimum wage has a null effect on employment rates" at all, ever, no matter what. It's not as if increasing the minimum wage to something impossible to afford like $100 trillion wouldn't force everyone to get laid off, but nobody generally cares about the limiting case like that, as that's relatively unlikely. The interesting part is non-linearity in the response, seeing where and how much employment rates might change given the magnitude of a particular minimum wage increase, what sort of profit margins the affected industries have, elasticity of the demand for labor and other adaptations by businesses in response to increases, not whether it's a thing that can happen at all. And we're seeing a lot more such adaptions these days. There are McDonalds around here that have computers in the lobby where you input your order yourself and I've gotten drones to deliver my food instead of DoorDash drivers. That kind of thing was not yet practical back in 1989, when I remember using an Apple ][ GS and it's not clear that findings like this should be relied upon too heavily given that some of the adaptations available to businesses now were not practical back then, especially when technology may change that even more in the future.

deleted 1y ago

[deleted]

SubiculumCode 1y ago

One of my most celebrated recent "null result" was published in Biological Psychiatry, a journal with an impact Factor about 13 last I checked. Even though it was a null result, it was decently powered longitudinal study that provided evidence inconsistent with the prevailing view in autism research. Even if there exists a difference that we failed to detect, it probably much smaller than what others had expected on the basis of second-grade evidence (cross-sectional group differences).

aydyn 1y ago

I agree. More succinctly, it confounds two types of results: a null result due to statistical noise (big error bars, experiment failure) and a null result where the null model is more likely (actually, the effect doesnt exist). Like many things in statistics, this is solved by Bayesian analysis: instead of asking if we can reject the null hypothesis, the question should be which model is more likely, the null model or the alternate model.