4 ms·

That sounds like regression. If this problem of regression occurs as regularly as your quote implies then the fault is not in these proposed alternatives, or e

That sounds like regression. If this problem of regression occurs as regularly as your quote implies then the fault is not in these proposed alternatives, or even in the likely faulty existing model, but in the gaping wide holes for testing these things quickly and objectively. That is why us dumb software guys have test automation.

bubblyworld 2y ago
I think automated hypothesis testing against new data in science is itself an incredibly difficult problem. Every experiment has its own methodology and particular interpretation, often you need to custom build models for your experimental setup to test a given hypothesis, there are lots of data cleanup and aggregation steps that don't generalise, etc. My partner is in neuroscience, for instance, and merging another lab's data into their own workflows is a whole project unto itself. Test automation in the software context is comparatively trivial. Formal systems make much better guarantees than the universe. (not to say I think it's a bad idea - it would be incredible! - but perhaps the juice isn't worth the squeeze?)
- austin-cheney 2y ago
  > Every experiment has its own methodology That is bias. Bias is always an implicit default in any initiative and requires a deliberate concerted effort to identify. None of what you said is unique to any form of science or engineering. Perhaps the only thing about this unique to this field of science, as well as microbiology, is the shear size and diversity of the data. From an objective perspective test automation is not more or less trivial to any given subject. The triviality of testing is directly determined by the tests written and their quality (speed and reproducibility). The juice is always worth the squeeze. Its a business problem that can be answered with math in consideration of risk, velocity, and confidence.
  - bubblyworld 2y ago
    Respectfully I disagree - the situation is far more complex in science than software engineering disciplines. I agree that different tests require different amounts of effort (obviously), but even the simplest "unit tests" you could conceive of for scientific domains are very complex, as there's no standard (or even unique) way to translate a scientific problem into a formally checkable system. Theories are frameworks within which experiments can be judged, but this is rarely unambiguous, and often requires a great deal of domain-specific knowledge - in analogy to programming it would be like the semantics of your language changing with every program you write. On the other hand, any programmer in a modern language can add useful tests to a codebase with (relatively) little effort. We are talking hours versus months or even years here! The experiment informs the ontology which informs the experiment. I don't think this is reducible to bias, although that certainly exists. Rather to me it's inherent uncertainty in the domain that experiments seek to address. Business practice, as you use the term, evolved to serve very different needs. Automated testing is useful for building software, but that effort may be better spent in science developing new experiments and hypotheses. It's very much an open problem whether the juice is worth the squeeze - in fact the lack of such efforts is (weak) evidence that it might not be. Scientists are not stupid.
    - austin-cheney 2y ago
      > We are talking hours versus months or even years here! That is why there are engineers that specialize in how to perform testing so that it doesn't take so long. For example long tests don't need to run at all if more critical short tests fail. The problems you describe for astrophysics are not unique to astrophysics even if the scale, size, and diversity of the data is so unique. Likewise, all the excuses I hear to avoid testing are the very same excuses software developers make. The reality is that these are only 25% valid. On their face these excuses are complete garbage, but a validation cannot occur faster than the underlying system allows. If the system being tested is remarkably slow then any testing upon it will be, at best, just as remarkably slow. That is not a fault of the test or the testing, but is entirely the fault of that underlying system.
      - bubblyworld 2y ago
abdullahkhalids 2y ago
You are oversimplifying science, especially theoretic physics. At the point where we are, there are neither any quick/cheap tests, and there is no objectivity. The space of possible correct theories is infinite, and humans are simply not smart enough to come up frameworks to objectively truncate the space. If we were, we would have made progress already. There is a lot of subjectivity and art to designing good experiments, not to mention a lot of philosophical insight. I know a lot of scientists deny the role of philosophy in science, but I see all the top physicists in my fields liberally use philosophy - not philosopher type philosophy but physicist type philosophy - to guide their scientific exploration.
- austin-cheney 2y ago
  I am not. You are using bias as an excuse to qualify poor objectivity. I am fully aware that astrophysics contains a scale and diversity of data beyond my imagination, but that volume of data does not excuse an absence of common business practices. > The space of possible correct theories is infinite That is not unique to any form of science, engineering, or even software products. > and humans are simply not smart enough to... That is why test automation is a thing.
- naasking 2y ago
  > and humans are simply not smart enough to come up frameworks to objectively truncate the space. We are, but some people stubbornly resist such things. For instance, MOND reproducing the Tully-Fisher relation and being unexpectedly successful at making many other predictions suggests that any theories purporting to explain dark matter/gravitational anomalies should probably have MOND-like qualities in some limit. That would effectively prune the space of possible theories. Instead, they've gone in the complete opposite direction, basically ignoring MOND and positing different matter distributions just to fit observations, while MOND, against all odds since it's not ultimately correct, continues to make successful predictions we're now seeing in JWST data.
  - abdullahkhalids 2y ago
    Note that when I say humans, I mean how the whole worldwide social institution/network of scientists, with all its inherent hierarchies and politics. Because that is the social network that ultimately comes to some consensus on what the best theories at any given moment are. Its indeed possible, in fact "necessary" that some individual scientists within this network by luck or brains come up with much better theories, but sometimes those theories are not accepted by others or they are. But ultimately all that matters is the consensus.

bubblyworld 2y ago

I think automated hypothesis testing against new data in science is itself an incredibly difficult problem. Every experiment has its own methodology and particular interpretation, often you need to custom build models for your experimental setup to test a given hypothesis, there are lots of data cleanup and aggregation steps that don't generalise, etc. My partner is in neuroscience, for instance, and merging another lab's data into their own workflows is a whole project unto itself. Test automation in the software context is comparatively trivial. Formal systems make much better guarantees than the universe. (not to say I think it's a bad idea - it would be incredible! - but perhaps the juice isn't worth the squeeze?)

austin-cheney 2y ago
> Every experiment has its own methodology That is bias. Bias is always an implicit default in any initiative and requires a deliberate concerted effort to identify. None of what you said is unique to any form of science or engineering. Perhaps the only thing about this unique to this field of science, as well as microbiology, is the shear size and diversity of the data. From an objective perspective test automation is not more or less trivial to any given subject. The triviality of testing is directly determined by the tests written and their quality (speed and reproducibility). The juice is always worth the squeeze. Its a business problem that can be answered with math in consideration of risk, velocity, and confidence.
- bubblyworld 2y ago
  Respectfully I disagree - the situation is far more complex in science than software engineering disciplines. I agree that different tests require different amounts of effort (obviously), but even the simplest "unit tests" you could conceive of for scientific domains are very complex, as there's no standard (or even unique) way to translate a scientific problem into a formally checkable system. Theories are frameworks within which experiments can be judged, but this is rarely unambiguous, and often requires a great deal of domain-specific knowledge - in analogy to programming it would be like the semantics of your language changing with every program you write. On the other hand, any programmer in a modern language can add useful tests to a codebase with (relatively) little effort. We are talking hours versus months or even years here! The experiment informs the ontology which informs the experiment. I don't think this is reducible to bias, although that certainly exists. Rather to me it's inherent uncertainty in the domain that experiments seek to address. Business practice, as you use the term, evolved to serve very different needs. Automated testing is useful for building software, but that effort may be better spent in science developing new experiments and hypotheses. It's very much an open problem whether the juice is worth the squeeze - in fact the lack of such efforts is (weak) evidence that it might not be. Scientists are not stupid.
  - austin-cheney 2y ago
    > We are talking hours versus months or even years here! That is why there are engineers that specialize in how to perform testing so that it doesn't take so long. For example long tests don't need to run at all if more critical short tests fail. The problems you describe for astrophysics are not unique to astrophysics even if the scale, size, and diversity of the data is so unique. Likewise, all the excuses I hear to avoid testing are the very same excuses software developers make. The reality is that these are only 25% valid. On their face these excuses are complete garbage, but a validation cannot occur faster than the underlying system allows. If the system being tested is remarkably slow then any testing upon it will be, at best, just as remarkably slow. That is not a fault of the test or the testing, but is entirely the fault of that underlying system.
    - bubblyworld 2y ago

abdullahkhalids 2y ago

You are oversimplifying science, especially theoretic physics. At the point where we are, there are neither any quick/cheap tests, and there is no objectivity. The space of possible correct theories is infinite, and humans are simply not smart enough to come up frameworks to objectively truncate the space. If we were, we would have made progress already. There is a lot of subjectivity and art to designing good experiments, not to mention a lot of philosophical insight. I know a lot of scientists deny the role of philosophy in science, but I see all the top physicists in my fields liberally use philosophy - not philosopher type philosophy but physicist type philosophy - to guide their scientific exploration.

austin-cheney 2y ago
I am not. You are using bias as an excuse to qualify poor objectivity. I am fully aware that astrophysics contains a scale and diversity of data beyond my imagination, but that volume of data does not excuse an absence of common business practices. > The space of possible correct theories is infinite That is not unique to any form of science, engineering, or even software products. > and humans are simply not smart enough to... That is why test automation is a thing.
naasking 2y ago
> and humans are simply not smart enough to come up frameworks to objectively truncate the space. We are, but some people stubbornly resist such things. For instance, MOND reproducing the Tully-Fisher relation and being unexpectedly successful at making many other predictions suggests that any theories purporting to explain dark matter/gravitational anomalies should probably have MOND-like qualities in some limit. That would effectively prune the space of possible theories. Instead, they've gone in the complete opposite direction, basically ignoring MOND and positing different matter distributions just to fit observations, while MOND, against all odds since it's not ultimately correct, continues to make successful predictions we're now seeing in JWST data.
- abdullahkhalids 2y ago
  Note that when I say humans, I mean how the whole worldwide social institution/network of scientists, with all its inherent hierarchies and politics. Because that is the social network that ultimately comes to some consensus on what the best theories at any given moment are. Its indeed possible, in fact "necessary" that some individual scientists within this network by luck or brains come up with much better theories, but sometimes those theories are not accepted by others or they are. But ultimately all that matters is the consensus.