5 ms·

I'll concede that the story is different when you're relying on commercial software that you've purchased to implement an algorithm correctly, but in the limite

I'll concede that the story is different when you're relying on commercial software that you've purchased to implement an algorithm correctly, but in the limited case of replicating results (whether they're flawed or not), the onus really should be on the user to record what version of software they're using and how they configured it. The article does a lot of gymnastics to make software correctness and research reproducibility feel like the same issue.

s-shellfish 8y ago
Versioning/Configuration in industry that has automated some aspect of their build process to include external dependencies leads to label accumulation (I'm intuiting that this is greater than exponential growth - assuming there exist dependencies for a system A that are dependent on other dependencies that system A is dependent on). The problem is you can't document everything while also trying to automate everything. It's always a balance of figuring out which information is most important, and if 'oneself' were to attempt to construct a model that represents everything, 'oneself' would simply be stuck in a catatonic state of analysis paralysis, in perpetuity and invariably, until the heat death of the universe. Yes, the onus is on the user, but who is the 'user' in the industry of development? How do you maintain structure in records for information systems that are fundamentally, at core, completely opened%? I believe this problem maps back to the science of collecting and verifying the veracity of information, up to isomorphism (quite literally, this is the definition of the graph isomorphism problem). % The intrinsic nature of science and discovery is 'we discover new things because we assume there is more to know (~ we don't know everything)'. Therefore, all components of such a system are subject to change. There is nothing that can be said to be permanent in knowledge discovery systems where the premise of discovery rests on 'assertions' that can not be provably demonstrated to be factual AND absolute.
- emiliobumachar 8y ago
  "Versioning/Configuration in industry that has automated some aspect of their build process to include external dependencies leads to label accumulation" Okay, but most software shops shove all those labels into their product version, usually something simple like "ToolFoo 3.4.11". I think that this numbering and basic "here's the binary, run at your own risk and don't call me" legacy support are enough on the vendor's part. And I agree with gp that if the researcher fails to document the top-level tool version they're using, plus the changes they made to default configuration, then it's the researcher's fault. Don't you agree?
  - s-shellfish 8y ago
    I don't know if I can agree with that. Pretend that all of that has been automated - you have some dependency management system that auto-documents the versions of all software you are using. How do you check that all of that documentation is correct across all systems with one singular 'system'? There's the obvious - make everyone perfect and know everything they need to know for the work they do. The individual is responsible. Okay, but how do you check that all of that is functioning the way it's supposed to? It literally simply creates a copy of the original problem. There's the idea to put the onus on the individual researcher or developer for managing their specific 'part', but the problem is that their specific part is coupled with other parts they do not have a complete understanding of, and may never have a complete understanding of. It's literally uncertain whether it's possible to actually have a complete understanding. Oneself can tell oneself all they want "I know all of the things I need to know" but I think when oneself does that, oneself forgets how to know they might not know all of the things they need to know. But that might just be the insanity inside of me speaking. I know I don't know everything, even the things I think I'm supposed to know, I don't know if I know them. Because as a developer working with other developers, code changes, interpretation of code changes, there's always variation in understanding 'the model' everyone is working off of and working towards. This seems to influence things, and what if all that does is converge understanding towards self proving (e.g. reconstructing a duplicate of the original data) but it's done in a way where the simple mere fact that everyone has agreed to hold those 'models/beliefs' as strongest is what causes those beliefs to manifest externally? Is that actually provably correct? I don't think so. I just think people forget how to know the difference between what is a 'bug/feature' of the nature of knowing and thinking, and what is an error. This may be tangential to the nature of software correctness and code, but 1. we don't know how programs get used in the future unless the person doing the programming is also the only person using the program. and 2. This bug/feature nature of knowing and thinking is what allows research in things like the social sciences% to not finalize permanently into some static model that is unchanging til the end of time. You can balance both of these awarenesses but, it's not a trivial understanding. It's very, very complicated and it makes the most mundane details of life seem like sheer insanity. % referring to > The replication crisis that continues to engulf the social sciences% is largely concerned with the known problems,
  - 8y ago
lomnakkus 8y ago
> the onus really should be on the user to record what version of software they're using and how they configured it. Recording version, etc. is fine, but what use is it if it's not actually possible to install that software to reproduce whatever results were obtained? (This also extends to hardware, btw., but VMs can probably attain enough fidelity for that to not be an issue as long as the physical media are still readable.) Example: Reasearch done on Windows XP at some point in time past SP3. Unless you're prepared well ahead of EoL on Windows XP there's really no way to reproduce that environment. (I'll definitely grant that it's probably unlikely that the obscure details of the OS have that much impact on whatever results were obtained by programs running atop it, but it's at least moderately plausible.)

s-shellfish 8y ago

Versioning/Configuration in industry that has automated some aspect of their build process to include external dependencies leads to label accumulation (I'm intuiting that this is greater than exponential growth - assuming there exist dependencies for a system A that are dependent on other dependencies that system A is dependent on). The problem is you can't document everything while also trying to automate everything. It's always a balance of figuring out which information is most important, and if 'oneself' were to attempt to construct a model that represents everything, 'oneself' would simply be stuck in a catatonic state of analysis paralysis, in perpetuity and invariably, until the heat death of the universe. Yes, the onus is on the user, but who is the 'user' in the industry of development? How do you maintain structure in records for information systems that are fundamentally, at core, completely opened%? I believe this problem maps back to the science of collecting and verifying the veracity of information, up to isomorphism (quite literally, this is the definition of the graph isomorphism problem). % The intrinsic nature of science and discovery is 'we discover new things because we assume there is more to know (~ we don't know everything)'. Therefore, all components of such a system are subject to change. There is nothing that can be said to be permanent in knowledge discovery systems where the premise of discovery rests on 'assertions' that can not be provably demonstrated to be factual AND absolute.

emiliobumachar 8y ago
"Versioning/Configuration in industry that has automated some aspect of their build process to include external dependencies leads to label accumulation" Okay, but most software shops shove all those labels into their product version, usually something simple like "ToolFoo 3.4.11". I think that this numbering and basic "here's the binary, run at your own risk and don't call me" legacy support are enough on the vendor's part. And I agree with gp that if the researcher fails to document the top-level tool version they're using, plus the changes they made to default configuration, then it's the researcher's fault. Don't you agree?
- s-shellfish 8y ago
  I don't know if I can agree with that. Pretend that all of that has been automated - you have some dependency management system that auto-documents the versions of all software you are using. How do you check that all of that documentation is correct across all systems with one singular 'system'? There's the obvious - make everyone perfect and know everything they need to know for the work they do. The individual is responsible. Okay, but how do you check that all of that is functioning the way it's supposed to? It literally simply creates a copy of the original problem. There's the idea to put the onus on the individual researcher or developer for managing their specific 'part', but the problem is that their specific part is coupled with other parts they do not have a complete understanding of, and may never have a complete understanding of. It's literally uncertain whether it's possible to actually have a complete understanding. Oneself can tell oneself all they want "I know all of the things I need to know" but I think when oneself does that, oneself forgets how to know they might not know all of the things they need to know. But that might just be the insanity inside of me speaking. I know I don't know everything, even the things I think I'm supposed to know, I don't know if I know them. Because as a developer working with other developers, code changes, interpretation of code changes, there's always variation in understanding 'the model' everyone is working off of and working towards. This seems to influence things, and what if all that does is converge understanding towards self proving (e.g. reconstructing a duplicate of the original data) but it's done in a way where the simple mere fact that everyone has agreed to hold those 'models/beliefs' as strongest is what causes those beliefs to manifest externally? Is that actually provably correct? I don't think so. I just think people forget how to know the difference between what is a 'bug/feature' of the nature of knowing and thinking, and what is an error. This may be tangential to the nature of software correctness and code, but 1. we don't know how programs get used in the future unless the person doing the programming is also the only person using the program. and 2. This bug/feature nature of knowing and thinking is what allows research in things like the social sciences% to not finalize permanently into some static model that is unchanging til the end of time. You can balance both of these awarenesses but, it's not a trivial understanding. It's very, very complicated and it makes the most mundane details of life seem like sheer insanity. % referring to > The replication crisis that continues to engulf the social sciences% is largely concerned with the known problems,
- 8y ago

lomnakkus 8y ago

> the onus really should be on the user to record what version of software they're using and how they configured it. Recording version, etc. is fine, but what use is it if it's not actually possible to install that software to reproduce whatever results were obtained? (This also extends to hardware, btw., but VMs can probably attain enough fidelity for that to not be an issue as long as the physical media are still readable.) Example: Reasearch done on Windows XP at some point in time past SP3. Unless you're prepared well ahead of EoL on Windows XP there's really no way to reproduce that environment. (I'll definitely grant that it's probably unlikely that the obscure details of the OS have that much impact on whatever results were obtained by programs running atop it, but it's at least moderately plausible.)