5 ms·

> not just for absolute correctness, but perhaps even for consistency with past versions (a novel concept) Sorry, can you explain what you mean by this furthe

> not just for absolute correctness, but perhaps even for consistency with past versions (a novel concept) Sorry, can you explain what you mean by this further?

_bxg1 8y ago
Normally, software functions are only tested against what is considered to be the universally correct outcome. The wiggle room - i.e., undefined aspects of the behavior - within that definition of "correct" is not tested. It would be interesting for testing of scientific software to incorporate a second metric: "how close is it to what it was last time (assuming last time was also 'correct')?" For the sake of reproducibility, maybe it's meaningful whether or not the current version of correctness is in line with the previous version of correctness. There's virtually no other domain in which you'd want to test software this way, which is what makes it novel and interesting. For example, maybe a floating point operation rounds the very last digit down. Then through code changes, it starts truncating the very last digit instead. Your tests may still consider this result "correct", but it might be worth considering the fact that it differed from the previous "correct" result.
- _bxg1 8y ago
  Or, maybe your "equals" function, when given two floats (floats are typically considered not-comparable in most languages), happens to return true in a particular case, then later on through code changes, it ends up returning false for the same two numbers. Since <float> == <float> is undefined, neither behavior is "incorrect", but it might be worthwhile to at least keep the behavior consistent.
- s-shellfish 8y ago
  Specifically for social sciences, wouldn't this mechanic be able to be exploited in the same way p-values are? In the case of software testing, I think I agree with Dijkstra on this one, except for different reasons: > We could, for instance, begin with cleaning up our language by no longer calling a bug a bug but by calling it an error. It is much more honest because it squarely puts the blame where it belongs, viz. with the programmer who made the error. The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation. The nice thing of this simple change of vocabulary is that it has such a profound effect: while, before, a program with only one bug used to be "almost correct", afterwards a program with an error is just "wrong" (because in error). I don't think 'blame' is the right word, or calling the 'programmer' who made the error intellectually dishonest. A fuzzy 'language' might be helpful for 'developers' who are not quite developers. A developer would categorize such an occurrence in discrete terminology - WARNING, ERROR, INFO, DEBUG, coupled with checks and tests throughout the code - in order to retain discrete structure across code, which is much easier to reason about mathematically (and much easier to isolate the source of an error), when such discrete objects and discrete structures stack. The problem is you still need these structures in the code. You still need to include the tests in the code. You still need to know what to look for, in the code. You still need to know where the code might change, before you know where the code has changed. I don't think it's a futile effort, but I suppose I'm scratching my head a little on how this solves the problem? It's much simpler to deal with a compilation error, or any kind of explicit error that prevents some aspect of the program from running in a way that is defined as invalid. It's annoying, but it at least says "definitely broken code", instead of "maybe broken code?" which may run for a very long time, accumulating "maybe errors?". IMHO, that behavior can be perverted quite easily, because, some "maybe errors" can be ignored, other "maybe errors" can be focused on, depending on who is looking, and who isn't. I honestly think if social scientists are going to complain about code, they need to learn how to code. A computer isn't a magic fantasy box that just confirms beliefs. It takes effort, skill, hard work, and honestly, dealing with bugs you can't always determine the origin of is very humbling (you, other developer?), so they should learn some of that too. And I'm sure they'd say "we already do!" but there we are again, going full circle, choosing to believe what we can't prove, but tricking ourselves into finding a magical way to prove what we wanted to prove, about ourselves, about others, etc. Bugs will happen. It's better when they are clearly defined, because at least the source of the error can be isolated, tested, identified, and corrected. Stacks on stacks of 'maybe error?' can lead to it literally conveying information that is the opposite of what it intends to. There needs to be some compassion for people who are developers. We can't know where all the bugs are all the time. The best we can do is make code as bug-free as we possibly can. Coding is just hard! You think you get better at it over time, but it's really easy to be chugging along, coding coding coding, and all of a sudden - bug you didn't think of! And that just, it's humbling - and you know - this is your life, this is the path you chose. This is what you will be doing until you stop coding. Social scientists need to be nicer to developers. Same problems they deal with, just shifted. > Testing shows the presence, not the absence of bugs (Dijkstra)
  - perl4ever 8y ago
    I disagree with calling bugs "errors" that the programmer made. A program has a context, and bugs depend on the context, hence the old joke about "features". For a really simple example, suppose I write a program to do mail merge for myself and it has my name hard-coded in it. If you try to use it, now it has a bug, because it prints the wrong name. But it doesn't have a bug when I use it. And if the context determines whether a bug exists, one can deduce that even if a program is provably correct by some standard and works for a billion years, the context could then change and oops, it has a bug.
    - s-shellfish 8y ago
      It's called an error when it's being used in a way it wasn't intended to be used. That matters when future you forgets how you programmed your code.
- perl4ever 8y ago
  I have the impression that issues with a software version not matching a previous version are not only not novel, but have been plentiful with any kind of software since the beginning of time. Microsoft is proverbial for Windows being under the constraint of maintaining past behavior however imperfect. To wit, it's such a common thing that there is an xkcd for it: https://imgs.xkcd.com/comics/workflow.png https://imgs.xkcd.com/comics/workflow.png P.S. I was momentarily repressing my first hand experience, but in a past job, there were many times I wanted to fix things and wasn't allowed to, because if the results changed, it would upset someone. The cardinal, CYA rule, was if we've been reporting something wrong for years, it better be consistent forever. Which is really painful if you care about correctness and compulsively investigate it.
  - _bxg1 8y ago
    That sounds awful.

> not just for absolute correctness, but perhaps even for consistency with past versions (a novel concept) Sorry, can you explain what you mean by this further?

_bxg1 8y ago

Normally, software functions are only tested against what is considered to be the universally correct outcome. The wiggle room - i.e., undefined aspects of the behavior - within that definition of "correct" is not tested. It would be interesting for testing of scientific software to incorporate a second metric: "how close is it to what it was last time (assuming last time was also 'correct')?" For the sake of reproducibility, maybe it's meaningful whether or not the current version of correctness is in line with the previous version of correctness. There's virtually no other domain in which you'd want to test software this way, which is what makes it novel and interesting. For example, maybe a floating point operation rounds the very last digit down. Then through code changes, it starts truncating the very last digit instead. Your tests may still consider this result "correct", but it might be worth considering the fact that it differed from the previous "correct" result.

_bxg1 8y ago
Or, maybe your "equals" function, when given two floats (floats are typically considered not-comparable in most languages), happens to return true in a particular case, then later on through code changes, it ends up returning false for the same two numbers. Since <float> == <float> is undefined, neither behavior is "incorrect", but it might be worthwhile to at least keep the behavior consistent.
s-shellfish 8y ago
Specifically for social sciences, wouldn't this mechanic be able to be exploited in the same way p-values are? In the case of software testing, I think I agree with Dijkstra on this one, except for different reasons: > We could, for instance, begin with cleaning up our language by no longer calling a bug a bug but by calling it an error. It is much more honest because it squarely puts the blame where it belongs, viz. with the programmer who made the error. The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation. The nice thing of this simple change of vocabulary is that it has such a profound effect: while, before, a program with only one bug used to be "almost correct", afterwards a program with an error is just "wrong" (because in error). I don't think 'blame' is the right word, or calling the 'programmer' who made the error intellectually dishonest. A fuzzy 'language' might be helpful for 'developers' who are not quite developers. A developer would categorize such an occurrence in discrete terminology - WARNING, ERROR, INFO, DEBUG, coupled with checks and tests throughout the code - in order to retain discrete structure across code, which is much easier to reason about mathematically (and much easier to isolate the source of an error), when such discrete objects and discrete structures stack. The problem is you still need these structures in the code. You still need to include the tests in the code. You still need to know what to look for, in the code. You still need to know where the code might change, before you know where the code has changed. I don't think it's a futile effort, but I suppose I'm scratching my head a little on how this solves the problem? It's much simpler to deal with a compilation error, or any kind of explicit error that prevents some aspect of the program from running in a way that is defined as invalid. It's annoying, but it at least says "definitely broken code", instead of "maybe broken code?" which may run for a very long time, accumulating "maybe errors?". IMHO, that behavior can be perverted quite easily, because, some "maybe errors" can be ignored, other "maybe errors" can be focused on, depending on who is looking, and who isn't. I honestly think if social scientists are going to complain about code, they need to learn how to code. A computer isn't a magic fantasy box that just confirms beliefs. It takes effort, skill, hard work, and honestly, dealing with bugs you can't always determine the origin of is very humbling (you, other developer?), so they should learn some of that too. And I'm sure they'd say "we already do!" but there we are again, going full circle, choosing to believe what we can't prove, but tricking ourselves into finding a magical way to prove what we wanted to prove, about ourselves, about others, etc. Bugs will happen. It's better when they are clearly defined, because at least the source of the error can be isolated, tested, identified, and corrected. Stacks on stacks of 'maybe error?' can lead to it literally conveying information that is the opposite of what it intends to. There needs to be some compassion for people who are developers. We can't know where all the bugs are all the time. The best we can do is make code as bug-free as we possibly can. Coding is just hard! You think you get better at it over time, but it's really easy to be chugging along, coding coding coding, and all of a sudden - bug you didn't think of! And that just, it's humbling - and you know - this is your life, this is the path you chose. This is what you will be doing until you stop coding. Social scientists need to be nicer to developers. Same problems they deal with, just shifted. > Testing shows the presence, not the absence of bugs (Dijkstra)
- perl4ever 8y ago
  I disagree with calling bugs "errors" that the programmer made. A program has a context, and bugs depend on the context, hence the old joke about "features". For a really simple example, suppose I write a program to do mail merge for myself and it has my name hard-coded in it. If you try to use it, now it has a bug, because it prints the wrong name. But it doesn't have a bug when I use it. And if the context determines whether a bug exists, one can deduce that even if a program is provably correct by some standard and works for a billion years, the context could then change and oops, it has a bug.
  - s-shellfish 8y ago
    It's called an error when it's being used in a way it wasn't intended to be used. That matters when future you forgets how you programmed your code.
perl4ever 8y ago
I have the impression that issues with a software version not matching a previous version are not only not novel, but have been plentiful with any kind of software since the beginning of time. Microsoft is proverbial for Windows being under the constraint of maintaining past behavior however imperfect. To wit, it's such a common thing that there is an xkcd for it: https://imgs.xkcd.com/comics/workflow.png https://imgs.xkcd.com/comics/workflow.png P.S. I was momentarily repressing my first hand experience, but in a past job, there were many times I wanted to fix things and wasn't allowed to, because if the results changed, it would upset someone. The cardinal, CYA rule, was if we've been reporting something wrong for years, it better be consistent forever. Which is really painful if you care about correctness and compulsively investigate it.
- _bxg1 8y ago
  That sounds awful.