5 ms·

I was under the impression that OpenBSD's unit testing files reside in the src/regress directory. Here is the relevant directory for file: http://cvsweb.openbs

I was under the impression that OpenBSD's unit testing files reside in the src/regress directory. Here is the relevant directory for file: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin/file/ http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin... The files in that directory sure look like tests to me. Am I missing something obvious?

acqq 11y ago
Yes, the obvious thing you miss is that according to the OP the unit tests should contain the tests for the bugs detected by recent third-party fuzzing and the "file" should contain the fix to that. Instead, there most recent test is 6 years old but there's the whole brand new implementation of the whole "file" utility. It does sound CADT-y. http://www.jwz.org/doc/cadt.html http://www.jwz.org/doc/cadt.html "It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8? But that's what happens when there is no incentive for people to do the parts of programming that aren't fun. Fixing bugs isn't fun; going through the bug list isn't fun; but rewriting everything from scratch is fun (because "this time it will be done right", ha ha) and so that's what happens, over and over again." Specifically, why didn't somebody just remove the ELF part from the existing code, if the goal was "reducing the attack surface by removing the features?" (which can be a reasonable strategy for increasing the security).
- nossabova 11y ago
  I agree with you that more specific tests should be included for the recent fuzzing results, but the great-grandparent comment seems to suggest that there are no tests at all, which is clearly not true. Instead there most recent test is 6 years old I don't feel that this detail is relevant. Because file is a basic utility, and its function is not supposed to change much over time, a test written 6 years ago for file should still be relevant to file today. Specifically, why didn't somebody just remove the ELF part from the existing code I think this was not done because much of the file code followed outdated coding conventions, and would have to be replaced anyways. Take a look at this diff comparison of file.h: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/file.h.diff?r1=1.24&r2=1.25&f=h http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/fi...
  - acqq 11y ago
    I think you also miss that the reason the "file" got the attention were the recent fuzzing actions. Which discovered the bugs obviously not present in the quoted 6 year old tests. Otherwise there would be no discovery at all. So no, the 6-year tests weren't sufficient. And yes, the updated tests and fixed old implementation is more important step than ""this time it will be done right", ha ha" Additionally, replacing the "defined" constants from the file.h with the enum in another .h is hardly a reason for the rewrite.
    - nossabova 11y ago
      I think I understand your point better now. More thorough tests in addition to the current set would certainly be an improvement. However, I don't understand why you suggest that fixing the old implementation to adhere to new programming standards would be better than re-writing the program from scratch, because most of the program would have to be re-written anyways, and so the developer would barely save any time and effort by fixing rather than re-writing. Also, if more thorough tests were to be included, then either fixing the program or re-writing it would have the same impact security-wise, as the improved tests would account for any discrepancy in either route. IMO, in this case, whether to fix the old software or to re-write a new version were both acceptable options.
      - acqq 11y ago
        I neither suggest anything like "fixing the old implementation to adhere to new programming standards" nor "re-writing the program from scratch." I suggest extending the tests, fixing the problems discovered in the existing implementation and then doing something completely new, somewhere else.

acqq 11y ago

Yes, the obvious thing you miss is that according to the OP the unit tests should contain the tests for the bugs detected by recent third-party fuzzing and the "file" should contain the fix to that. Instead, there most recent test is 6 years old but there's the whole brand new implementation of the whole "file" utility. It does sound CADT-y. http://www.jwz.org/doc/cadt.html http://www.jwz.org/doc/cadt.html "It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8? But that's what happens when there is no incentive for people to do the parts of programming that aren't fun. Fixing bugs isn't fun; going through the bug list isn't fun; but rewriting everything from scratch is fun (because "this time it will be done right", ha ha) and so that's what happens, over and over again." Specifically, why didn't somebody just remove the ELF part from the existing code, if the goal was "reducing the attack surface by removing the features?" (which can be a reasonable strategy for increasing the security).

nossabova 11y ago
I agree with you that more specific tests should be included for the recent fuzzing results, but the great-grandparent comment seems to suggest that there are no tests at all, which is clearly not true. Instead there most recent test is 6 years old I don't feel that this detail is relevant. Because file is a basic utility, and its function is not supposed to change much over time, a test written 6 years ago for file should still be relevant to file today. Specifically, why didn't somebody just remove the ELF part from the existing code I think this was not done because much of the file code followed outdated coding conventions, and would have to be replaced anyways. Take a look at this diff comparison of file.h: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/file.h.diff?r1=1.24&r2=1.25&f=h http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/fi...
- acqq 11y ago
  I think you also miss that the reason the "file" got the attention were the recent fuzzing actions. Which discovered the bugs obviously not present in the quoted 6 year old tests. Otherwise there would be no discovery at all. So no, the 6-year tests weren't sufficient. And yes, the updated tests and fixed old implementation is more important step than ""this time it will be done right", ha ha" Additionally, replacing the "defined" constants from the file.h with the enum in another .h is hardly a reason for the rewrite.
  - nossabova 11y ago
    I think I understand your point better now. More thorough tests in addition to the current set would certainly be an improvement. However, I don't understand why you suggest that fixing the old implementation to adhere to new programming standards would be better than re-writing the program from scratch, because most of the program would have to be re-written anyways, and so the developer would barely save any time and effort by fixing rather than re-writing. Also, if more thorough tests were to be included, then either fixing the program or re-writing it would have the same impact security-wise, as the improved tests would account for any discrepancy in either route. IMO, in this case, whether to fix the old software or to re-write a new version were both acceptable options.
    - acqq 11y ago
      I neither suggest anything like "fixing the old implementation to adhere to new programming standards" nor "re-writing the program from scratch." I suggest extending the tests, fixing the problems discovered in the existing implementation and then doing something completely new, somewhere else.