12 ms·

AFAIK (correct me if I am wrong) OpenBSD does not have a single unit test. No unit tests. There is so much code that parses text input and yet I can’t find a

by st3fan 7y ago

AFAIK (correct me if I am wrong) OpenBSD does not have a single unit test. No unit tests. There is so much code that parses text input and yet I can’t find a single test that actually shows that the code works and that it properly deals with malicious input. How do OpenBSD developers test? How do they make sure changes do not regress things. Why is unit testing just not a thing? (Same for many other C based projects)

clarry 7y ago
Which one of these bugs would have been prevented by unit tests? Probably none, because you'd have to know about the bug in order to craft a test that exploits it.
- sildur 7y ago
  Unit tests could prevent reintroducing old bugs.
  - clarry 7y ago
    You have regression tests for those. http://cvsweb.openbsd.org/src/regress/ http://cvsweb.openbsd.org/src/regress/
    - deleted 7y ago
      [deleted]
  - loeg 7y ago
    This subcategory of unit tests is known as regression tests, and those exist too.
- st3fan 7y ago
  Sometimes that is definitely true. But very often when you spend time to write unit tests you discover limitations and bugs in your code while you come up with test scenarios. The only thing that the FTP bug needed was a "test_redirect_responses()" test. The SMTPd bug was all about parsing user input. Ideal candidate for unit testing and probably even some fuzzing.
  - Someone 7y ago
    FTP doesn’t have redirect, so the original code wouldn’t have that test. The writers who added handling of HTTP would only have added it if they had realized the HTTP library they used could do a redirect. Even if they did, they would have to have thought about redirecting to some specially crafted URL.
    - st3fan 7y ago
      Excuses excuses excuses ...
brohee 7y ago
You're wrong. They are in their own directory in the source tree (https://cvsweb.openbsd.org/src/regress/ https://cvsweb.openbsd.org/src/regress/). ld.so for example is pretty well exercised (https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/libexec/ld.so/#dirlist https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/libexe...)
- st3fan 7y ago
  Can you point me to a unit test that validates for example parsing of incoming commands to opensmtpd or ftpd or ntpd? Nothing. It is just not a common practice it seems. Now I don't think unit tests are a silver bullet. But having just code and zero tests is definitely a huge red flag for me.

by st3fan 7y ago

clarry 7y ago

Which one of these bugs would have been prevented by unit tests? Probably none, because you'd have to know about the bug in order to craft a test that exploits it.

sildur 7y ago
Unit tests could prevent reintroducing old bugs.
- clarry 7y ago
  You have regression tests for those. http://cvsweb.openbsd.org/src/regress/ http://cvsweb.openbsd.org/src/regress/
  - deleted 7y ago
    [deleted]
- loeg 7y ago
  This subcategory of unit tests is known as regression tests, and those exist too.
st3fan 7y ago
Sometimes that is definitely true. But very often when you spend time to write unit tests you discover limitations and bugs in your code while you come up with test scenarios. The only thing that the FTP bug needed was a "test_redirect_responses()" test. The SMTPd bug was all about parsing user input. Ideal candidate for unit testing and probably even some fuzzing.
- Someone 7y ago
  FTP doesn’t have redirect, so the original code wouldn’t have that test. The writers who added handling of HTTP would only have added it if they had realized the HTTP library they used could do a redirect. Even if they did, they would have to have thought about redirecting to some specially crafted URL.
  - st3fan 7y ago
    Excuses excuses excuses ...

brohee 7y ago

You're wrong. They are in their own directory in the source tree (https://cvsweb.openbsd.org/src/regress/ https://cvsweb.openbsd.org/src/regress/). ld.so for example is pretty well exercised (https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/libexec/ld.so/#dirlist https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/regress/libexe...)

st3fan 7y ago
Can you point me to a unit test that validates for example parsing of incoming commands to opensmtpd or ftpd or ntpd? Nothing. It is just not a common practice it seems. Now I don't think unit tests are a silver bullet. But having just code and zero tests is definitely a huge red flag for me.