5 ms·

Automated testing is useful and good. But I really feel it's reached a lever of fetishisation that is quite concerning. Testing code is code which needs to be

Automated testing is useful and good. But I really feel it's reached a lever of fetishisation that is quite concerning. Testing code is code which needs to be written, read, maintained, refactored. Very often nowadays I have to wade through tests which test nothing useful, except syntax. Even worse, with developers who adopt the mock-everything approach, I often find tests which only verify that the implementation is exactly the one they wrote, which is even worse: it makes refactoring a pain, because, even if you rewrote a method in a better way which produces exactly the results you wanted, the test will fail. So, the ratio of testing code vs implementation code is a completely wrong proxy for code quality. EDIT: I'm not criticising SQLite and their code quality - which I never studie - but the idea that you can judge code quality for a project just by the ratio of test code vs implementation code.

lolc 8y ago
While I agree in general I disagree here. If you read about the Sqlite tests you will find that they do test sensibly. One suite I'm particularly impressed with will run tests from zero bytes with slowly increasing available memory until the program passes. The tests verify that at no point the DB is corrupted by an OOM event.
- danmaz74 8y ago
  Just to clarify, I wasn't criticising sqlite, I was criticising the idea of judging their code quality "for this reason alone!" - ie that they have so much test code vs implementation.
  - lolc 8y ago
    Sure! Can't judge a book by its size. As a heuristic the code versus test-code ratio serves well as an indicator of quality. Just like consistent indentation does. You don't know whether a well-indented program is good. But if the indentation is inconsistent you'll expect worse.
faitswulff 8y ago
They actually have to test to that degree to follow aviation standards (DO-178b [0]) because they're used in aviation equipment. Dr. Hipp said he started really following it when Android came out and included SQLite and suddenly there were 200M mobile SQLite users finding edge cases: https://youtu.be/Jib2AmRb_rk?t=3413 https://youtu.be/Jib2AmRb_rk?t=3413 Lightly edited transcript here: > It made a huge difference. That that was when Android was just kicking off. In fact Android might not have been publicly announced, but we had been called in to help with getting Android going with SQLite. [Actually], they had been publicly announced and there were a bunch of Android phones out and we were getting flooded with problems coming in from Android. > I mean it worked great in the lab it worked great in all the testing and then [...] you give it to 200 million people and let them start clicking on their phone all day and suddenly bugs come up. And this is a big problem for us. > So I started doing following this DO-178b process and it took a good solid year to get us there. Good solid year of 12 hour days, six days a week, I mean we really really pushed but we got it there. And you know, once we got SQLite to the point where it was at that DO-178b level, standard, we still get bugs but you know they're very manageable. They're infrequent and they don't affect nearly as many people. > So it's been a huge huge thing. If you're writing an application deal ones, you know a website, a DO-178b/a is way overkill, okay? It's just because it's very expensive and very time-consuming, but if you're running an infrastructure thing like SQL, it's the only way to do it. [0]: https://youtu.be/Jib2AmRb_rk?t=677 https://youtu.be/Jib2AmRb_rk?t=677 "SQLite: The Database at the Edge of the Network with Dr. Richard Hipp"
- Nokinside 8y ago
  SQlite is very high quality software, but they use DO-178b "inspired" testing process. As far as I know they don't have version of software that is or can be used in safety critical parts despite their boasting. They say in their site that: > Airbus confirms that SQLite is being used in the flight software for the A350 XWB family of aircraft. Flight software does not imply safety critical parts of avionics. It can be the entertainment system or some logging that is not critical.
  - faitswulff 8y ago
    Good point. The video I linked to merely says that he was contacted by someone in the aviation space about the standard, which I took to mean that it was used in avionics.
rb808 8y ago
Agreed on the testing fetish and usually they're not valuable. Uncle Bob wrote about this last year and its something I wish everyone would read https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContravariance.html https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...
- arcticbull 8y ago
  Yes, bad tests are bad. Yes, mocks are bad. Good tests, however, are good. To expound on that, designing for testability allows you to sidestep the need for mocks almost entirely, and forces you into easier, more reliable and more consistent code. Then when you choose to test it, the tests are simple, straightforward and valuable.
jim_bailie 8y ago
Oh boy, I would give your comment an infinite number of up-votes. Yes, testing has reached fetish-like levels. Some of the test code I've encountered recently has been more voluminous, complex and has taken more man hours to develop and maintain than the application or library it's assigned to. For the love of God, develop the damned software! It's either going to work or it's not.
- kyberias 8y ago
  > has taken more man hours to develop and maintain than the application But that is perfectly normal when developing quality code. There is no rule that says that test code development should take LESS time. Certainly different applications have different quality requirements. Perhaps the software you are developing doesn't have that high requirements?
  - jim_bailie 8y ago
    Yeah, yeah. Just venting. The products I work on aren't the most important, but they certainly are quite important and most of the testing infrastructure that has been built thus far has a lot of goofy sh@t in it. I just don't have a high tolerance for needless complexity and gee-whiz-look-what-I-can-do while the clock is running.
forgotmypw 8y ago
You may not be able to judge code quality by the presence of test code, but you can by its absence.

lolc 8y ago

While I agree in general I disagree here. If you read about the Sqlite tests you will find that they do test sensibly. One suite I'm particularly impressed with will run tests from zero bytes with slowly increasing available memory until the program passes. The tests verify that at no point the DB is corrupted by an OOM event.

danmaz74 8y ago
Just to clarify, I wasn't criticising sqlite, I was criticising the idea of judging their code quality "for this reason alone!" - ie that they have so much test code vs implementation.
- lolc 8y ago
  Sure! Can't judge a book by its size. As a heuristic the code versus test-code ratio serves well as an indicator of quality. Just like consistent indentation does. You don't know whether a well-indented program is good. But if the indentation is inconsistent you'll expect worse.

faitswulff 8y ago

They actually have to test to that degree to follow aviation standards (DO-178b [0]) because they're used in aviation equipment. Dr. Hipp said he started really following it when Android came out and included SQLite and suddenly there were 200M mobile SQLite users finding edge cases: https://youtu.be/Jib2AmRb_rk?t=3413 https://youtu.be/Jib2AmRb_rk?t=3413 Lightly edited transcript here: > It made a huge difference. That that was when Android was just kicking off. In fact Android might not have been publicly announced, but we had been called in to help with getting Android going with SQLite. [Actually], they had been publicly announced and there were a bunch of Android phones out and we were getting flooded with problems coming in from Android. > I mean it worked great in the lab it worked great in all the testing and then [...] you give it to 200 million people and let them start clicking on their phone all day and suddenly bugs come up. And this is a big problem for us. > So I started doing following this DO-178b process and it took a good solid year to get us there. Good solid year of 12 hour days, six days a week, I mean we really really pushed but we got it there. And you know, once we got SQLite to the point where it was at that DO-178b level, standard, we still get bugs but you know they're very manageable. They're infrequent and they don't affect nearly as many people. > So it's been a huge huge thing. If you're writing an application deal ones, you know a website, a DO-178b/a is way overkill, okay? It's just because it's very expensive and very time-consuming, but if you're running an infrastructure thing like SQL, it's the only way to do it. [0]: https://youtu.be/Jib2AmRb_rk?t=677 https://youtu.be/Jib2AmRb_rk?t=677 "SQLite: The Database at the Edge of the Network with Dr. Richard Hipp"

Nokinside 8y ago
SQlite is very high quality software, but they use DO-178b "inspired" testing process. As far as I know they don't have version of software that is or can be used in safety critical parts despite their boasting. They say in their site that: > Airbus confirms that SQLite is being used in the flight software for the A350 XWB family of aircraft. Flight software does not imply safety critical parts of avionics. It can be the entertainment system or some logging that is not critical.
- faitswulff 8y ago
  Good point. The video I linked to merely says that he was contacted by someone in the aviation space about the standard, which I took to mean that it was used in avionics.

rb808 8y ago

Agreed on the testing fetish and usually they're not valuable. Uncle Bob wrote about this last year and its something I wish everyone would read https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContravariance.html https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...

arcticbull 8y ago
Yes, bad tests are bad. Yes, mocks are bad. Good tests, however, are good. To expound on that, designing for testability allows you to sidestep the need for mocks almost entirely, and forces you into easier, more reliable and more consistent code. Then when you choose to test it, the tests are simple, straightforward and valuable.

jim_bailie 8y ago

Oh boy, I would give your comment an infinite number of up-votes. Yes, testing has reached fetish-like levels. Some of the test code I've encountered recently has been more voluminous, complex and has taken more man hours to develop and maintain than the application or library it's assigned to. For the love of God, develop the damned software! It's either going to work or it's not.

kyberias 8y ago
> has taken more man hours to develop and maintain than the application But that is perfectly normal when developing quality code. There is no rule that says that test code development should take LESS time. Certainly different applications have different quality requirements. Perhaps the software you are developing doesn't have that high requirements?
- jim_bailie 8y ago
  Yeah, yeah. Just venting. The products I work on aren't the most important, but they certainly are quite important and most of the testing infrastructure that has been built thus far has a lot of goofy sh@t in it. I just don't have a high tolerance for needless complexity and gee-whiz-look-what-I-can-do while the clock is running.

forgotmypw 8y ago

You may not be able to judge code quality by the presence of test code, but you can by its absence.