6 ms·

One third of C++ devs in this survey don't write unit tests at all? That's insane in this day and age.

NavinF 5y ago
Is it really? Most unit tests I've encountered are garbage. They only test for bugs that are incredibly unlikely to happen. Integration tests and fuzz testing is where it's at. Anyone can add or improve an integration test within an hour and find several bugs that real users would encounter.
- ternaryoperator 5y ago
  All that could be true and it still would not suggest that no unit tests at all is a valid choice.
- Jtsummers 5y ago
  > They only test for bugs that are incredibly unlikely to happen. Well, that might be the problem. Testing "for bugs" isn't a terribly helpful activity. Testing for behavior is much more helpful, and unit tests can definitely be applied to that aspect of the system. Of course, it requires you to properly decompose the system into subsystems (modules) with loose coupling that can be individually (or in small sets) loaded into test harnesses for testing.
- ddlutz 5y ago
  Maybe you are not working on the right projects or with the right people, because good unit tests are extremely valuable. At a minimum, ensuring that making a change to some existing code doesn't break existing behavior is valuable.
- titzer 5y ago
  Strong disagree. Effective unit tests accelerate development by allowing fast turnaround and refactoring with confidence. There are bad unit tests, of course, that are highly coupled with the code that they test. I have written tens of thousands of unit tests in my career, and by far the most useful ones are the ones that point you to the exact subsystem in which a bug you've just introduced during refactoring/re-implementing lies. Bad unit tests are brittle "change detectors" and make you constantly adjust them for no real reason. I find mocks and interaction-based tests (rather than simple input/output tests) are this way. I flat-out do not use mocks for this very reason. They are intricate, involved, and take huge amounts of time to load context when they fail. Don't write such tests. If your unit tests are garbage, it's because they were not thought out very much and commit sins.
  - bcrosby95 5y ago
    > I flat-out do not use mocks for this very reason. They are intricate, involved, and take huge amounts of time to load context when they fail. Don't write such tests. This is why I prefer integration tests. Most of my projects interact heavily with a database. Making it an integration test unless I mock it out. Which I don't do, because in my experience, doing so tends to hide lots of bugs.
    - titzer 5y ago
      > This is why I prefer integration tests. IMO integration tests are great for making sure that your system actually functions with all its parts together. Nothing but an integration test can do that. But unit tests serve a different role: reducing the scope of the system under test so that you can pinpoint exactly what part failed when it does fail. They help you iterate quickly in refactoring, and when something goes wrong in production (or a bug report from the field), they narrow the cone of uncertainty by eliminating from consideration the parts of the system that are working correctly. In my experience, this rapidly cuts down on the amount of wild goose chases one goes on and overall makes a team more efficient. Then once a bug in the wild is found, the system responsible for the failure gets new regression tests and unit tests. Sometimes, I wrote unit tests for units I am suspicious of as I am debugging, and even if those tests don't find a bug, I might keep them. I worked for 6+ years on V8 and the parts of the system that were the least unit tested seemed to be home to far more production issues than others. It also helps a lot to keep your unit tests running super fast. Under 1 second is ideal.
  - deleted 5y ago
    [deleted]
  - thom 5y ago
    I think I agree with this, although I also think a lot of people would still call this integration testing. Basically a lot of code units are implementation details and you want to be free to refactor them fearlessly without having to write double the code each time tightly unit testing each part. Much better to narrow your system to a few well defined boundaries that deliver some clear value. The code behind that interface can be as deep as you like but you don’t really need to push tests down into the depths. As I say, some people will still think of this as integration testing just because there’s a lot of moving parts working together. But often you’re just testing fairly pure code with no external or IO dependencies so I’m happy thinking of these as unit tests still. Where those types of dependencies exist I do still mock though.
- Rd6n6 5y ago
  Except for unusually complex logic, almost every bug I’ve ever seen has been configuration or integration related. I do unit tests in safety critical software, but if I actually need to catch a bug I usually worry more about integration or end to end tests
iainctduncan 5y ago
These kind of survey's have terrible sampling biases. One of those is always going to be ability level, with more beginner, early career, or hobbyist folks interested in doing it. Case in point: I'm a hobby C++ hacker, and I read this. I'm a long time pro Python hacker and would not have opened something similar for Python! I'm like "the last thing I need to read about in my leisure time is what other people are doing with Python, that's WORK." haha.
munk-a 5y ago
C++ not having a "default" unit testing framework that pretty much ships with all the IDEs hurts it a lot - the other factor is changing culture, I think. Even fifteen years ago unit tests were a pretty alien thing to most folks and a lot of software in C++ has a very long lifespan. It's really hard to build momentum in an organization to actually set aside the time to write unit test coverage over existing code.
- deleted 5y ago
  [deleted]
p0nce 5y ago
Also half of Javascript and C developers, from the same set of Jetbrains studies.
marksmith2996 5y ago
I think its most likely greater than one third don't but some of them just say they do. And its even more likely that of the ones that do, most don't write any that have any value.
armchairhacker 5y ago
Only 57% of programmers surveyed say they write unit tests in any language (https://www.jetbrains.com/lp/devecosystem-2021/testing/ https://www.jetbrains.com/lp/devecosystem-2021/testing/). In Javascript only ~60% of developers test, and in Swift/ObjC only ~40%. Surprisingly 93% of Java users surveyed write unit tests though. I thought Kotlin would be similar, but for some reason JetBrains doesn't show any responses for Kotlin users.
- gravypod 5y ago
  I'd love to see an age distribution of Java develops vs the other languages. My impression is more experienced developers test their code more often. My other impression is some other developers have large egos and think "I don't need to write tests because I know how to test my code and everything I ship works".
  - johncena33 5y ago
    > My impression is more experienced developers test their code more often. Java percentage is unit test is so high most probably because it's an enterprise language. And the whole unit test craze started with Java and jUnit.
    - gravypod 5y ago
      > Java ... because it's an enterprise language I'm not sure that's the main factor. Compared to C# & C++? I don't think "enterprise" has much to do with it.
    - ivanche 5y ago
      I'm sure that "unit test craze" has absolutely nothing to do with Smalltalk, SUnit, Kent Beck and the year 1989. /s
    - TeMPOraL 5y ago
      I think it's even simpler than Java being an enterprise language. Java is a language with a simple grammar and got a working reflection system early on - both facilitated creation of supporting tooling like test coverage reports embedded in the IDE. It's much easier to cover more code with tests when your IDE makes it almost effortless, and constantly reports how well you're doing. And the easier something is, the more often people do it.
deleted 5y ago
[deleted]
jandrewrogers 5y ago
Some types of code commonly written in C++ is intrinsically not very amenable to unit testing when designed correctly. And the unit testing that is possible is sufficiently trivial that it isn’t worth it because aggressive use of assertions and strong integration testing will cover the same ground more cleanly. Of course, I am sure many of those survey results are people that don’t write unit tests when they could and should.
- jasonpeacock 5y ago
  Everything is testable, if it's not then it was designed poorly. I've seen 100%+ code coverage on cross-compiled bare-metal C++ embedded code. If that can be tested, then anything can be tested.
  - jandrewrogers 5y ago
    Everything is testable. Not everything is usefully testable with unit tests. Unit tests assume that correctness can be determined by evaluating isolated units of code. There are broad classes of software, notably high-performance data infrastructure, where unit testing manifestly doesn’t make sense. So people don’t write many unit tests even though this is mission-critical high-reliability software. Other kinds of testing is extremely thorough though. The kinds of code units that are not unit testable are those where the correctness of the implementation behavior is almost completely contextualized by the behavior of external code units. As in, the unit test may be correct or broken based on the behavior of external code modules, not based on the inputs to the module. Software that is very schedule driven, like modern database kernels, has this property generally. Good integration testing picks this up very well. Writing a proper “unit test” under these constraints is effectively equivalent to a relatively comprehensive integration test. And for parts of the code that can be unit tested, I tend to use exhaustive or quasi-exhaustive testing of their properties as part of release qualification, which is too expensive to be a true unit test but picks up more obscure errors than unit testing is ever likely to find and serves a similar function. I believe in brutal and thorough testing. But unit tests specifically are mostly a waste of time for some of the common applications where C++ excels as a programming language.

Is it really? Most unit tests I've encountered are garbage. They only test for bugs that are incredibly unlikely to happen. Integration tests and fuzz testing is where it's at. Anyone can add or improve an integration test within an hour and find several bugs that real users would encounter.

ternaryoperator 5y ago
All that could be true and it still would not suggest that no unit tests at all is a valid choice.
Jtsummers 5y ago
> They only test for bugs that are incredibly unlikely to happen. Well, that might be the problem. Testing "for bugs" isn't a terribly helpful activity. Testing for behavior is much more helpful, and unit tests can definitely be applied to that aspect of the system. Of course, it requires you to properly decompose the system into subsystems (modules) with loose coupling that can be individually (or in small sets) loaded into test harnesses for testing.
ddlutz 5y ago
Maybe you are not working on the right projects or with the right people, because good unit tests are extremely valuable. At a minimum, ensuring that making a change to some existing code doesn't break existing behavior is valuable.
titzer 5y ago
Strong disagree. Effective unit tests accelerate development by allowing fast turnaround and refactoring with confidence. There are bad unit tests, of course, that are highly coupled with the code that they test. I have written tens of thousands of unit tests in my career, and by far the most useful ones are the ones that point you to the exact subsystem in which a bug you've just introduced during refactoring/re-implementing lies. Bad unit tests are brittle "change detectors" and make you constantly adjust them for no real reason. I find mocks and interaction-based tests (rather than simple input/output tests) are this way. I flat-out do not use mocks for this very reason. They are intricate, involved, and take huge amounts of time to load context when they fail. Don't write such tests. If your unit tests are garbage, it's because they were not thought out very much and commit sins.
- bcrosby95 5y ago
  > I flat-out do not use mocks for this very reason. They are intricate, involved, and take huge amounts of time to load context when they fail. Don't write such tests. This is why I prefer integration tests. Most of my projects interact heavily with a database. Making it an integration test unless I mock it out. Which I don't do, because in my experience, doing so tends to hide lots of bugs.
  - titzer 5y ago
    > This is why I prefer integration tests. IMO integration tests are great for making sure that your system actually functions with all its parts together. Nothing but an integration test can do that. But unit tests serve a different role: reducing the scope of the system under test so that you can pinpoint exactly what part failed when it does fail. They help you iterate quickly in refactoring, and when something goes wrong in production (or a bug report from the field), they narrow the cone of uncertainty by eliminating from consideration the parts of the system that are working correctly. In my experience, this rapidly cuts down on the amount of wild goose chases one goes on and overall makes a team more efficient. Then once a bug in the wild is found, the system responsible for the failure gets new regression tests and unit tests. Sometimes, I wrote unit tests for units I am suspicious of as I am debugging, and even if those tests don't find a bug, I might keep them. I worked for 6+ years on V8 and the parts of the system that were the least unit tested seemed to be home to far more production issues than others. It also helps a lot to keep your unit tests running super fast. Under 1 second is ideal.
- deleted 5y ago
  [deleted]
- thom 5y ago
  I think I agree with this, although I also think a lot of people would still call this integration testing. Basically a lot of code units are implementation details and you want to be free to refactor them fearlessly without having to write double the code each time tightly unit testing each part. Much better to narrow your system to a few well defined boundaries that deliver some clear value. The code behind that interface can be as deep as you like but you don’t really need to push tests down into the depths. As I say, some people will still think of this as integration testing just because there’s a lot of moving parts working together. But often you’re just testing fairly pure code with no external or IO dependencies so I’m happy thinking of these as unit tests still. Where those types of dependencies exist I do still mock though.
Rd6n6 5y ago
Except for unusually complex logic, almost every bug I’ve ever seen has been configuration or integration related. I do unit tests in safety critical software, but if I actually need to catch a bug I usually worry more about integration or end to end tests

iainctduncan 5y ago

These kind of survey's have terrible sampling biases. One of those is always going to be ability level, with more beginner, early career, or hobbyist folks interested in doing it. Case in point: I'm a hobby C++ hacker, and I read this. I'm a long time pro Python hacker and would not have opened something similar for Python! I'm like "the last thing I need to read about in my leisure time is what other people are doing with Python, that's WORK." haha.

munk-a 5y ago

C++ not having a "default" unit testing framework that pretty much ships with all the IDEs hurts it a lot - the other factor is changing culture, I think. Even fifteen years ago unit tests were a pretty alien thing to most folks and a lot of software in C++ has a very long lifespan. It's really hard to build momentum in an organization to actually set aside the time to write unit test coverage over existing code.

deleted 5y ago
[deleted]

p0nce 5y ago

Also half of Javascript and C developers, from the same set of Jetbrains studies.

marksmith2996 5y ago

I think its most likely greater than one third don't but some of them just say they do. And its even more likely that of the ones that do, most don't write any that have any value.

armchairhacker 5y ago

Only 57% of programmers surveyed say they write unit tests in any language (https://www.jetbrains.com/lp/devecosystem-2021/testing/ https://www.jetbrains.com/lp/devecosystem-2021/testing/). In Javascript only ~60% of developers test, and in Swift/ObjC only ~40%. Surprisingly 93% of Java users surveyed write unit tests though. I thought Kotlin would be similar, but for some reason JetBrains doesn't show any responses for Kotlin users.

gravypod 5y ago
I'd love to see an age distribution of Java develops vs the other languages. My impression is more experienced developers test their code more often. My other impression is some other developers have large egos and think "I don't need to write tests because I know how to test my code and everything I ship works".
- johncena33 5y ago
  > My impression is more experienced developers test their code more often. Java percentage is unit test is so high most probably because it's an enterprise language. And the whole unit test craze started with Java and jUnit.
  - gravypod 5y ago
    > Java ... because it's an enterprise language I'm not sure that's the main factor. Compared to C# & C++? I don't think "enterprise" has much to do with it.
  - ivanche 5y ago
    I'm sure that "unit test craze" has absolutely nothing to do with Smalltalk, SUnit, Kent Beck and the year 1989. /s
  - TeMPOraL 5y ago
    I think it's even simpler than Java being an enterprise language. Java is a language with a simple grammar and got a working reflection system early on - both facilitated creation of supporting tooling like test coverage reports embedded in the IDE. It's much easier to cover more code with tests when your IDE makes it almost effortless, and constantly reports how well you're doing. And the easier something is, the more often people do it.

deleted 5y ago

[deleted]

jandrewrogers 5y ago

Some types of code commonly written in C++ is intrinsically not very amenable to unit testing when designed correctly. And the unit testing that is possible is sufficiently trivial that it isn’t worth it because aggressive use of assertions and strong integration testing will cover the same ground more cleanly. Of course, I am sure many of those survey results are people that don’t write unit tests when they could and should.

jasonpeacock 5y ago
Everything is testable, if it's not then it was designed poorly. I've seen 100%+ code coverage on cross-compiled bare-metal C++ embedded code. If that can be tested, then anything can be tested.
- jandrewrogers 5y ago
  Everything is testable. Not everything is usefully testable with unit tests. Unit tests assume that correctness can be determined by evaluating isolated units of code. There are broad classes of software, notably high-performance data infrastructure, where unit testing manifestly doesn’t make sense. So people don’t write many unit tests even though this is mission-critical high-reliability software. Other kinds of testing is extremely thorough though. The kinds of code units that are not unit testable are those where the correctness of the implementation behavior is almost completely contextualized by the behavior of external code units. As in, the unit test may be correct or broken based on the behavior of external code modules, not based on the inputs to the module. Software that is very schedule driven, like modern database kernels, has this property generally. Good integration testing picks this up very well. Writing a proper “unit test” under these constraints is effectively equivalent to a relatively comprehensive integration test. And for parts of the code that can be unit tested, I tend to use exhaustive or quasi-exhaustive testing of their properties as part of release qualification, which is too expensive to be a true unit test but picks up more obscure errors than unit testing is ever likely to find and serves a similar function. I believe in brutal and thorough testing. But unit tests specifically are mostly a waste of time for some of the common applications where C++ excels as a programming language.