8 ms·

I find it interesting that while logic bugs (e.g. off by one errors, switched if conditions, etc) make up less than 1/3 of bugs I encounter day to day, it seems

by pydry 5y ago

I find it interesting that while logic bugs (e.g. off by one errors, switched if conditions, etc) make up less than 1/3 of bugs I encounter day to day, it seems to comprise of about 80% of all discussion around testing. Bugs caused by misunderstood or unexpected interactions between subsystems and specification bugs seem to predominate these days IME but discussion about sophisticated ways of dealing with these things seems fairly scant.

smugglerFlynn 5y ago
I guess it depends on a system you build and its maturity - in my area (automation for finance and revenue management) 80% of bugs are related to business logic and misinterpretation of user requirements.
- marginalia_nu 5y ago
  Errors are highly domain specific. When I worked in finance, I would have very much agreed with your description. But then, the code I wrote was fairly trivial, the challenge was communicating with the users and sometimes getting the users to understand what they were asking for wasn't what they wanted. I, the same developer, spent a good hour recently trying to iron out the bugs in a piece of code that concatenated two integers bit-wise and calculated the offset point. That is, given two integers whose bits are 00000000,00000000,00000000,0PPPPPPP 00000000,000000QQ,QQQQQQQQ,QQQQQQQQ The code should produce the shortest an array of bytes that looked like ZZZZZZZZ,0000000P,PPPPPPQQ,QQQQQQQQ,QQQQQQQQ Where Z in this case would be 18, the offset of P's bits. This is finicky in a way I've never encountered while I worked in finance. It's not exorbitantly difficult, but there are just a lot of potential logic errors to make. The code, ultimately, looked like this (it's Java hence the extra awkward bit twiddling) int pw = bitWidth(p); int qw = bitWidth(q); int pwr = (pw%8); int totalSize = (pw + qw)/8 + (((pw+qw)%8)>0?1:0); byte[] data = new byte[totalSize + 1]; int b = data.length - 1; while (p > 0) { data[b]=(byte)(p&0xFF); p = p >>> 8; if (p != 0) { b--; } } if (pwr != 0) { while (q > 0) { data[b--] |= ((byte) ((q << (pwr)) & 0xFF)); q = q >>> (8-pwr); data[b] |= (byte) (q & 0xFF); q = q >>> pwr; } } else { while (q > 0) { data[--b] |= (byte)(q&0xFF); q = q >>> 8; } } data[0] = (byte) pw; return data; It's like 40 lines of code. It shouldn't be that hard. Yet it is. Bit-twiddling is error prone and hard to reason about and, dunno, maybe pwr isn't a great variable name, but the code really doesn't get particularly expressive even if you change it to pWidthRemainder or whatever. It's still basically a bunch of cthulhu fhtagn as far as I'm concerned.
  - hiptobecubic 5y ago
    What on Earth required this? Is this part of some home grown compression scheme?
    - marginalia_nu 5y ago
      It is indeed, it's part of the compression scheme for the keyword lexicon for my search engine. It's a domain-specific compression that uses the fact that the keyword dictionary is a dictionary of keywords to compress N-grams into dense blobs using this function. Overall it achieves about 70% compression. The lexicon supports several billion entries entirely in memory, which requires a lot these eldritch coding tricks (since Java certainly wasn't built for that many objects, GC and header overhead extremely prohibitive so objects are entirely out of the question). Most of the data is kept in contiguous memory mapped byte arrays.
  - Supermancho 5y ago
    That's at least 5 functions of processing munged into one.
    - marginalia_nu 5y ago
      You're welcome to demonstrate that I'm wrong, but in my experience, Uncle Bobbing this type of code doesn't make it much clearer unless you introduce so many abstractions you see significant penalties. This is hot code with an invocation count in the billions, so among the requirements is that it shouldn't allocate objects while at the same time being thread safe. That's a tricky combo.
      - Supermancho 5y ago
        Regardless of the language, the function is too complex. Every while could be it's own function and now you have a bunch of simple unit tests. Unit tests are literally the double accounting for the logic, which is the value. Any unit tests for that code is going to be horrendous to get decent coverage...and this is how it's easy to know it's too complex. Functions, especially if they are small, can be considered free, given the JVM inlines them into branches anyway....but I understand this is a hot path. If you want to avoid re-allocation, Java gets into your way. You could split up the function and use a singletonClass for holding the array, without having to pass the array by value and still being able to reference the singletonClass...blah blah blah. > Uncle Bobbing this type of code doesn't make it much clearer I get it and agree, while still recognizing the problem. Not you not the code not me. Java. It's really unfortunate that Java has damaged programming practices so subtly and completely by being unnecessarily byzantine AND pushing horrible solutions like Spring.
Cthulhu_ 5y ago
That's because these are knowable errors; you can't talk about a bug if you don't know it might occur. Testing is not a panacea, by design you can only test for what you know and predict. In addition to testing, your setup needs to be resilient enough that unpredictable bugs can be solved quickly and won't lead to data loss.
- pydry 5y ago
  I don't think we do this because integration and specification bugs are inherently "unknowable". I think it's more cultural than technical. I've also noticed a parallel phenomenon where teams double down on unit tests for branch conditions, coverage %, etc. when the bugs that are really hurting them predominantly cant even really be detected by unit tests.
  - faizshah 5y ago
    I’m with you, we spend most of our time testing for bugs that we are more comfortable with but they all have some underlying assumption about how we integrate with outside systems. Defensive programming has often taken a backseat to delivering features and testing has more so become a rubber stamp for regressions rather than a test of resilience. A lot of modern software, especially web apps are very brittle, when something stops working the way it’s supposed to the whole app goes into an undefined failure mode instead of giving a user partial functionality.
    - piyh 5y ago
      I inherited a distributed app that causes me a lot of off hours work. My goal build pipeline brings up the app in docker-compose, then randomly kills services while running integration tests. So many failures are because people don't consider the happy path in a distributed/highly modular system.
- thaumasiotes 5y ago
  > Testing is not a panacea, by design you can only test for what you know and predict. Fuzz testing is still a type of testing.
lumost 5y ago
This sounds like the problem that despite all of the effort at preventing simple programming errors, 1/3rd of all the bugs you encounter are "simple" programming errors. Programming errors are of special interest to programmers, as we tend to judge each other a bit harshly when these errors emerge as the problem is not on the business side or other teams making requirements.
pierrebai 5y ago
The roots of the problem you described are many. One is that there are very good unit test libraries out there, which encourages writing unit tests. That sounds good, and is a good starting point, but it often become the be-all end-all of testing. In my experience working on many very large software, I have encountered one that was head and shoulder above the others in quality. This is not just a hunch. There were metrics tracked, thing like MTBF, and percent of sessions with clean exits, etc and this software was ridiculously better than the others. The difference lied entirely on the fact that the better software had almost no unit test, but had 21K+ integration tests. I've worked on other software that also had more integration tests than average, and they too were of better quality. The problem is making integration testing easy needs to be designed in from the start and requires not just one simple trick but multiple inter-locked design decisions that accumulate into making integration testing easy. In this particular case, one could write a new test just by using the software and a new test would be generated automatically. That's how you achieve writing 21k integration tests. There are many unit test framework, but I've yet to see a generic integration framework. The reason is that integration tests are run with the whole software, so the whole software needs to be designed around them.
- rixed 5y ago
  > In this particular case, one could write a new test just by using the software and a new test would be generated automatically I'm interested in the topic. Could you talk more about it, or point to some resources describing in more details how this was achieved/what decisions had to be taken?
- usefulcat 5y ago
  I’ve written a great deal of tests over the years and this sounds very interesting to me, yet it’s not obvious to me from your comment what this would look like. I’d like it very much if you could elaborate further.
- eternityforest 5y ago
  Integration tests are hard to do right, but easy to hack together. You might have to do the whole "Start with a clean DB, fake push a button, check for this record" thing, but at least you'll have some tests. They might be a maintenance nightmare but... they'll probably be better than no tests

by pydry 5y ago

smugglerFlynn 5y ago

I guess it depends on a system you build and its maturity - in my area (automation for finance and revenue management) 80% of bugs are related to business logic and misinterpretation of user requirements.

marginalia_nu 5y ago
Errors are highly domain specific. When I worked in finance, I would have very much agreed with your description. But then, the code I wrote was fairly trivial, the challenge was communicating with the users and sometimes getting the users to understand what they were asking for wasn't what they wanted. I, the same developer, spent a good hour recently trying to iron out the bugs in a piece of code that concatenated two integers bit-wise and calculated the offset point. That is, given two integers whose bits are 00000000,00000000,00000000,0PPPPPPP 00000000,000000QQ,QQQQQQQQ,QQQQQQQQ The code should produce the shortest an array of bytes that looked like ZZZZZZZZ,0000000P,PPPPPPQQ,QQQQQQQQ,QQQQQQQQ Where Z in this case would be 18, the offset of P's bits. This is finicky in a way I've never encountered while I worked in finance. It's not exorbitantly difficult, but there are just a lot of potential logic errors to make. The code, ultimately, looked like this (it's Java hence the extra awkward bit twiddling) int pw = bitWidth(p); int qw = bitWidth(q); int pwr = (pw%8); int totalSize = (pw + qw)/8 + (((pw+qw)%8)>0?1:0); byte[] data = new byte[totalSize + 1]; int b = data.length - 1; while (p > 0) { data[b]=(byte)(p&0xFF); p = p >>> 8; if (p != 0) { b--; } } if (pwr != 0) { while (q > 0) { data[b--] |= ((byte) ((q << (pwr)) & 0xFF)); q = q >>> (8-pwr); data[b] |= (byte) (q & 0xFF); q = q >>> pwr; } } else { while (q > 0) { data[--b] |= (byte)(q&0xFF); q = q >>> 8; } } data[0] = (byte) pw; return data; It's like 40 lines of code. It shouldn't be that hard. Yet it is. Bit-twiddling is error prone and hard to reason about and, dunno, maybe pwr isn't a great variable name, but the code really doesn't get particularly expressive even if you change it to pWidthRemainder or whatever. It's still basically a bunch of cthulhu fhtagn as far as I'm concerned.
- hiptobecubic 5y ago
  What on Earth required this? Is this part of some home grown compression scheme?
  - marginalia_nu 5y ago
    It is indeed, it's part of the compression scheme for the keyword lexicon for my search engine. It's a domain-specific compression that uses the fact that the keyword dictionary is a dictionary of keywords to compress N-grams into dense blobs using this function. Overall it achieves about 70% compression. The lexicon supports several billion entries entirely in memory, which requires a lot these eldritch coding tricks (since Java certainly wasn't built for that many objects, GC and header overhead extremely prohibitive so objects are entirely out of the question). Most of the data is kept in contiguous memory mapped byte arrays.
- Supermancho 5y ago
  That's at least 5 functions of processing munged into one.
  - marginalia_nu 5y ago
    You're welcome to demonstrate that I'm wrong, but in my experience, Uncle Bobbing this type of code doesn't make it much clearer unless you introduce so many abstractions you see significant penalties. This is hot code with an invocation count in the billions, so among the requirements is that it shouldn't allocate objects while at the same time being thread safe. That's a tricky combo.
    - Supermancho 5y ago
      Regardless of the language, the function is too complex. Every while could be it's own function and now you have a bunch of simple unit tests. Unit tests are literally the double accounting for the logic, which is the value. Any unit tests for that code is going to be horrendous to get decent coverage...and this is how it's easy to know it's too complex. Functions, especially if they are small, can be considered free, given the JVM inlines them into branches anyway....but I understand this is a hot path. If you want to avoid re-allocation, Java gets into your way. You could split up the function and use a singletonClass for holding the array, without having to pass the array by value and still being able to reference the singletonClass...blah blah blah. > Uncle Bobbing this type of code doesn't make it much clearer I get it and agree, while still recognizing the problem. Not you not the code not me. Java. It's really unfortunate that Java has damaged programming practices so subtly and completely by being unnecessarily byzantine AND pushing horrible solutions like Spring.

Cthulhu_ 5y ago

That's because these are knowable errors; you can't talk about a bug if you don't know it might occur. Testing is not a panacea, by design you can only test for what you know and predict. In addition to testing, your setup needs to be resilient enough that unpredictable bugs can be solved quickly and won't lead to data loss.

pydry 5y ago
I don't think we do this because integration and specification bugs are inherently "unknowable". I think it's more cultural than technical. I've also noticed a parallel phenomenon where teams double down on unit tests for branch conditions, coverage %, etc. when the bugs that are really hurting them predominantly cant even really be detected by unit tests.
- faizshah 5y ago
  I’m with you, we spend most of our time testing for bugs that we are more comfortable with but they all have some underlying assumption about how we integrate with outside systems. Defensive programming has often taken a backseat to delivering features and testing has more so become a rubber stamp for regressions rather than a test of resilience. A lot of modern software, especially web apps are very brittle, when something stops working the way it’s supposed to the whole app goes into an undefined failure mode instead of giving a user partial functionality.
  - piyh 5y ago
    I inherited a distributed app that causes me a lot of off hours work. My goal build pipeline brings up the app in docker-compose, then randomly kills services while running integration tests. So many failures are because people don't consider the happy path in a distributed/highly modular system.
thaumasiotes 5y ago
> Testing is not a panacea, by design you can only test for what you know and predict. Fuzz testing is still a type of testing.

lumost 5y ago

This sounds like the problem that despite all of the effort at preventing simple programming errors, 1/3rd of all the bugs you encounter are "simple" programming errors. Programming errors are of special interest to programmers, as we tend to judge each other a bit harshly when these errors emerge as the problem is not on the business side or other teams making requirements.

pierrebai 5y ago

The roots of the problem you described are many. One is that there are very good unit test libraries out there, which encourages writing unit tests. That sounds good, and is a good starting point, but it often become the be-all end-all of testing. In my experience working on many very large software, I have encountered one that was head and shoulder above the others in quality. This is not just a hunch. There were metrics tracked, thing like MTBF, and percent of sessions with clean exits, etc and this software was ridiculously better than the others. The difference lied entirely on the fact that the better software had almost no unit test, but had 21K+ integration tests. I've worked on other software that also had more integration tests than average, and they too were of better quality. The problem is making integration testing easy needs to be designed in from the start and requires not just one simple trick but multiple inter-locked design decisions that accumulate into making integration testing easy. In this particular case, one could write a new test just by using the software and a new test would be generated automatically. That's how you achieve writing 21k integration tests. There are many unit test framework, but I've yet to see a generic integration framework. The reason is that integration tests are run with the whole software, so the whole software needs to be designed around them.

rixed 5y ago
> In this particular case, one could write a new test just by using the software and a new test would be generated automatically I'm interested in the topic. Could you talk more about it, or point to some resources describing in more details how this was achieved/what decisions had to be taken?
usefulcat 5y ago
I’ve written a great deal of tests over the years and this sounds very interesting to me, yet it’s not obvious to me from your comment what this would look like. I’d like it very much if you could elaborate further.
eternityforest 5y ago
Integration tests are hard to do right, but easy to hack together. You might have to do the whole "Start with a clean DB, fake push a button, check for this record" thing, but at least you'll have some tests. They might be a maintenance nightmare but... they'll probably be better than no tests