6 ms·

I guess it depends on a system you build and its maturity - in my area (automation for finance and revenue management) 80% of bugs are related to business logic

I guess it depends on a system you build and its maturity - in my area (automation for finance and revenue management) 80% of bugs are related to business logic and misinterpretation of user requirements.

marginalia_nu 5y ago
Errors are highly domain specific. When I worked in finance, I would have very much agreed with your description. But then, the code I wrote was fairly trivial, the challenge was communicating with the users and sometimes getting the users to understand what they were asking for wasn't what they wanted. I, the same developer, spent a good hour recently trying to iron out the bugs in a piece of code that concatenated two integers bit-wise and calculated the offset point. That is, given two integers whose bits are 00000000,00000000,00000000,0PPPPPPP 00000000,000000QQ,QQQQQQQQ,QQQQQQQQ The code should produce the shortest an array of bytes that looked like ZZZZZZZZ,0000000P,PPPPPPQQ,QQQQQQQQ,QQQQQQQQ Where Z in this case would be 18, the offset of P's bits. This is finicky in a way I've never encountered while I worked in finance. It's not exorbitantly difficult, but there are just a lot of potential logic errors to make. The code, ultimately, looked like this (it's Java hence the extra awkward bit twiddling) int pw = bitWidth(p); int qw = bitWidth(q); int pwr = (pw%8); int totalSize = (pw + qw)/8 + (((pw+qw)%8)>0?1:0); byte[] data = new byte[totalSize + 1]; int b = data.length - 1; while (p > 0) { data[b]=(byte)(p&0xFF); p = p >>> 8; if (p != 0) { b--; } } if (pwr != 0) { while (q > 0) { data[b--] |= ((byte) ((q << (pwr)) & 0xFF)); q = q >>> (8-pwr); data[b] |= (byte) (q & 0xFF); q = q >>> pwr; } } else { while (q > 0) { data[--b] |= (byte)(q&0xFF); q = q >>> 8; } } data[0] = (byte) pw; return data; It's like 40 lines of code. It shouldn't be that hard. Yet it is. Bit-twiddling is error prone and hard to reason about and, dunno, maybe pwr isn't a great variable name, but the code really doesn't get particularly expressive even if you change it to pWidthRemainder or whatever. It's still basically a bunch of cthulhu fhtagn as far as I'm concerned.
- hiptobecubic 5y ago
  What on Earth required this? Is this part of some home grown compression scheme?
  - marginalia_nu 5y ago
    It is indeed, it's part of the compression scheme for the keyword lexicon for my search engine. It's a domain-specific compression that uses the fact that the keyword dictionary is a dictionary of keywords to compress N-grams into dense blobs using this function. Overall it achieves about 70% compression. The lexicon supports several billion entries entirely in memory, which requires a lot these eldritch coding tricks (since Java certainly wasn't built for that many objects, GC and header overhead extremely prohibitive so objects are entirely out of the question). Most of the data is kept in contiguous memory mapped byte arrays.
- Supermancho 5y ago
  That's at least 5 functions of processing munged into one.
  - marginalia_nu 5y ago
    You're welcome to demonstrate that I'm wrong, but in my experience, Uncle Bobbing this type of code doesn't make it much clearer unless you introduce so many abstractions you see significant penalties. This is hot code with an invocation count in the billions, so among the requirements is that it shouldn't allocate objects while at the same time being thread safe. That's a tricky combo.
    - Supermancho 5y ago
      Regardless of the language, the function is too complex. Every while could be it's own function and now you have a bunch of simple unit tests. Unit tests are literally the double accounting for the logic, which is the value. Any unit tests for that code is going to be horrendous to get decent coverage...and this is how it's easy to know it's too complex. Functions, especially if they are small, can be considered free, given the JVM inlines them into branches anyway....but I understand this is a hot path. If you want to avoid re-allocation, Java gets into your way. You could split up the function and use a singletonClass for holding the array, without having to pass the array by value and still being able to reference the singletonClass...blah blah blah. > Uncle Bobbing this type of code doesn't make it much clearer I get it and agree, while still recognizing the problem. Not you not the code not me. Java. It's really unfortunate that Java has damaged programming practices so subtly and completely by being unnecessarily byzantine AND pushing horrible solutions like Spring.

marginalia_nu 5y ago

Errors are highly domain specific. When I worked in finance, I would have very much agreed with your description. But then, the code I wrote was fairly trivial, the challenge was communicating with the users and sometimes getting the users to understand what they were asking for wasn't what they wanted. I, the same developer, spent a good hour recently trying to iron out the bugs in a piece of code that concatenated two integers bit-wise and calculated the offset point. That is, given two integers whose bits are 00000000,00000000,00000000,0PPPPPPP 00000000,000000QQ,QQQQQQQQ,QQQQQQQQ The code should produce the shortest an array of bytes that looked like ZZZZZZZZ,0000000P,PPPPPPQQ,QQQQQQQQ,QQQQQQQQ Where Z in this case would be 18, the offset of P's bits. This is finicky in a way I've never encountered while I worked in finance. It's not exorbitantly difficult, but there are just a lot of potential logic errors to make. The code, ultimately, looked like this (it's Java hence the extra awkward bit twiddling) int pw = bitWidth(p); int qw = bitWidth(q); int pwr = (pw%8); int totalSize = (pw + qw)/8 + (((pw+qw)%8)>0?1:0); byte[] data = new byte[totalSize + 1]; int b = data.length - 1; while (p > 0) { data[b]=(byte)(p&0xFF); p = p >>> 8; if (p != 0) { b--; } } if (pwr != 0) { while (q > 0) { data[b--] |= ((byte) ((q << (pwr)) & 0xFF)); q = q >>> (8-pwr); data[b] |= (byte) (q & 0xFF); q = q >>> pwr; } } else { while (q > 0) { data[--b] |= (byte)(q&0xFF); q = q >>> 8; } } data[0] = (byte) pw; return data; It's like 40 lines of code. It shouldn't be that hard. Yet it is. Bit-twiddling is error prone and hard to reason about and, dunno, maybe pwr isn't a great variable name, but the code really doesn't get particularly expressive even if you change it to pWidthRemainder or whatever. It's still basically a bunch of cthulhu fhtagn as far as I'm concerned.

hiptobecubic 5y ago
What on Earth required this? Is this part of some home grown compression scheme?
- marginalia_nu 5y ago
  It is indeed, it's part of the compression scheme for the keyword lexicon for my search engine. It's a domain-specific compression that uses the fact that the keyword dictionary is a dictionary of keywords to compress N-grams into dense blobs using this function. Overall it achieves about 70% compression. The lexicon supports several billion entries entirely in memory, which requires a lot these eldritch coding tricks (since Java certainly wasn't built for that many objects, GC and header overhead extremely prohibitive so objects are entirely out of the question). Most of the data is kept in contiguous memory mapped byte arrays.
Supermancho 5y ago
That's at least 5 functions of processing munged into one.
- marginalia_nu 5y ago
  You're welcome to demonstrate that I'm wrong, but in my experience, Uncle Bobbing this type of code doesn't make it much clearer unless you introduce so many abstractions you see significant penalties. This is hot code with an invocation count in the billions, so among the requirements is that it shouldn't allocate objects while at the same time being thread safe. That's a tricky combo.
  - Supermancho 5y ago
    Regardless of the language, the function is too complex. Every while could be it's own function and now you have a bunch of simple unit tests. Unit tests are literally the double accounting for the logic, which is the value. Any unit tests for that code is going to be horrendous to get decent coverage...and this is how it's easy to know it's too complex. Functions, especially if they are small, can be considered free, given the JVM inlines them into branches anyway....but I understand this is a hot path. If you want to avoid re-allocation, Java gets into your way. You could split up the function and use a singletonClass for holding the array, without having to pass the array by value and still being able to reference the singletonClass...blah blah blah. > Uncle Bobbing this type of code doesn't make it much clearer I get it and agree, while still recognizing the problem. Not you not the code not me. Java. It's really unfortunate that Java has damaged programming practices so subtly and completely by being unnecessarily byzantine AND pushing horrible solutions like Spring.