5 ms·

Sometimes automated testing is the only way to go. Consider a case when you have a code that takes about 100 input parameters and makes a decision on whether s

Sometimes automated testing is the only way to go. Consider a case when you have a code that takes about 100 input parameters and makes a decision on whether something should or should not happen. The logic that determines the decision is about 1,000 LOC that doesn't divide easily into smaller chunks. The code consists of a lot of different conditions and calculations based on the input parameters and some constants. Nobody knows for sure or can explain how every single line works or why is it there, but it seems that everything works correctly based on the decision. Writing tests manually might be feasible, but is very impractical considering that you don't know whether the code is correct or not in the first place. The code is being modified on regular basis: new inputs are added and logic is changed. Any such change would break the manual test. In my experience, nobody will spend an hour studying why the test broke and do the "right" fix. It will most likely be the easiest fix that makes the test pass.

cdogl 5y ago
I'm intrigued by this example. What's an example of a domain or problem set that produces a function with 100 input parameters and a boolean result that can't be broken down cleanly?
- decremental 5y ago
  Adding a product with many attributes to a database, where some of the attributes depend on the values of others, might be one example.
- zerof1l 5y ago
  The decision is actually 5 different variables. Types are: boolean, a decimal percent and integer being a monetary value. The domain is fintech: making a decision on how much money to give and some other aspects about the loan.
  - choeger 5y ago
    That sounds like no one knows what the correct result is.
  - 4140tm 5y ago
    > The domain is fintech What you describe is the opposite of what I expect from (modern) fintech. * no tests * god method with no proper documentation/requirements * no one actually understanding the function deciding how much money to send! Isn't this one of the core functionalities of fintech products?
    - stjohnswarts 5y ago
      Just plain scary especially with money if they can't at the very least test the extremes and some in the middle values. Good gravy! What is the name of this company so I can avoid it...
  - cjfd 5y ago
    The whole thing does not sound very competent. Like noone could possibly guarantee that the result is not off by a factor of 2 or 3 or perhaps even 10 or more in some cases. What would happen in that case? It also sounds like a domain where mistakes could be very expensive or lead to lawsuits or similar such things.
    - zerof1l 5y ago
      Things are not that bad. While the whole thing is kind of a monolith, there are at least broad sections that split code into logical sections. With some practice, it is possible to easily narrow down changes and any issues to 10-20 LOC. Later, breaking code by them could be a starting place for refactoring. In the end, the decision is manually reviewed. There's also a log that tells how and why the code had arrived at the numbers. So, if you know that client cannot be X but the log says X then you know that there's a bug for example. This code was around before I joined. It started small, but more and more requirements were added until it became like that. Not much effort went into structuring it. Now it's my responsibility, and I'm trying to make it better for me and anyone who will have to deal with it in the future.
  - totorovirus 5y ago
    Does this mean there are at least 2^5 = 32 paths that a procedure can take? Seriously, is it even documentable?
- CJefferson 5y ago
  This might not be quite the same, but logic systems -- we might have a logic expression on a hundred variables, and need to check if it is true. While one would unit test with tiny expressions, often bugs only show up in large integration tests, as systems use many optimisations to cache and speed up behaviour, which can interact poorly
helsinkiandrew 5y ago
This all sounds like a race to the bottom, where automated testing will drive further kludge upon kludge and the code will get less understood and harder to maintain and write tests for. Whilst an automated test suite can at least check if the code may have broken - you'll never know if the break is significant or just an inconsequential side effect of a new piece of data. And if both the code and tests aren't understood, as you say the "right" fix won't be done and something will be added quickly to fix the test.
Chris2048 5y ago
So: - you don't know whether the code is correct or not - code is being modified on regular basis... and logic is changed If someone knows enough about the code to regularly change it, then they should be able to provide some documentation? How else are they able to modify a "correct" function and know that it's still correct? Can I guess that the code modifiers are quants, or something?
4140tm 5y ago
If no one knows whether the code is correct a generated test could easily do more harm than good. If no one is able/willing to refactor the code into understandable and testable bits, there's not much left to guarantee the correctness of it.
dmos62 5y ago
What you're describing is testing without understanding the code. I've not yet encountered inunderstandable code, and can't conceive what it might look like, though I've encountered code that I'd rather not try to understand.
mcherm 5y ago
In that situation, why do you WANT tests? If, for example, someone intends to refactor the code into something more maintainable and wants tests to prevent any accidental changes in functionality, then this is useful. If the goal is to catch bugs or to facilitate future development or to meet some test coverage standard then I argue you might be better off not having tests rather than having auto-generated tests that have not been carefully reviewed.
Ensorceled 5y ago
Years ago, in a similar situation, I wrote a tool to capture parameters and results for the algorithm while QA ran their full application test suite. Ended up with a few thousand cases and expected results; results where QA had verified the software worked as expected. Used that to create a test suite that we could then maintain normally. Much better than random.

cdogl 5y ago

I'm intrigued by this example. What's an example of a domain or problem set that produces a function with 100 input parameters and a boolean result that can't be broken down cleanly?

decremental 5y ago
Adding a product with many attributes to a database, where some of the attributes depend on the values of others, might be one example.
zerof1l 5y ago
The decision is actually 5 different variables. Types are: boolean, a decimal percent and integer being a monetary value. The domain is fintech: making a decision on how much money to give and some other aspects about the loan.
- choeger 5y ago
  That sounds like no one knows what the correct result is.
- 4140tm 5y ago
  > The domain is fintech What you describe is the opposite of what I expect from (modern) fintech. * no tests * god method with no proper documentation/requirements * no one actually understanding the function deciding how much money to send! Isn't this one of the core functionalities of fintech products?
  - stjohnswarts 5y ago
    Just plain scary especially with money if they can't at the very least test the extremes and some in the middle values. Good gravy! What is the name of this company so I can avoid it...
- cjfd 5y ago
  The whole thing does not sound very competent. Like noone could possibly guarantee that the result is not off by a factor of 2 or 3 or perhaps even 10 or more in some cases. What would happen in that case? It also sounds like a domain where mistakes could be very expensive or lead to lawsuits or similar such things.
  - zerof1l 5y ago
    Things are not that bad. While the whole thing is kind of a monolith, there are at least broad sections that split code into logical sections. With some practice, it is possible to easily narrow down changes and any issues to 10-20 LOC. Later, breaking code by them could be a starting place for refactoring. In the end, the decision is manually reviewed. There's also a log that tells how and why the code had arrived at the numbers. So, if you know that client cannot be X but the log says X then you know that there's a bug for example. This code was around before I joined. It started small, but more and more requirements were added until it became like that. Not much effort went into structuring it. Now it's my responsibility, and I'm trying to make it better for me and anyone who will have to deal with it in the future.
- totorovirus 5y ago
  Does this mean there are at least 2^5 = 32 paths that a procedure can take? Seriously, is it even documentable?
CJefferson 5y ago
This might not be quite the same, but logic systems -- we might have a logic expression on a hundred variables, and need to check if it is true. While one would unit test with tiny expressions, often bugs only show up in large integration tests, as systems use many optimisations to cache and speed up behaviour, which can interact poorly

helsinkiandrew 5y ago

This all sounds like a race to the bottom, where automated testing will drive further kludge upon kludge and the code will get less understood and harder to maintain and write tests for. Whilst an automated test suite can at least check if the code may have broken - you'll never know if the break is significant or just an inconsequential side effect of a new piece of data. And if both the code and tests aren't understood, as you say the "right" fix won't be done and something will be added quickly to fix the test.

Chris2048 5y ago

So: - you don't know whether the code is correct or not - code is being modified on regular basis... and logic is changed If someone knows enough about the code to regularly change it, then they should be able to provide some documentation? How else are they able to modify a "correct" function and know that it's still correct? Can I guess that the code modifiers are quants, or something?

4140tm 5y ago

If no one knows whether the code is correct a generated test could easily do more harm than good. If no one is able/willing to refactor the code into understandable and testable bits, there's not much left to guarantee the correctness of it.

dmos62 5y ago

What you're describing is testing without understanding the code. I've not yet encountered inunderstandable code, and can't conceive what it might look like, though I've encountered code that I'd rather not try to understand.

mcherm 5y ago

In that situation, why do you WANT tests? If, for example, someone intends to refactor the code into something more maintainable and wants tests to prevent any accidental changes in functionality, then this is useful. If the goal is to catch bugs or to facilitate future development or to meet some test coverage standard then I argue you might be better off not having tests rather than having auto-generated tests that have not been carefully reviewed.

Ensorceled 5y ago

Years ago, in a similar situation, I wrote a tool to capture parameters and results for the algorithm while QA ran their full application test suite. Ended up with a few thousand cases and expected results; results where QA had verified the software worked as expected. Used that to create a test suite that we could then maintain normally. Much better than random.