4 ms·

>Depends on how you're defining both "developed apps" and "indie". Spent time (and money) on your own initiative to develop an app that's been published in the

>Depends on how you're defining both "developed apps" and "indie". Spent time (and money) on your own initiative to develop an app that's been published in the Android and iOS stores. I spent my own time developing a game that's in the Android and iOS app stores. > How likely is it that you'd successfully handle all execution of your app even with real testing? My point wasn't about bugs. My point was about execution path coverage: If code has never been hit, then it may contain hidden bugs. And if I'm doing things right, 100% of the code would be executed in testing. There are techniques to specifically track code coverage, so you don't need to wonder whether a code path has been triggered, but in general I don't write code and then never execute it. > But I still think you're continuing to overstate the difficulty of minimizing the combinatorial effects and general difficulty of handling missing permissions. It's your right to believe as you will.

MaulingMonkey 13y ago
> on your own initiative Then no. > My point wasn't about bugs. My point was about execution path coverage: If code has never been hit, then it may contain hidden bugs. "it may contain hidden bugs" sounds like, at the end of the day, your point was about bugs. Any distinction you're trying to make is too subtle for me to pick up on without spelling it out more. > And if I'm doing things right, 100% of the code would be executed in testing. There's a big difference between executing all code, and executing all permutable paths into said code on a micro level (e.g. function by function), and executing all permutable paths into said code on a macro level. I've been in the unfortunate position of having even the first option being literally impossible for parts of a codebase. Lack of complete test environments for your mandatory 3rd party APIs suck! You can write your own mocks, but that's not exactly the same thing -- especially when you don't know what kind of quirks to expect. And as you might expect, it did lead to a bug! Fortunately, patching on both Android and iOS is significantly easier, and equally fortunately, the bug was later found, patched, and the patch shipped successfully. And while I think "complete" test coverage of every possible permutation is a nice thing to strive for on a micro level, at a sufficiently macro level it becomes effectively impossible. Techniques like fuzzing let you test that a significant majority of combinations works out in your favor, but there will always be the 0.0001% corner case. Well before "impossible" comes "overkill". I suspect we have a fundamental disagreement of opinion as to what constitutes "overkill".
- SomeCallMeTim 13y ago
  >your point was about bugs Of course it was. It's always about bugs. But code that has never been executed can have orders of magnitude more categories of bug than code that works at least once. And I'm good enough that 99% of the time I don't miss the corner cases -- it's the more blatant problems that tend to bite me, and those are eliminated by making sure all the code gets executed at least once. Once my code is wired up right, it tends to work. But I won't know if it's wired up right until it's been executed once. >Lack of complete test environments for your mandatory 3rd party APIs suck! Try writing code for systems without the ability to debug. Or where something is going wrong and there's no way to figure out why based on logging or debugging. Or without complete (or correct!) documentation. Or for projects where, once you ship, there is no option to patch your app, so it needs to be as close as possible to bug free. Thankfully modern platforms (mostly!) avoid the last pitfall, but I've been there, along with the other situations. And I win these battles. >I suspect we have a fundamental disagreement of opinion as to what constitutes "overkill". No, I'm just not communicating effectively. And it's no longer worth it for me to try any harder. FWIW, I'm not talking about complete test coverage. The apps I usually write (games) are almost entirely pointless to write tests for. Ever. Sure you can for some specific libraries you're using, but for the actual game itself? Hardly ever worth it. I've shipped games with zero chance for patching, some pretty popular -- and I didn't hear of any bugs worth mentioning after release. And there were no tests involved in the development process (other than the kind where QA tests the game and tells me what doesn't work). I know that concept is a bit alien on HN, but it's rare to find a game (or even a game engine) with a nontrivial test suite. Most game bugs wouldn't be prevented with unit tests anyway -- probably only one in a hundred at best. They almost always result from emergent complexity, not from a single function returning a bad value.
  - MaulingMonkey 13y ago
    > it's the more blatant problems that tend to bite me, and those are eliminated by making sure all the code gets executed at least once Maybe I'm taking your repeated reference to testing all 256 permutations too literally. > Try writing code for systems without the ability to debug. Or where something is going wrong and there's no way to figure out why based on logging or debugging. Or without complete (or correct!) documentation. Or for projects where, once you ship, there is no option to patch your app, so it needs to be as close as possible to bug free. Thankfully modern platforms (mostly!) avoid the last pitfall, but I've been there, along with the other situations. And I win these battles. At least I've dodged one of those bullets: I haven't worked on anything outright unpatchable just yet :). > No, I'm just not communicating effectively. And it's no longer worth it for me to try any harder. I appreciate the effort you've made.

MaulingMonkey 13y ago

> on your own initiative Then no. > My point wasn't about bugs. My point was about execution path coverage: If code has never been hit, then it may contain hidden bugs. "it may contain hidden bugs" sounds like, at the end of the day, your point was about bugs. Any distinction you're trying to make is too subtle for me to pick up on without spelling it out more. > And if I'm doing things right, 100% of the code would be executed in testing. There's a big difference between executing all code, and executing all permutable paths into said code on a micro level (e.g. function by function), and executing all permutable paths into said code on a macro level. I've been in the unfortunate position of having even the first option being literally impossible for parts of a codebase. Lack of complete test environments for your mandatory 3rd party APIs suck! You can write your own mocks, but that's not exactly the same thing -- especially when you don't know what kind of quirks to expect. And as you might expect, it did lead to a bug! Fortunately, patching on both Android and iOS is significantly easier, and equally fortunately, the bug was later found, patched, and the patch shipped successfully. And while I think "complete" test coverage of every possible permutation is a nice thing to strive for on a micro level, at a sufficiently macro level it becomes effectively impossible. Techniques like fuzzing let you test that a significant majority of combinations works out in your favor, but there will always be the 0.0001% corner case. Well before "impossible" comes "overkill". I suspect we have a fundamental disagreement of opinion as to what constitutes "overkill".

SomeCallMeTim 13y ago
>your point was about bugs Of course it was. It's always about bugs. But code that has never been executed can have orders of magnitude more categories of bug than code that works at least once. And I'm good enough that 99% of the time I don't miss the corner cases -- it's the more blatant problems that tend to bite me, and those are eliminated by making sure all the code gets executed at least once. Once my code is wired up right, it tends to work. But I won't know if it's wired up right until it's been executed once. >Lack of complete test environments for your mandatory 3rd party APIs suck! Try writing code for systems without the ability to debug. Or where something is going wrong and there's no way to figure out why based on logging or debugging. Or without complete (or correct!) documentation. Or for projects where, once you ship, there is no option to patch your app, so it needs to be as close as possible to bug free. Thankfully modern platforms (mostly!) avoid the last pitfall, but I've been there, along with the other situations. And I win these battles. >I suspect we have a fundamental disagreement of opinion as to what constitutes "overkill". No, I'm just not communicating effectively. And it's no longer worth it for me to try any harder. FWIW, I'm not talking about complete test coverage. The apps I usually write (games) are almost entirely pointless to write tests for. Ever. Sure you can for some specific libraries you're using, but for the actual game itself? Hardly ever worth it. I've shipped games with zero chance for patching, some pretty popular -- and I didn't hear of any bugs worth mentioning after release. And there were no tests involved in the development process (other than the kind where QA tests the game and tells me what doesn't work). I know that concept is a bit alien on HN, but it's rare to find a game (or even a game engine) with a nontrivial test suite. Most game bugs wouldn't be prevented with unit tests anyway -- probably only one in a hundred at best. They almost always result from emergent complexity, not from a single function returning a bad value.
- MaulingMonkey 13y ago
  > it's the more blatant problems that tend to bite me, and those are eliminated by making sure all the code gets executed at least once Maybe I'm taking your repeated reference to testing all 256 permutations too literally. > Try writing code for systems without the ability to debug. Or where something is going wrong and there's no way to figure out why based on logging or debugging. Or without complete (or correct!) documentation. Or for projects where, once you ship, there is no option to patch your app, so it needs to be as close as possible to bug free. Thankfully modern platforms (mostly!) avoid the last pitfall, but I've been there, along with the other situations. And I win these battles. At least I've dodged one of those bullets: I haven't worked on anything outright unpatchable just yet :). > No, I'm just not communicating effectively. And it's no longer worth it for me to try any harder. I appreciate the effort you've made.