3 ms·

What does your test coverage look like? Perhaps you're missing something there that would have caught that bug? Testing is, of course, no silver bullet. Tests

What does your test coverage look like? Perhaps you're missing something there that would have caught that bug? Testing is, of course, no silver bullet. Tests are written by humans, and humans make mistakes—and it's pretty difficult to achieve 100% test coverage in a production system. The goal of testing is to have confidence in the code you've written. Tests often don't need to cross version control boundaries. You can use mock data—like would be produced by the library—on the consumer side, because you can delegate responsibility for testing of that library to the library repository itself. If your tests work great with the mock data, but things are still failing, then you can infer that the mock data and the actual data are different, and your bug is in the library.

hinkley 9y ago
For instance, I have a piece of code in a particularly gnarly modules that at this moment is disabled and has been for two sprints due to emergent behavior. First sprint it had adequate unit tests but not enough functional tests to exhibit a problem. Second sprint I fixed the testing deficiency and got the code to work end to end. Or so I thought. The first time I turned it on it preprod I couldn't turn it back off again because some piece of data that came from five function calls away was being shared, and nobody who participated in the PR recalled that fact. Most of the code I'm dealing with is in a single module. I have been chipping away at fixing the insane ball of mud as I can. My coworkers often aren't that lucky. They come to me for advice on how to deal with this sort of problem but crossing 2 or three modules. There's no low-friction way for them to fix any of this. They can't just refactor because of the coordination costs, and also the loss of historical information when you move a block of code across module boundaries or try to change module boundaries. This is the prime argument for monorepos in the literature - not making irreversible decisions on Law of Demeter problems. It's not my biggest reason, but it's sufficient for most people.
- peterevans 9y ago
  Yeah, I sympathize—that's a tough situation. I think there's two ways you can look at your choice of configuration: ease of debugging, and ease of organization. When Google lays out why they use a monorepo, they are doing so because it simplifies their organization—there are no longer so many versions of so many libraries and apps they need to support; there's only one version of anything to support. Either everything works or everything fails. But in your case, you're looking at it from the debugging point of view. It's easier to play around with the code in a monorepo. And that's totally fair point of view to have, particularly in your predicament. That choice of a monorepo doesn't necessarily improve the quality of your code organization and interoperability. It's still going to be a bad bug to fix. It's just a little bit easier to debug.
crdoconnor 9y ago
>Tests often don't need to cross version control boundaries. You can use mock data—like would be produced by the library—on the consumer side, because you can delegate responsibility for testing of that library to the library repository itself. I take it as a rule of thumb that a more realistic test is better than a less realistic test. I can't think of any reason why you would want to mock anything if using the real thing is cheap and easy, building mocks is expensive, and testing against the real thing will increase the chances of detecting real bugs. I also prefer it when my tests detect bugs in other libraries which my code depends upon because as far as users are concerned, bugs in libraries my code depends upon are bugs in my code. I have in the past written a bunch of functional tests which check out / pull and build code from other repos to run with my code.
- mannykannot 9y ago
  Indeed, plus using a mock adds a whole new class of bugs, arising from the mock not precisely mimicking the real component or subsystem. As the sorts of bugs you are trying flush out in integration testing generally result from unforeseen interactions between components, it is more likely than chance that the mock will not behave in a way that will reveal the presence of the error.
perfmode 9y ago
Locally reasonable decisions can have globally unreasonable effects.

hinkley 9y ago

For instance, I have a piece of code in a particularly gnarly modules that at this moment is disabled and has been for two sprints due to emergent behavior. First sprint it had adequate unit tests but not enough functional tests to exhibit a problem. Second sprint I fixed the testing deficiency and got the code to work end to end. Or so I thought. The first time I turned it on it preprod I couldn't turn it back off again because some piece of data that came from five function calls away was being shared, and nobody who participated in the PR recalled that fact. Most of the code I'm dealing with is in a single module. I have been chipping away at fixing the insane ball of mud as I can. My coworkers often aren't that lucky. They come to me for advice on how to deal with this sort of problem but crossing 2 or three modules. There's no low-friction way for them to fix any of this. They can't just refactor because of the coordination costs, and also the loss of historical information when you move a block of code across module boundaries or try to change module boundaries. This is the prime argument for monorepos in the literature - not making irreversible decisions on Law of Demeter problems. It's not my biggest reason, but it's sufficient for most people.

peterevans 9y ago
Yeah, I sympathize—that's a tough situation. I think there's two ways you can look at your choice of configuration: ease of debugging, and ease of organization. When Google lays out why they use a monorepo, they are doing so because it simplifies their organization—there are no longer so many versions of so many libraries and apps they need to support; there's only one version of anything to support. Either everything works or everything fails. But in your case, you're looking at it from the debugging point of view. It's easier to play around with the code in a monorepo. And that's totally fair point of view to have, particularly in your predicament. That choice of a monorepo doesn't necessarily improve the quality of your code organization and interoperability. It's still going to be a bad bug to fix. It's just a little bit easier to debug.

crdoconnor 9y ago

>Tests often don't need to cross version control boundaries. You can use mock data—like would be produced by the library—on the consumer side, because you can delegate responsibility for testing of that library to the library repository itself. I take it as a rule of thumb that a more realistic test is better than a less realistic test. I can't think of any reason why you would want to mock anything if using the real thing is cheap and easy, building mocks is expensive, and testing against the real thing will increase the chances of detecting real bugs. I also prefer it when my tests detect bugs in other libraries which my code depends upon because as far as users are concerned, bugs in libraries my code depends upon are bugs in my code. I have in the past written a bunch of functional tests which check out / pull and build code from other repos to run with my code.

mannykannot 9y ago
Indeed, plus using a mock adds a whole new class of bugs, arising from the mock not precisely mimicking the real component or subsystem. As the sorts of bugs you are trying flush out in integration testing generally result from unforeseen interactions between components, it is more likely than chance that the mock will not behave in a way that will reveal the presence of the error.

perfmode 9y ago

Locally reasonable decisions can have globally unreasonable effects.