5 ms·

Do testing frameworks normally catch memory leaks?

by plopz 8y ago

mikestew 8y ago
Mine do.
- fhood 8y ago
  You can't possibly say that with any confidence. I literally just finished reproducing a memory leak caused by an insane combination of circumstances. This was not something valgrind or any similar tool would find. It was not something that any rational human being would have thought to write tests for. Once code is no longer synchronous, and tasks are being juggled around and swapped between, it becomes virtually impossible to predict what might cause this sort of bug.
  - whatshisface 8y ago
    The parent is saying that their tests normally catch memory leaks, not that their tests catch all memory leaks.
  - tekstar 8y ago
    What's an example of a memory leak that can't be found by valgrind?
    - viraptor 8y ago
      One happening in an embedded language with its own GC. Unless you can switch all allocations to explicit management for the tests, it's almost impossible to differentiate between live objects, free lists, cycles with deferred collection, and arenas that are supposed to be freed on exit.
      - spc476 8y ago
        That happened with me at work. A certain class of error would leak references and over time they would pile up. Fortunately, Lua (the embedded language in use) allows one to mark certain references as weak references and allow collection. The trick was in determining what to mark as weak references.
  - kbenson 8y ago
    Tests only catch what you test for. In this case "memory leak" can be replaced with "site functions as expected", and the same argument you made applies, and yet it's still useful. Just because you can't determine the exact cause of the leak, or even detect all of them, doesn't mean that having some indication of memory usage initially and under test loads of different (reproducible) amounts at different intervals and comparing them across builds can't get you a hell of a lot of utility.
    - chrisweekly 8y ago
      > "Tests only catch what you test for." Er, sort of. Integration and e2e tests often catch problems nobody specifically, explicitly envisioned. That's part of what makes them so valuable! Right? https://m.youtube.com/watch?v=0GypdsJulKE https://m.youtube.com/watch?v=0GypdsJulKE
      - kbenson 8y ago
        Sure. I think we're just working on slightly different meanings of what I said, which wasn't meant in the exact way it's often rolled out as a criticism. Part of the reason you do different, vague tests is the hope you'll catch some odd errors someone missed. So you are testing, vaguely and imperfectly but over a much wider area, in the hopes of finding things you can't specifically think of. Sort of like if you are planning to drive some car you recently did a bunch of work yourself to on a long trip. You've probably already tested all the things you think might be wrong, but that doesn't mean it's not worth taking it for a drive for an hour around town and on the freeway to see if the unexpected happens, so you can deal with it close to home. You're testing the car, just in a vague "shake it and see what falls out" way. You are specifically testing for that, it's just not guaranteed to find any or all problems. It's still better than nothing most times.
  - mikestew 8y ago
    You can't possibly say that with any confidence In answer to the original question, I stand behind my answer with 100% confidence. The test infrastructures I build catch memory leaks all of time, and I do not consider it abnormal for them to do so. If you mean to say that I cannot be confident that I have caught all of the memory leaks, well duh, of course I can't. But that's not what was asked, and that's not what I said. To expand on what was actually asked, though, whether a framework catches all, some, or no memory leaks is irrelevant. Because if a team is at least testing for it, I'll bet they're just a bit more rigorous in their coding than a team that just deploys to production and waits to see what breaks.
    - dawidw 8y ago
      > The test infrastructures I build catch memory leaks all of time [...] Could you elaborate on that, please? I'm curious. Thank you in advance!
      - mikestew 8y ago
        It would be very specific to the project. For instance, if it's an iOS/Mac project, Xcode has nice profiling tools to catch memory leaks. And of course clang's static analyzer as well. What I'm working on now is embedded Linux, so valgrind and some test suite to exercise the code. But in general, no matter the project, you need a few key pieces: 1. A test suite that thoroughly exercises the code. You'll need this for... 2. A tool such as valgrind to watch memory as your tests in #1 run. Build time, ad hoc, stick in the pipeline anywhere you like. 3. A static analyzer to take a first pass at the code and say, "hmm, this might leak/crash/kill puppies." So, yeah, the order is off as this should be the first thing you do. It can run on your dev machine or on the build machine. Expect lots of false positives. (optional) 4. A code coverage tool to make sure you're not missing key pieces of code that need exercise.
  - selimthegrim 8y ago
    Interested to see an AAR
  - bunderbunder 8y ago
    > You can't possibly say that with any confidence. Assuming your module's interface isn't too complicated, it shouldn't be terribly difficult to write a test fixture that fuzzes a module, and verifies that its memory consumption does not go outside a certain bound. The first round of bugs you'll catch might be design flaws more than true memory leaks - places where the module isn't designed to limit its own memory usage in the face of adverse (or adversarial) input. Get those pinned down, and then you can use it to catch memory leaks with confidence. Let it run for less time if you're only worried about catching fast leaks, and for more time if you need to catch the slow ones, too.
- aLifeBot 8y ago
  Which framework do you use?
  - flukus 8y ago
    Not OP but make. It's not exactly a framework but I've got a project where every test get's run twice, first as a test to check the functionality, then run under valgrind. It won't catch all possible errors but it will catch most. On the plus side it will only nag about actual errors, not potential errors like "safe languages".
yashap 8y ago
If you look at this quote one the article: “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.” That’s exactly the problem that thorough automated test suites try to solve. When you have a robust mix of unit, integration, end-to-end and performance tests, you can greatly reduce the number of unintended regressions that make it into production. Not eliminate them completely, but definitely reduce them a tonne.

Do testing frameworks normally catch memory leaks?

by plopz 8y ago

Do testing frameworks normally catch memory leaks?

mikestew 8y ago

Mine do.

fhood 8y ago
You can't possibly say that with any confidence. I literally just finished reproducing a memory leak caused by an insane combination of circumstances. This was not something valgrind or any similar tool would find. It was not something that any rational human being would have thought to write tests for. Once code is no longer synchronous, and tasks are being juggled around and swapped between, it becomes virtually impossible to predict what might cause this sort of bug.
- whatshisface 8y ago
  The parent is saying that their tests normally catch memory leaks, not that their tests catch all memory leaks.
- tekstar 8y ago
  What's an example of a memory leak that can't be found by valgrind?
  - viraptor 8y ago
    One happening in an embedded language with its own GC. Unless you can switch all allocations to explicit management for the tests, it's almost impossible to differentiate between live objects, free lists, cycles with deferred collection, and arenas that are supposed to be freed on exit.
    - spc476 8y ago
      That happened with me at work. A certain class of error would leak references and over time they would pile up. Fortunately, Lua (the embedded language in use) allows one to mark certain references as weak references and allow collection. The trick was in determining what to mark as weak references.
- kbenson 8y ago
  Tests only catch what you test for. In this case "memory leak" can be replaced with "site functions as expected", and the same argument you made applies, and yet it's still useful. Just because you can't determine the exact cause of the leak, or even detect all of them, doesn't mean that having some indication of memory usage initially and under test loads of different (reproducible) amounts at different intervals and comparing them across builds can't get you a hell of a lot of utility.
  - chrisweekly 8y ago
    > "Tests only catch what you test for." Er, sort of. Integration and e2e tests often catch problems nobody specifically, explicitly envisioned. That's part of what makes them so valuable! Right? https://m.youtube.com/watch?v=0GypdsJulKE https://m.youtube.com/watch?v=0GypdsJulKE
    - kbenson 8y ago
      Sure. I think we're just working on slightly different meanings of what I said, which wasn't meant in the exact way it's often rolled out as a criticism. Part of the reason you do different, vague tests is the hope you'll catch some odd errors someone missed. So you are testing, vaguely and imperfectly but over a much wider area, in the hopes of finding things you can't specifically think of. Sort of like if you are planning to drive some car you recently did a bunch of work yourself to on a long trip. You've probably already tested all the things you think might be wrong, but that doesn't mean it's not worth taking it for a drive for an hour around town and on the freeway to see if the unexpected happens, so you can deal with it close to home. You're testing the car, just in a vague "shake it and see what falls out" way. You are specifically testing for that, it's just not guaranteed to find any or all problems. It's still better than nothing most times.
- mikestew 8y ago
  You can't possibly say that with any confidence In answer to the original question, I stand behind my answer with 100% confidence. The test infrastructures I build catch memory leaks all of time, and I do not consider it abnormal for them to do so. If you mean to say that I cannot be confident that I have caught all of the memory leaks, well duh, of course I can't. But that's not what was asked, and that's not what I said. To expand on what was actually asked, though, whether a framework catches all, some, or no memory leaks is irrelevant. Because if a team is at least testing for it, I'll bet they're just a bit more rigorous in their coding than a team that just deploys to production and waits to see what breaks.
  - dawidw 8y ago
    > The test infrastructures I build catch memory leaks all of time [...] Could you elaborate on that, please? I'm curious. Thank you in advance!
    - mikestew 8y ago
      It would be very specific to the project. For instance, if it's an iOS/Mac project, Xcode has nice profiling tools to catch memory leaks. And of course clang's static analyzer as well. What I'm working on now is embedded Linux, so valgrind and some test suite to exercise the code. But in general, no matter the project, you need a few key pieces: 1. A test suite that thoroughly exercises the code. You'll need this for... 2. A tool such as valgrind to watch memory as your tests in #1 run. Build time, ad hoc, stick in the pipeline anywhere you like. 3. A static analyzer to take a first pass at the code and say, "hmm, this might leak/crash/kill puppies." So, yeah, the order is off as this should be the first thing you do. It can run on your dev machine or on the build machine. Expect lots of false positives. (optional) 4. A code coverage tool to make sure you're not missing key pieces of code that need exercise.
- selimthegrim 8y ago
  Interested to see an AAR
- bunderbunder 8y ago
  > You can't possibly say that with any confidence. Assuming your module's interface isn't too complicated, it shouldn't be terribly difficult to write a test fixture that fuzzes a module, and verifies that its memory consumption does not go outside a certain bound. The first round of bugs you'll catch might be design flaws more than true memory leaks - places where the module isn't designed to limit its own memory usage in the face of adverse (or adversarial) input. Get those pinned down, and then you can use it to catch memory leaks with confidence. Let it run for less time if you're only worried about catching fast leaks, and for more time if you need to catch the slow ones, too.
aLifeBot 8y ago
Which framework do you use?
- flukus 8y ago
  Not OP but make. It's not exactly a framework but I've got a project where every test get's run twice, first as a test to check the functionality, then run under valgrind. It won't catch all possible errors but it will catch most. On the plus side it will only nag about actual errors, not potential errors like "safe languages".

yashap 8y ago

If you look at this quote one the article: “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.” That’s exactly the problem that thorough automated test suites try to solve. When you have a robust mix of unit, integration, end-to-end and performance tests, you can greatly reduce the number of unintended regressions that make it into production. Not eliminate them completely, but definitely reduce them a tonne.