5 ms·

You can't possibly say that with any confidence. I literally just finished reproducing a memory leak caused by an insane combination of circumstances. This was

by fhood 8y ago

You can't possibly say that with any confidence. I literally just finished reproducing a memory leak caused by an insane combination of circumstances. This was not something valgrind or any similar tool would find. It was not something that any rational human being would have thought to write tests for. Once code is no longer synchronous, and tasks are being juggled around and swapped between, it becomes virtually impossible to predict what might cause this sort of bug.

whatshisface 8y ago
The parent is saying that their tests normally catch memory leaks, not that their tests catch all memory leaks.
tekstar 8y ago
What's an example of a memory leak that can't be found by valgrind?
- viraptor 8y ago
  One happening in an embedded language with its own GC. Unless you can switch all allocations to explicit management for the tests, it's almost impossible to differentiate between live objects, free lists, cycles with deferred collection, and arenas that are supposed to be freed on exit.
  - spc476 8y ago
    That happened with me at work. A certain class of error would leak references and over time they would pile up. Fortunately, Lua (the embedded language in use) allows one to mark certain references as weak references and allow collection. The trick was in determining what to mark as weak references.
kbenson 8y ago
Tests only catch what you test for. In this case "memory leak" can be replaced with "site functions as expected", and the same argument you made applies, and yet it's still useful. Just because you can't determine the exact cause of the leak, or even detect all of them, doesn't mean that having some indication of memory usage initially and under test loads of different (reproducible) amounts at different intervals and comparing them across builds can't get you a hell of a lot of utility.
- chrisweekly 8y ago
  > "Tests only catch what you test for." Er, sort of. Integration and e2e tests often catch problems nobody specifically, explicitly envisioned. That's part of what makes them so valuable! Right? https://m.youtube.com/watch?v=0GypdsJulKE https://m.youtube.com/watch?v=0GypdsJulKE
  - kbenson 8y ago
    Sure. I think we're just working on slightly different meanings of what I said, which wasn't meant in the exact way it's often rolled out as a criticism. Part of the reason you do different, vague tests is the hope you'll catch some odd errors someone missed. So you are testing, vaguely and imperfectly but over a much wider area, in the hopes of finding things you can't specifically think of. Sort of like if you are planning to drive some car you recently did a bunch of work yourself to on a long trip. You've probably already tested all the things you think might be wrong, but that doesn't mean it's not worth taking it for a drive for an hour around town and on the freeway to see if the unexpected happens, so you can deal with it close to home. You're testing the car, just in a vague "shake it and see what falls out" way. You are specifically testing for that, it's just not guaranteed to find any or all problems. It's still better than nothing most times.
mikestew 8y ago
You can't possibly say that with any confidence In answer to the original question, I stand behind my answer with 100% confidence. The test infrastructures I build catch memory leaks all of time, and I do not consider it abnormal for them to do so. If you mean to say that I cannot be confident that I have caught all of the memory leaks, well duh, of course I can't. But that's not what was asked, and that's not what I said. To expand on what was actually asked, though, whether a framework catches all, some, or no memory leaks is irrelevant. Because if a team is at least testing for it, I'll bet they're just a bit more rigorous in their coding than a team that just deploys to production and waits to see what breaks.
- dawidw 8y ago
  > The test infrastructures I build catch memory leaks all of time [...] Could you elaborate on that, please? I'm curious. Thank you in advance!
  - mikestew 8y ago
    It would be very specific to the project. For instance, if it's an iOS/Mac project, Xcode has nice profiling tools to catch memory leaks. And of course clang's static analyzer as well. What I'm working on now is embedded Linux, so valgrind and some test suite to exercise the code. But in general, no matter the project, you need a few key pieces: 1. A test suite that thoroughly exercises the code. You'll need this for... 2. A tool such as valgrind to watch memory as your tests in #1 run. Build time, ad hoc, stick in the pipeline anywhere you like. 3. A static analyzer to take a first pass at the code and say, "hmm, this might leak/crash/kill puppies." So, yeah, the order is off as this should be the first thing you do. It can run on your dev machine or on the build machine. Expect lots of false positives. (optional) 4. A code coverage tool to make sure you're not missing key pieces of code that need exercise.
selimthegrim 8y ago
Interested to see an AAR
bunderbunder 8y ago
> You can't possibly say that with any confidence. Assuming your module's interface isn't too complicated, it shouldn't be terribly difficult to write a test fixture that fuzzes a module, and verifies that its memory consumption does not go outside a certain bound. The first round of bugs you'll catch might be design flaws more than true memory leaks - places where the module isn't designed to limit its own memory usage in the face of adverse (or adversarial) input. Get those pinned down, and then you can use it to catch memory leaks with confidence. Let it run for less time if you're only worried about catching fast leaks, and for more time if you need to catch the slow ones, too.

by fhood 8y ago

whatshisface 8y ago

The parent is saying that their tests normally catch memory leaks, not that their tests catch all memory leaks.

tekstar 8y ago

What's an example of a memory leak that can't be found by valgrind?

viraptor 8y ago
One happening in an embedded language with its own GC. Unless you can switch all allocations to explicit management for the tests, it's almost impossible to differentiate between live objects, free lists, cycles with deferred collection, and arenas that are supposed to be freed on exit.
- spc476 8y ago
  That happened with me at work. A certain class of error would leak references and over time they would pile up. Fortunately, Lua (the embedded language in use) allows one to mark certain references as weak references and allow collection. The trick was in determining what to mark as weak references.

kbenson 8y ago

Tests only catch what you test for. In this case "memory leak" can be replaced with "site functions as expected", and the same argument you made applies, and yet it's still useful. Just because you can't determine the exact cause of the leak, or even detect all of them, doesn't mean that having some indication of memory usage initially and under test loads of different (reproducible) amounts at different intervals and comparing them across builds can't get you a hell of a lot of utility.

chrisweekly 8y ago
> "Tests only catch what you test for." Er, sort of. Integration and e2e tests often catch problems nobody specifically, explicitly envisioned. That's part of what makes them so valuable! Right? https://m.youtube.com/watch?v=0GypdsJulKE https://m.youtube.com/watch?v=0GypdsJulKE
- kbenson 8y ago
  Sure. I think we're just working on slightly different meanings of what I said, which wasn't meant in the exact way it's often rolled out as a criticism. Part of the reason you do different, vague tests is the hope you'll catch some odd errors someone missed. So you are testing, vaguely and imperfectly but over a much wider area, in the hopes of finding things you can't specifically think of. Sort of like if you are planning to drive some car you recently did a bunch of work yourself to on a long trip. You've probably already tested all the things you think might be wrong, but that doesn't mean it's not worth taking it for a drive for an hour around town and on the freeway to see if the unexpected happens, so you can deal with it close to home. You're testing the car, just in a vague "shake it and see what falls out" way. You are specifically testing for that, it's just not guaranteed to find any or all problems. It's still better than nothing most times.

mikestew 8y ago

You can't possibly say that with any confidence In answer to the original question, I stand behind my answer with 100% confidence. The test infrastructures I build catch memory leaks all of time, and I do not consider it abnormal for them to do so. If you mean to say that I cannot be confident that I have caught all of the memory leaks, well duh, of course I can't. But that's not what was asked, and that's not what I said. To expand on what was actually asked, though, whether a framework catches all, some, or no memory leaks is irrelevant. Because if a team is at least testing for it, I'll bet they're just a bit more rigorous in their coding than a team that just deploys to production and waits to see what breaks.

dawidw 8y ago
> The test infrastructures I build catch memory leaks all of time [...] Could you elaborate on that, please? I'm curious. Thank you in advance!
- mikestew 8y ago
  It would be very specific to the project. For instance, if it's an iOS/Mac project, Xcode has nice profiling tools to catch memory leaks. And of course clang's static analyzer as well. What I'm working on now is embedded Linux, so valgrind and some test suite to exercise the code. But in general, no matter the project, you need a few key pieces: 1. A test suite that thoroughly exercises the code. You'll need this for... 2. A tool such as valgrind to watch memory as your tests in #1 run. Build time, ad hoc, stick in the pipeline anywhere you like. 3. A static analyzer to take a first pass at the code and say, "hmm, this might leak/crash/kill puppies." So, yeah, the order is off as this should be the first thing you do. It can run on your dev machine or on the build machine. Expect lots of false positives. (optional) 4. A code coverage tool to make sure you're not missing key pieces of code that need exercise.

selimthegrim 8y ago

Interested to see an AAR

bunderbunder 8y ago

> You can't possibly say that with any confidence. Assuming your module's interface isn't too complicated, it shouldn't be terribly difficult to write a test fixture that fuzzes a module, and verifies that its memory consumption does not go outside a certain bound. The first round of bugs you'll catch might be design flaws more than true memory leaks - places where the module isn't designed to limit its own memory usage in the face of adverse (or adversarial) input. Get those pinned down, and then you can use it to catch memory leaks with confidence. Let it run for less time if you're only worried about catching fast leaks, and for more time if you need to catch the slow ones, too.