4 ms·

I wrote a UDF for MySQL. It had just shy of 100% coverage. It used a malloc. I could not figure out how to trigger a malloc failure inside of a imported library

by dalke 6y ago

I wrote a UDF for MySQL. It had just shy of 100% coverage. It used a malloc. I could not figure out how to trigger a malloc failure inside of a imported library running on a MySQL instance. Do I not check for malloc failure (in order to get 100%)? Or do I develop some sort of instrumented version of MySQL which lets me test malloc failure? That seemed far more overkill than warranted for the small project I worked on, though I certainly know some projects which do that! I decided to accept 99.5% or so coverage and manual inspection of the failure case.

mathgladiator 6y ago
A core cultural challenge that we have to deal with is that people are very soft about testing, and they don't make their artifacts or platforms testable. There is a bit of diminishing returns to what I believe, but it also depends on ones dependencies and what one considers 100%. As an example, I designed a multi-TB file-format and synchronization engine. I made that engine a dependency free library such that all the math, contracts, threading, and scheduling could be 100% tested before integration. This forced the messy integration into the core application which then was E2E tested with hundreds of randomized tests. The key concept at play was beating all the math into the ground such that it was damn near perfect because a single flaw would lose data.
- dalke 6y ago
  Yes, I figure it would have been easy to get to 100% if MySQL had been designed so that plugins could be coverage tested. For example, there might be an "environment" API which abstracts memory management and file I/O. The plugin could use this API instead of direct calls. Then if MySQL had an option to use a specific environment implementation for a plugin, then testers could drop in their own implementation under test. Getting that sort of support by MySQL is an example of what I think you mean by "cultural challenge."
strken 6y ago
I believe you can override libc's malloc by passing in LD_PRELOAD, at which point you've got the basic building blocks to test malloc failure if you can find a way to tell your malloc call apart from the others. Having said that, I don't disagree with your choice, because messing with LD_PRELOAD is difficult, scary, and probably overkill anyway.
nullc 6y ago
> Could not figure out how to trigger a malloc failure inside of a imported library Glibc has (or at least has had) special (unsupported) instrumentation that lets you substitute out the libc malloc at runtime with another function. You can use this to shim in a version that fails after a specific number of executions in order to inject malloc faults. E.g. https://github.com/xiph/opus/blob/master/tests/test_opus_api.c#L1751 https://github.com/xiph/opus/blob/master/tests/test_opus_api...
- dalke 6y ago
  I considered using a shim, but couldn't figure out which malloc comes from my UDF and which comes from MySQL, and I was highly doubtful that MySQL would have identical numbers of malloc by the time my code runs.

by dalke 6y ago

mathgladiator 6y ago

A core cultural challenge that we have to deal with is that people are very soft about testing, and they don't make their artifacts or platforms testable. There is a bit of diminishing returns to what I believe, but it also depends on ones dependencies and what one considers 100%. As an example, I designed a multi-TB file-format and synchronization engine. I made that engine a dependency free library such that all the math, contracts, threading, and scheduling could be 100% tested before integration. This forced the messy integration into the core application which then was E2E tested with hundreds of randomized tests. The key concept at play was beating all the math into the ground such that it was damn near perfect because a single flaw would lose data.

dalke 6y ago
Yes, I figure it would have been easy to get to 100% if MySQL had been designed so that plugins could be coverage tested. For example, there might be an "environment" API which abstracts memory management and file I/O. The plugin could use this API instead of direct calls. Then if MySQL had an option to use a specific environment implementation for a plugin, then testers could drop in their own implementation under test. Getting that sort of support by MySQL is an example of what I think you mean by "cultural challenge."

strken 6y ago

I believe you can override libc's malloc by passing in LD_PRELOAD, at which point you've got the basic building blocks to test malloc failure if you can find a way to tell your malloc call apart from the others. Having said that, I don't disagree with your choice, because messing with LD_PRELOAD is difficult, scary, and probably overkill anyway.

nullc 6y ago

> Could not figure out how to trigger a malloc failure inside of a imported library Glibc has (or at least has had) special (unsupported) instrumentation that lets you substitute out the libc malloc at runtime with another function. You can use this to shim in a version that fails after a specific number of executions in order to inject malloc faults. E.g. https://github.com/xiph/opus/blob/master/tests/test_opus_api.c#L1751 https://github.com/xiph/opus/blob/master/tests/test_opus_api...

dalke 6y ago
I considered using a shim, but couldn't figure out which malloc comes from my UDF and which comes from MySQL, and I was highly doubtful that MySQL would have identical numbers of malloc by the time my code runs.