3 ms·
Sampling some directories and files in the test suite of this project, I see a problem: testing is done by translating C to rust and compiling it, and then test
by kazinator 4y ago
Sampling some directories and files in the test suite of this project, I see a problem: testing is done by translating C to rust and compiling it, and then testing the run-time behavior of the result. I don't see test cases which cover the behavior of the translator directly: like that a certain C language input maps to a certain Rust output.
- samus 4y agoThis tool is there to automate the boring parts of converting codebases. The real work is verifying the parts that rely on undefined behavior. High fidelity and reliability would be a concern if the transpiler is used to regularly sync between C and Rust versions of a same codebase. It's less of a problem for one-off efforts. Those will usually be heavily tested and inspected before the result is trusted. Edit: it would of course be nice to see proper quality control applied, but prototypes have to proof quickly that they are worth the time spent refining them.
- kazinator 4y agoA source-to-source translator is a textbook example of something that is slam-dunk unit testable. There is almost no valid argument against doing it. Note: maybe this project has it in there; I haven't exhaustively looked into every subdirectory. > if the transpiler is used to regularly sync between C and Rust versions of a same codebase How do you know it won't be used that way? Because the maintainers of every C codebase will stop what they are doing in C, and follow the Rust conversion as soon as they hear about it? Even if you use this tool to permanently cut some code base over to Rust, it would be nice there to be some assurance about what it's doing beyond just "the converted code seems to do the same thing". A conversion could be done two or more times even if the C code isn't changing. Say you do the conversion. Then hack on the converted code. A new version of the converter comes out claiming to fix bugs. You might want to re-run it on the original code again, see if anything changed, and merge those changes to the current code stream that already contains modifications. > The real work is verifying the parts that rely on undefined behavior. That is neither here nor there. A construct that is confirmed undefined in C can be translated to the call to a Rust function that makes damons fly out of your nose. And there can be a couple of unit tests confirming this translation strategy. Or else, something else can be done. E.g. let's say the C code relies on wraparound two's complement arithmetic. The translator can oblige and generate code which makes that work (making that translator more helpful than some modern C compilers).
- samus 4y agoTests could verify that the transpiler correctly maps certain C constructs, but that almost doesn't matter. There is a fair chance that the translation won't work anyways if there's too much C sorcery and undefined behavior involved. The idea of continuous two-way synchronisation between codebases is migraine-inducing to begin with. Even though Rust can probably transpiled to C with way less risk of losing fidelity. But I wouldn't be so sure that this always works out the more `unsafe` blocks the Rust version requires. C compilers are not designed to minimize UB after all, and the results can be very surprising. Yes, I hope people are sane enough to eventually commit to a somewhat tidied-up Rust version and to only keep the C version around to conduct software archeology. Of course, this will produce a hard fork of the codebase, and the usual political reasons specific to such efforts apply.
- maxbond 4y agoI'd argue that these tests are more robust to changes in Rust and changes in C2Rust that change the output in trivial ways. I don't see how you could maintain the sort of test suite you're describing in a project like this. If you made a change that changed the Rust output, you'd invalidate huge parts of your test suite and generate lots of noisey failures. It'd make it fast too expensive to introduce all but the most critical changes. We don't care what Rust gets generated; we care that the Rust which is generated has the correct behavior. Testing that is where the value is.
- kazinator 4y ago> If you made a change that changed the Rust output, you'd invalidate huge parts of your test suite and generate lots of noisey failures. If that change was unintended, you'd be thanking yourself for unit tests. The unit test suite doesn't have to be all that large. It's the behavioral test suite which has to be large in order to generate confidence. > It'd make it fast too expensive to introduce all but the most critical changes. You can easily have diffs between the expected output of those cases and the new output. You can review those and merge them, which is time-consuming work, but of great value. You can spot bugs in the review, like whoa, this thing is now being translated in a bad way. If the project is in a state of flux, the new expected outputs can be more or less blindly merged; still better than nothing, and there is a record that can be revisited. Ah, that test case is actually confirming the wrong thing, which was right previous to this commit when the output changed.
- maxbond 4y agoMy experience with things like this has been, when you write tests like that you end up writing many tests (I've found you end up writing more unit tests than end to end tests - I know you asserted the opposite, but I'm just not sure why you think that, I'd be interested to discuss further), which isn't necessarily a problem, but the tests are brittle and so many of them break for any given change, and it's very difficult to tell which of the breakages are real and quite easy to overlook it & end up copying over the new output. If you have 100 breaking tests and only 1 of them is important, it's really easy not to find it. For a project this large and complex, you could easily break that many tests, were the test suite to be brittle. So let's say we did introduce some unintended code in our output. Is this a problem? It's a problem if it changes the behavior of our program. If it doesn't change the behavior - it isn't a problem, unless proven otherwise (eg, it ends up producing something which is correct but confusing, and we get feedback from our customers). If it changes the behavior of our program, there exists an end to end test which will catch it. It may or may not be part of our test suite, but that's the case with every test suite; you do your best to anticipate errors, and when you discover you missed something, you add it. At my last position we had CYA tests which made naive assertions about the output of machine learning models. Every time we shipped a new model it'd break a bunch of tests. I'm not aware of any bugs that were caught this way but we did ship models with bugs several times. I tried to replace them, wherever I could, with a test that took some sort of measurement which was more stable. When those failed, they were worth investigating. Whether a C program maintains it's behavior when compiled to Rust is the perfect thing to measure. It captures the definition of a bug in a way which is invariant across versions. (Note: I do support writing unit tests generally, this is an exceptional case where there are lots of moving parts, not all of which we can control, and an absolutely massive surface to test; in cases like this, you need to create a defensible position, optimizing for fewest issues impacting customers and not eliminating all issues, because that isn't an attainable goal.)