3 ms·

I'd argue that these tests are more robust to changes in Rust and changes in C2Rust that change the output in trivial ways. I don't see how you could maintain t

I'd argue that these tests are more robust to changes in Rust and changes in C2Rust that change the output in trivial ways. I don't see how you could maintain the sort of test suite you're describing in a project like this. If you made a change that changed the Rust output, you'd invalidate huge parts of your test suite and generate lots of noisey failures. It'd make it fast too expensive to introduce all but the most critical changes. We don't care what Rust gets generated; we care that the Rust which is generated has the correct behavior. Testing that is where the value is.

kazinator 4y ago
> If you made a change that changed the Rust output, you'd invalidate huge parts of your test suite and generate lots of noisey failures. If that change was unintended, you'd be thanking yourself for unit tests. The unit test suite doesn't have to be all that large. It's the behavioral test suite which has to be large in order to generate confidence. > It'd make it fast too expensive to introduce all but the most critical changes. You can easily have diffs between the expected output of those cases and the new output. You can review those and merge them, which is time-consuming work, but of great value. You can spot bugs in the review, like whoa, this thing is now being translated in a bad way. If the project is in a state of flux, the new expected outputs can be more or less blindly merged; still better than nothing, and there is a record that can be revisited. Ah, that test case is actually confirming the wrong thing, which was right previous to this commit when the output changed.
- maxbond 4y ago
  My experience with things like this has been, when you write tests like that you end up writing many tests (I've found you end up writing more unit tests than end to end tests - I know you asserted the opposite, but I'm just not sure why you think that, I'd be interested to discuss further), which isn't necessarily a problem, but the tests are brittle and so many of them break for any given change, and it's very difficult to tell which of the breakages are real and quite easy to overlook it & end up copying over the new output. If you have 100 breaking tests and only 1 of them is important, it's really easy not to find it. For a project this large and complex, you could easily break that many tests, were the test suite to be brittle. So let's say we did introduce some unintended code in our output. Is this a problem? It's a problem if it changes the behavior of our program. If it doesn't change the behavior - it isn't a problem, unless proven otherwise (eg, it ends up producing something which is correct but confusing, and we get feedback from our customers). If it changes the behavior of our program, there exists an end to end test which will catch it. It may or may not be part of our test suite, but that's the case with every test suite; you do your best to anticipate errors, and when you discover you missed something, you add it. At my last position we had CYA tests which made naive assertions about the output of machine learning models. Every time we shipped a new model it'd break a bunch of tests. I'm not aware of any bugs that were caught this way but we did ship models with bugs several times. I tried to replace them, wherever I could, with a test that took some sort of measurement which was more stable. When those failed, they were worth investigating. Whether a C program maintains it's behavior when compiled to Rust is the perfect thing to measure. It captures the definition of a bug in a way which is invariant across versions. (Note: I do support writing unit tests generally, this is an exceptional case where there are lots of moving parts, not all of which we can control, and an absolutely massive surface to test; in cases like this, you need to create a defensible position, optimizing for fewest issues impacting customers and not eliminating all issues, because that isn't an attainable goal.)

kazinator 4y ago

> If you made a change that changed the Rust output, you'd invalidate huge parts of your test suite and generate lots of noisey failures. If that change was unintended, you'd be thanking yourself for unit tests. The unit test suite doesn't have to be all that large. It's the behavioral test suite which has to be large in order to generate confidence. > It'd make it fast too expensive to introduce all but the most critical changes. You can easily have diffs between the expected output of those cases and the new output. You can review those and merge them, which is time-consuming work, but of great value. You can spot bugs in the review, like whoa, this thing is now being translated in a bad way. If the project is in a state of flux, the new expected outputs can be more or less blindly merged; still better than nothing, and there is a record that can be revisited. Ah, that test case is actually confirming the wrong thing, which was right previous to this commit when the output changed.

maxbond 4y ago
My experience with things like this has been, when you write tests like that you end up writing many tests (I've found you end up writing more unit tests than end to end tests - I know you asserted the opposite, but I'm just not sure why you think that, I'd be interested to discuss further), which isn't necessarily a problem, but the tests are brittle and so many of them break for any given change, and it's very difficult to tell which of the breakages are real and quite easy to overlook it & end up copying over the new output. If you have 100 breaking tests and only 1 of them is important, it's really easy not to find it. For a project this large and complex, you could easily break that many tests, were the test suite to be brittle. So let's say we did introduce some unintended code in our output. Is this a problem? It's a problem if it changes the behavior of our program. If it doesn't change the behavior - it isn't a problem, unless proven otherwise (eg, it ends up producing something which is correct but confusing, and we get feedback from our customers). If it changes the behavior of our program, there exists an end to end test which will catch it. It may or may not be part of our test suite, but that's the case with every test suite; you do your best to anticipate errors, and when you discover you missed something, you add it. At my last position we had CYA tests which made naive assertions about the output of machine learning models. Every time we shipped a new model it'd break a bunch of tests. I'm not aware of any bugs that were caught this way but we did ship models with bugs several times. I tried to replace them, wherever I could, with a test that took some sort of measurement which was more stable. When those failed, they were worth investigating. Whether a C program maintains it's behavior when compiled to Rust is the perfect thing to measure. It captures the definition of a bug in a way which is invariant across versions. (Note: I do support writing unit tests generally, this is an exceptional case where there are lots of moving parts, not all of which we can control, and an absolutely massive surface to test; in cases like this, you need to create a defensible position, optimizing for fewest issues impacting customers and not eliminating all issues, because that isn't an attainable goal.)