My experience with things like this has been, when you write tests like that you end up writing many tests (I've found you end up writing more unit tests than e

My experience with things like this has been, when you write tests like that you end up writing many tests (I've found you end up writing more unit tests than end to end tests - I know you asserted the opposite, but I'm just not sure why you think that, I'd be interested to discuss further), which isn't necessarily a problem, but the tests are brittle and so many of them break for any given change, and it's very difficult to tell which of the breakages are real and quite easy to overlook it & end up copying over the new output. If you have 100 breaking tests and only 1 of them is important, it's really easy not to find it. For a project this large and complex, you could easily break that many tests, were the test suite to be brittle. So let's say we did introduce some unintended code in our output. Is this a problem? It's a problem if it changes the behavior of our program. If it doesn't change the behavior - it isn't a problem, unless proven otherwise (eg, it ends up producing something which is correct but confusing, and we get feedback from our customers). If it changes the behavior of our program, there exists an end to end test which will catch it. It may or may not be part of our test suite, but that's the case with every test suite; you do your best to anticipate errors, and when you discover you missed something, you add it. At my last position we had CYA tests which made naive assertions about the output of machine learning models. Every time we shipped a new model it'd break a bunch of tests. I'm not aware of any bugs that were caught this way but we did ship models with bugs several times. I tried to replace them, wherever I could, with a test that took some sort of measurement which was more stable. When those failed, they were worth investigating. Whether a C program maintains it's behavior when compiled to Rust is the perfect thing to measure. It captures the definition of a bug in a way which is invariant across versions. (Note: I do support writing unit tests generally, this is an exceptional case where there are lots of moving parts, not all of which we can control, and an absolutely massive surface to test; in cases like this, you need to create a defensible position, optimizing for fewest issues impacting customers and not eliminating all issues, because that isn't an attainable goal.)