6 ms·
How does it feel to test a compiler?
- lucidguppy 2y agoCan you imagine being one of the five people in the universe that has confident knowledge of c++ undefined behavior?
- JonChesterfield 2y agoJetbrains have a really solid testing team. Lots of places have a two tier system where the real developers write the code and those who don't make the cut test the code, with pay delta and an aspiration of being promoted out of testing. Other places have a mandatory stint in testing for new developments as a way to get some headcount on the task. Jetbrains don't do that. Or at least they didn't sometime before covid when I met a bunch of their test devs at a conference. The developers mostly doing testing were equals to those mostly doing product work. Possibly with a more extreme bias towards case analysis. I don't think it's a coincidence that jetbrains treat their test team as peers to the others and that their software seems to mostly not fall over in the field.
- gleenn 2y agoI feel like developers who don't write tests while writing their production code aren't doing a real job in 2024. Sure you might have a QA team or something, but that should be a vast minority of the tests being written. Test driven development and Agile and all that had some people upset and whiny 20 years ago, but if I see code without tests, I assume it's broken. The only way anyone knows anything works both now AND when the next 10, 100, 1000 commits roll in, your manually-tested edge cases have long been forgotten and I doubt you even got it right at all. Let's keep normalizing writing tests and thinking, up front, how to ensure our code works. That's the best time to think about all those pesky edge cases, and it's the only practical way to keep code working. Don't hoist this on to some poor chum because you are lazy or don't know how. (Spelling)
- stouset 2y agoMost tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a test written by others. I have been a software developer for 25 years across multiple languages and ecosystems. Everyone jumped on the testing bandwagon but writing code that is testable is a learned skill that nobody bothers to learn. Instead we end up with overly-mocked tests that—in practice—test that “the code is written like it currently is” rather than that it actually behaves correctly. These kinds of tests actually provide negative value. Besides taking up time during every PR and build, they fire off on any attempt to clean up or refactor. Every time you edit the code you have to edit the tests which entirely defeats the point. The solution ends up being, unsurprisingly, lessons learned from the functional world: don’t access or manipulate external state, operate only on direct inputs and only manipulate your outputs. As a result code ends up being dumb, short, and obvious. But nobody learns to write code this way, so two hundred line spaghetti functions are the norm.
- kortilla 2y agoIt sounds like you’re specifically talking about unit tests and people that suck at writing those. I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do. Don’t throw the baby out with the bathwater
- SatvikBeri 2y agoIt's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?" At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.
- yreg 2y agoA class system with developers writing production code looking down on developers writing tests doesn't seem healthy to me.
- BobbyTables2 2y agoIt never is!
- kabdib 2y agoI've had "quality-team-as-peer" situations only a few times in my career, and it's been great every time. Getting meaningful feedback on specs and early builds from people who knew what they were doing (and were happy doing it) was like the first time I had a real editor (the human kind) marking up my writing, just a wonderful feeling that someone had your back and that you all had the same goal: Happy customers.
- BobbyTables2 2y agoIt is indeed shocking what happens when a company doesn’t treat its (offshore) QA team as third class peons relative to the developers. I’ve seen both and have actually seen much higher product quality when the QA team is smaller (or even almost non existent). Turns out white-box testing usually fails to catch things only SMEs would know.
- HideousKojima 2y agoI frankly think my company's offshore devs and QA (largely India-based) are a net negative. The existence of a decent sized team to do QA manually has made adoption of automated testing (and writing code that it easily testable) almost impossible because no one wants to change.
- spc476 2y agoAt my previous job, prior to new management (we got bought out), my team worked closely with QA during the development process---they would get daily, often times, more than daily, builds of our project and so by the time it got to the staging environment, it pretty much worked [1]. New management comes in, forces a hard wall between us and QA and pretty much, we went from "favored vendor" of our customer [2], to "least favorite vendor" in less than a year. One deployment took four attempts to get right. Bugs found left and right. All because the new management didn't want to "pollute" QA's testing mindset with implementation details. Sigh. [1] During the 11 years I was there, there were only two deployments that were reversed, prior to new management. And the bugs were found during the immediate testing after deployment and we were able to reverse quickly. [2] The Oligarchic Cell Phone Company.
- rtpg 2y agoI'm shocked that there are places in 2024 that have a separate set of devs writing test code. What decade is this?
- JonChesterfield 2y agoThe (unit) tests developers write themselves check the developer did what they intended to, and then that things don't change. Whatever case analysis the developer missed in the implementation tends to be missing from their tests as well, as they didn't think of it. Testing your own work also does nothing to protect against correctly implementing the wrong thing.
- rtpg 2y agoThis is a fair point. I imagine in a stricter environment you could (for example) have tests before implementations even to avoid huge feedback loops
- adrianN 2y agoHaving a separate test team is pretty much mandatory in safety critical development. You really want people who haven’t written the code go through the specification independently and write test cases for every requirement.
- deredede 2y agoHaving one set of developers write the code and another set of developers write tests from the same specification without reading the code is probably the most effective ways to use tests for high software quality and is typically required by certification bodies e.g. for avionics.
- rtpg 2y agoThat’s a good point for things with certain kinds of specs, but it seems a hell of a lot more straightforward for the person who is writing an implementation to also be in a testing mindset (if only to be writing testable code) Perhaps people in these environments are made of different stuff but touching both sides feels almost essential to me.
- high_na_euv 2y agoIve worked in hw industry and arch team created specs, dev was developing product and its own tests And val team was creating fake product which was used to help to write tests without product (so tests were available faster, so deploy was faster), or just tests - all of this basing on spec It worked well when it comes to bug/issues/inconsistencies catching
- gumby 2y agoFrom my years on GCC I can remember that 90% of regular users' bug reports were user error, not a compiler bug. But as with any code, compilers have bugs too and sometimes they can be quite surprising.
- adgjlsfhk1 2y agoto be fair most of that is GCC/C's fault. C has tons of UB that is very difficult to reason about, and GCC chooses to do weird things by default that are at best ambiguously standard compliant (e.g. evaluating floating point expressions in higher than specified by the user). Compilers of languages with more intuitive semantics tend to get less user error bug reports.
- jcranmer 2y ago> that are at best ambiguously standard compliant (e.g. evaluating floating point expressions in higher than specified by the user). That's not ambiguously standards-compliant, that is literally something that the standard has explicit mechanisms to indicate that's what the compiler is doing (look up FLT_EVAL_METHOD).
- adgjlsfhk1 2y agothe way GCC does it doesn't quote follow the meaning of any of the options of FLY_EVAL_METHOD at least as I read it. it's close to 2 (i.e. all operations and constants evaluate in the range and precision of long double), but it's actual semantics are that some random subset of it's math is done in 80 bit (which isn't the size of is long double), except when that doesn't happen and it used 64 bit instead. that said, I guess you could call this standard compliant, since the standard says that negative options other than -1 are implementation defined behavior, so maybe the GCC default is to set the option to one of the values allowed by, but not documented in, the standard. https://stackoverflow.com/a/20870215/5141328 https://stackoverflow.com/a/20870215/5141328 documents the insanity pretty well
- kragen 2y agoit may be helpful to have the context that gumby's years on gcc were something like 01987–01997, before most of the ub fuckery
- wslh 2y agoWe recently posted about fuzzying a compiler with success [1]. The article contains the details. There is an error on the Zest link that should point to [2]. The key is how to craft the generator. [1] https://www.coinfabrik.com/blog/why-the-fuzz-about-fuzzing-compilers/ https://www.coinfabrik.com/blog/why-the-fuzz-about-fuzzing-c... [2] https://dl.acm.org/doi/10.1145/3293882.3330576 https://dl.acm.org/doi/10.1145/3293882.3330576
- JonChesterfield 2y ago> Rust’s implementation of LibFuzzer, Would that be the LLVM one wrapped in some rust API? From the last month or so accidentally rabbit holed by fuzz testing I am really liking the LLVM one but have a vague sense that I should probably try AFL++ as well. The paper suggests it was a java implementation distinct from those two. The approach of turning the bytestring into a program is good, worth looking at the IR fuzzer in the LLVM suite - this looks like a fair reference for it https://arxiv.org/abs/2402.05256 https://arxiv.org/abs/2402.05256
- pfdietz 2y agoI was disappointed they apparently don't do high volume differential testing with randomly generated programs, a kind of property based testing. Each individual program has very little testing value, but when you can crank out millions or even billions of them they can find all sorts of bugs.
- Haugsevje 2y agoSomehow I came to think of 'Like a rolling stone'..Dylan I believe.
- azakharenko 2y agoIt partially inspired me, right :)
- tgma 2y agoThe article touches on basic testing strategies that apply to general software as well as compilers. I have been involved with compiler testing projects on production LLVM and GCC in my past life. One thing that makes compiler testing specifically more difficult than general software is the Oracle Problem: how do you verify the output is in fact correct? Crashes are relatively easy to find by random fuzzing, but in the general case, proving that the output program is not miscompiled is non-trivial. There are a couple effective techniques in the literature that might be useful here: - Differential testing[1]: generate a bunch of random, correct, deterministic programs; run them under different compilers or under different compilation flags and check if the output of the program is identical - Equivalence Modulo Inputs[2]: a class of techniques that can be used transform a program to a distinct program that is supposed to be equivalent to the original for a specific input. (shameless plug) [1]: https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf [2]: https://web.cs.ucdavis.edu/~su/publications/emi.pdf https://web.cs.ucdavis.edu/~su/publications/emi.pdf
- sojuz151 2y agoWhat about using multiple open source projects with already existing tests? You check if results of execution are the same between compiler versions? This should also provide you with a better coverage of various features
- tgma 2y agoSure that counts as differential testing. The issue is there are limited number of them compared to the amount of code you can generate mechanically, and especially in the case of C, it is not straightforward to write a standardized script to build and run a bunch of random projects.
- danmur 2y agoProbably better than paying money to test an IDE :P
- rbanffy 2y agoIt seems difficult to test. Being difficult to test is usually a sign of inconvenient APIs and modules. For instance, the hiding of elements happens in multiple levels - the front-end needs a test to make sure the right intermediate representation is being generated. The backend, in turn, needs to be tested to make sure the intermediate representation with a hidden element generates the proper linker information and the linker itself needs a test to make sure the correctly annotated object code results in the correct artefact lacking external information about the hidden symbol. Testing end-to-end seems very laborious and error prone. Root-cause-analysis should have pinpointed the place the issue originates and one or more possible paths where it propagates downstream to inform future tests. It also disturbs me that the author mentioned the order at which sources are compiled to matter in the final result. It should never matter. When we build software we should always make it in such way it’s trivial to write tests for it. If writing a test is easy, it indicates using the tool you wrote is also easy.
- alexvitkov 2y agoCompilers are one of the easiest and most fun pieces of software to test, because you can test very specific behavior without touching the internals at all. E.g. if I want to test that `*` has higher precedence than `+`, I would write something like this: assert_ast_equals(parse_expr("(a*b)+c"), parse_expr("a*b+c")) assert_ast_equals(parse_expr("a+(b*c)"), parse_expr("a+b*c")) You can rewrite the whole compiler if you want, but as long as you have some notion of a "parser", an "AST" and "two AST nodes being the same" this test will keep working. This is much more powerful than going into the parser internals and comparing get_operator_precedence('+') with get_operator_precedence('*') which is the default thing you would do if you're told to test every function after writing it.
- actionfromafar 2y agoJust be careful to not return null from all calls to parse_expr() :)
- 01HNNWZ0MV43FF 2y agoThat's what mutation tests are for :)
- MaxBarraclough 2y agoWell, low skill floor, sure, but there's a very high skill ceiling. John Regehr and his students have done impressively deep work in finding compiler bugs. • https://blog.regehr.org/archives/category/compilers https://blog.regehr.org/archives/category/compilers • https://blog.regehr.org/ https://blog.regehr.org/ (some of his compiler-related posts aren't properly tagged) • [PDF] https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf Finding and Understanding Bugs in C Compilers (edit I see someone else already posted this paper) • A (very) little discussion: https://news.ycombinator.com/item?id=7728035 https://news.ycombinator.com/item?id=7728035
- YorickPeterse 2y agoFor parsing this might be the case, but writing tests becomes a lot more tricky the further down the compilation pipeline you go, as the amount of boilerplate code needed to set things up increases.
- _0j9w 2y agonahh I don't buy it. You don't have friends OFTEN asking you how it feels to test a compiler bro stop the cap
- nj5rq 2y agoI am making a simple Lisp interpreter, and this is my whole testing stage: $ valgrind --leak-check=full --track-origins=yes ./lisp < test/test.lisp $ cat test/expected.txt What can I say, works for me.
- Neywiny 2y agoI did this for a class on compilers and interpreters where we wrote our own, each week expanding on functionality but keeping the 2 in feature parity. I wrote Python to auto generate test cases. I vaguely recall that the test was if the two operated the same (given the simplicity of the interpreter I viewed it as a "gold standard"). See [1] for his it worked. Among the bugs in my code I found 2 things through that exercise: 1. Functional programmers often write slow code. It turned out that my compiler was spending most of its time in my professor's code that while I'm sure was very mathematically pure, was a large consumer of immutable, short lifetime objects. Meaning under the hood mallocs. I should've valgrinded it but I'm certain it would've overflowed the counters (jokes) 2. If a comment spanned multiple lines in the resulting assembly, I could escape the comment and operate outside the bounds the professor setup, letting me use more assembly directives to solve the problems way easier. Ultimately we worked to fix that because usually it just means the student will try to compile part of a comment as assembly and that can be very confusing for the less assembler-error inclined. I used it for having a constant before a variable for type tagging. A 1 line solution. I believe the class's preferred way was putting the tag in a register and yadda yadda something that took a lot more finagling and effort. I did that maybe once before using my knowledge of the comment escape to do the arbitrary code injection. [1] interpreted was written in the language we were interpreting, so as long as there were no typos or logic errors, the functionally was perfect vs running the code in the programming language. The compiler would return back a series of objects that wrapped assembly. For example, Add(R2, R2, R3). Usually pretty transparent. The framework we were given would then write out the .s file, I believe it would call some gcc or other thing, and we'd run the binary to make sure it worked.
- fidotron 2y agoContrary to the Jetbrains praise in here, I despise Kotlin and the evidence is the tailrec keyword. “tailrec” is what you mention in function definitions to indicate that function is tail recursive, but it only actually worked when you called the same function itself in the return, and not any other tailrec function. The part of this which was idiocy was this would only manifest as a StackOverflowException at run time. (I found this as my language evaluation involved implementing a state machine idiomatically). If you are going to make tailrec only work as a while loop then have the compiler alert the programmer at build time, but this got all through their design and QA. Not exactly a well thought out process. They have probably fixed this now, but instead I went off to golang, where the features are few but when they exist they are done properly.
- kragen 2y agothe jvm is to blame here, not kotlin. clojure has the same problem
- fidotron 2y agoThe point is the kotlin compiler not handling tail recursive calls of other functions. If it cannot handle it that is a build time error, not a runtime one. It shouldn’t even get to the jvm. Clojure is way more dynamic so such errors would be expected. The whole thing is attempting to hack together something that superficially competes with c# but just doesn’t have the substance.
- kragen 2y ago`tailrec` in kotlin is a modifier to the function, not to the function call site. let's stipulate that that's a sensible decision (i don't think it is, but the rest of the thread is moot otherwise). are you suggesting that a `tailrec` function shouldn't be allowed to call any function other than in tail position? i think that would make it so restrictive as to be useless for most tail-recursive loops the clojure approach (as i understand it; i've only written a few dozen lines of clojure) is to use an explicit `recur` operator, at the call site, for a tail-recursive call. implicitly that operator invokes the same function, not a different one, because that's the best you can do on the jvm. it's not a runtime error
- deleted 2y ago[deleted]