11 ms·
Do Developers Read Compiler Error Messages? [pdf]
- RyJones 5y agoProbably not? For one project I worked on, I had the build process emit a message: “please upgrade to foo bar version 2” One of the top questions on stack overflow was “how do I fix this?” How about… upgrade?
- krisoft 5y agoBuild processes output a lot of useless junk messages all the time. First challenge is to make the relevant line stand out. Our build in a similar situation uses a cowsay ascii art where the cow tells you that your foobar is out of date. Stands out very clearly. Also makes it clear that this is not just some obscure system grumbling but that this is a very intentional message from your colleagues. The other thing “how do i fix this?” might mean is that the person asking doesn’t know how to update their foobar. Our system state the exact command the developer has to run to get their foobar to the correct version. No guesswork needed. Saves a ton of time.
- ithinkso 5y agoWhat I don't understand is why (at least in places I worked at) CI/build systems are configured to print absolutely everything they do, who reads that? Hopefully error msg is at the end so you can go to the end of the 100kB worth of logs to find what failed. If it's no at the end though it's kinda funny how you just mindlessly scroll through a garbage and sort of instinctively identify what 'looks like normal garbage' vs 'looks like a bit more suspicious garbage', just by it's shape
- jagged-chisel 5y agoBecause something will fail later on in the process based on an [unintentional?] change early in the process. Like the App Store helpfully upgraded Xcode over the weekend, and the new version won’t compile the 67th dependency in your project. So you learn to have Xcode dump its version number at the start so it’s in the log. And you tell the junior DevOps engineer about how to prevent automatic upgrades on the Mac mini he never touches. And also how it’d be good to have downloaded a few versions of Xcode and make selecting the one you need a step in the build process… but I digress
- gizmo686 5y agoIf something goes wrong it is very hard to debug unless the relevent information is in the logs. Unfourtuantly, the build system doesn't know what information will end up being relevent, so it just shows everything. Needing to sift through a giant log is annoying, but you can get pretty fast at it for "normal" errors. Needing to debug an issue that is not adequately logged is a nightmare.
- dudeman13 5y agoI agree with this assessment. Sure there are things such as 'too much data' and 'trash data'. The problem is discerning what you need and what you don't need. The answer isn't always obvious, but it is possible and 'easy' to filter data, whereas getting missing data can be impossible/real hard.
- not2b 5y agoThe build system for the product that I work on produces a giant log with all the commands that are run and a separate filtered version that only shows the errors (if any). So for a build breaking change the issue is quickly found in the filtered log. But to answer questions about why the build is taking so long we have the full log.
- amatecha 5y agoDiagnostic details so you can see the innocuous process steps that led to a failure/error :) It's entirely possible that a certain value/step early in the process is what ended up resulting in a breakage way later in the process. At least, in my experience that has been the case for all kinds of software/services.
- deleted 5y ago[deleted]
- jayd16 5y agoThe black box records everything a plane does. Who reads that?
- emsy 5y agoApples and oranges.
- kevincox 5y agoI don't think it is. These are both logging devices that you look at after the "incident". Sure there are differences, the CI job is likely easier and cheaper to reproduce to get more data, but at the end of the day you want to be able to figure out what happened by looking at the logs.
- exdsq 5y agoDifference is one will have multiple trained engineers working through the logs to investigate according to a standardised method, the other is one person skimming to work out what dependency is missing (in a repeatable environment). There must be a better abstraction than here is all the things, have fun.
- alpaca128 5y agoIt's fine to log all the info somewhere, but there's no reason to throw it all into one big chunk and then expect the developer to dig through the data. Especially when it's perfectly possible for the software to produce more structured output in the first place. Looking for a dependency issue shouldn't be remotely comparable to analyzing an airplane's crash logs in difficulty.
- scraptor 5y agoIf a plane crashes you can't retry with higher log level
- PretzelPirate 5y agoThis is the same in case of an official build: if some attacker was able to modify what happens in a build that makes it to production, I need logs to build an understanding and explain to auditor/governments exactly what happened, what the effect was, and how I’ll mitigate it. Since all logs should be immediately persisted off box, they are protected from tampering. Of course there’s no reason the default view of logs in the CI/CD system needs to be so verbose.
- danmur 5y agoI know what you mean, it should store everything but not print it all out. That's kind of on application developers too though, so many programs I've worked on had terrible logging (too much information, the right information not at the right level).
- numpad0 5y agoIt tells what it tried and what went right.
- UweSchmidt 5y agoIdeally debugging any issue should include an improvement of the relevant loging. Then, over time, the system can perform relevant self-checks and cross-references, and spit out a list of potential causes, while retaining detailed logs nested deeper to the right or to the bottom.
- formerly_proven 5y agoLiterally every single sibling comment is missing the point. Yes, this data should be recorded. Yes, it should be viewable. Should it be viewed as a continuous 34000 line log file? Of course not. But this is what many (most?) CI tools do. The correct way is to group output at least by step, make log output from various parts of the system collapsible etc., for example, dumping all environment variables and component versions at every step CAN be helpful to diagnose issue, but usually isn't. So it should be collapsed by default. Edit: I have the exact same gripe with all of the log4j inspired logging libraries, which in my opinion provide very little value over a print. To little structure, no way to hold back output until you have an error, temporary redirects are difficult and have to modify global state and so on. Uninspired design providing minimal utility.
- amelius 5y agoInstead of collapsing, you can grep through them.
- exdsq 5y agoAssuming you know the word you're looking for, at least collapsible errors are easier to explore
- lumost 5y agoBut then I can’t grep/search.
- cortesoft 5y ago“Collapsible” implies using some sort of interface for looking at the logs… that interface can also include grep/search functionality that searches all the lines. It could also include functionality to let you download the entire raw log to grep locally.
- Nextgrid 5y ago> The correct way is to group output at least by step, make log output from various parts of the system collapsible etc., for example, dumping all environment variables and component versions at every step CAN be helpful to diagnose issue, but usually isn't. So it should be collapsed by default. Which means test tools should conform to some agreed-upon universal output format for their console (and make sure none of the underlying libraries print anything to stdout) or have an out-of-band way to communicate with the CI system in real-time (XUnit/Junit isn’t good enough as those are only typically written at the end of the process).
- aspaceman 5y agoWait... This comment isn't serious. I read that. You just tail the file to find the error. But it's very useful to go through it all. It's not garbage to me. I find most logs quite readable.
- ithinkso 5y agoIt was serious. You haven't seen what I've seen. ENV variables at every step? print them, build has a parser/codegen step? print everything it does, sanity check scripts? I always need to know what they did, setup scripts? That's the most important part, every .o file compiled and linked? show me, with the compile command for a good measure so all the -Ilibs's are there? of course, set -x? why not, need to wget/curl something? output goes to the log, is it in parallel? just append as it goes, build failed? start the teardown but make sure you add everything it does to the log And as I wrote, I hope it all just terminates at the first error so I can tail the logs... but it's not always that pretty. I don't believe you've read those instead of tailing for an error or doing the fast scroll to find something looking somewhat off I mean, I'm used to it now, I'll grep, less, tail and I'll find what I need, I'm just baffled that it has so much unnecessary data for every single build. At one company I've got good enough at it that I could press-and-hold pageup from the bottom in less and staring at the streaming bytes Matrix-style looking for a place of interest
- dec0dedab0de 5y agoIt's because you're configuring it using a language you barely know, so when you're troubleshooting you don't know if it's logic or syntax.. and unless you're lucky someone else is actually owning the CI system and you have to worry about their permissions. So you end up debugging everything, and when it works you're afraid to remove the debugging because the last time you did it took hours, and you ended up leaving it in. Edit: also, because all the important messages are being sent to a group chat or other appropriate place.
- gregmac 5y agoWhat often happens is you get a log like (contrived example): Warning: http timeout to package server 1 Warning: could not find x in package server 2 ..(several lines later).. Warning: could not find dependency x in cache .... Compile Error: unknown reference to "x" Because the actual problem by itself isn't fatal -- it doesn't yet know if the package is in server 2, and it might already have a suitable package in cache -- it's not a fatal error. Later, the part of the compiler they needs the dependency doesn't have context to the other failures that happened; all it knows is something is missing. This is a universal problem in almost every application: very rarely can a single log line provide enough information to fix a problem or identify a bug. In my contrived example, the partial fix is that the package bit should fail. However the final log message needs all the context of previous errors as well or it isn't useful either. It's turtles all the way down. It's not that this isn't a solvable problem, it's that people building compilers and other tools make assumptions about context, people putting together the build don't spend the time to learn the nuance of every tool they use (rightfully so - who had time for that?), and everyone falls back on the crutch of giant log files. In other words, through the whole stack (language, compiler, tools, build system, and individual application build itself), no one has optimized or built anything based on the idea the CI server or build system should have a single, useful error message.
- marcosdumay 5y agoThe problem isn't the warnings and non-fatal errors. In fact, the largest problem is that the noise obscures them. The problem is the pages and pages of "did that, everything is fine". Since Linux distros stopped optimizing disk usage by omiting the update time of files a couple of decades ago, I haven't seen build tools lose any single step. The failure is never because something that should be done was ignored. Instead, the failure is often because of some non-fatal error that you have no hope of finding because it's mixed with 10k like of "build tool got here" spam.
- gregmac 5y agoActually I totally agree with that. I was being generous in my example, and often what I showed as "Warning" is just a informational-level message. This leads to much more insidious problems like: ....(thousand of lines)... Info: Package X version 1.2.3 successfully added ....(thousand of lines)... Compile error: Unknown type Z Actual problem: Z was added in Package X version 1.3.0, but some other dependency or reference that wasn't updated caused the incorrect version to be loaded.
- jdlyga 5y agoC++ compilers massively improved their error messages in the last couple decades. If you were missing a semicolon, it used to give you some syntax error when it tried to parse the next statement. Now, it just tells you that you forgot a semicolon. These kinds of user friendly improvements really make compilers easier to work with.
- loloquwowndueo 5y agoIf it can tell you forgot the semicolon it probably can do without it, so why doesn’t it? ;)
- astura 5y agoJavaScript has automatic semicolon insertion and I'm not a fan
- dudeman13 5y agoAh, but are you not a fan because of the automatic semicolon insertion or because... well, JavaScript :P
- junon 5y agoBecause it's a massive headache to debug code that has subtle bugs due to missing semi-colons.
- williamdclt 5y agoNever happened to me TBH. There's very few cases where the semicolon is actually significant, and when it happens my autoformatter produces garbage which makes it very obvious, or it adds the semicolon itself. That doesn't mean it never happens (most of us inherited a shitty codebase lacking things like code standards and autoformatting at some point), but I always find it strange that people keep talking about this thing I've never experienced in years writing JS professionally
- Forge36 5y agoI once wrote a message saying "if you see this call me at <number>" An hour later and 1000 messages shown zero calls. So I updated my test to grab their OS user name and called them. "I thought you were testing something so I didn't call" While they may read the message, it didn't mean the purpose was clear.
- hirsin 5y agoGetting people to believe error messages is surprisingly hard. "The server says I used the wrong certificate - what am I supposed to do here, what went wrong?". 9/10 times they used the wrong cert, and the rest of the time they forgot to register it to the right app. But they don't trust the errors...
- ufmace 5y agoPeople with exposure to tech support often complain that users never read error messages. It's almost understandable for low-skill users. But it's surprising and disturbing how many professional developers also don't read error messages. Error messages can be wrong or misleading sometimes, but how do you not at least think about what they said and if that could actually be what's wrong?
- dhosek 5y agoAnecdotally, based on questions on tex.stackexchange.com, there are a lot of questions that come from people completely ignoring error messages.
- geoduck14 5y agoI would be really interested in seeing data around this! Do you speak from anecdotal experience? Do you have any hard data (that you can share)?
- ectopod 5y ago>> Anecdotally, based on questions on tex.stackexchange.com ... > Do you speak from anecdotal experience? I think we can agree that people don't read things properly.
- dhosek 5y agoNot a chance. It would be a significant effort just to collect the data. I’m not interested enough to do it.
- mirker 5y agoLatex is my personal example of bad compiler messages, as well. The other day I added a duplicate bibtex entry and/or added two keys for the same entry. The error messages are a dump of the entire bibliography rather than a simple line number.
- dhosek 5y agoIn my LaTeX book, I have an appendix dedicated to offering up explanations of what’s really wrong for every error message. It’s kind of fun doing everything wrong n LaTeX to get at all these.
- mjevans 5y agoIt would be useful to be specific about the language involved in the title. Rather than "Do Developers Read Compiler Error Messages? [pdf]" a more descriptive title may be more appropriate E.G. "Study of JAVA errors and resolution effectiveness [pdf]"
- junon 5y agoThere's a rule about not editorializing titles on HN. The title here is the title in the paper, so it's the correct title. This sort of suggestion should be sent to the authors of the paper itself. Then again, dang has also broken this rule and had a justification for it, so who really knows for sure.
- bookofsand 5y agoNo, developers don't read compiler error messages, unless absolutely necessary. The economics doesn't work out: the compiler creates hundreds of error messages in a few milliseconds, whereas developers only have a very limited budget of attention. What developers do is to check where the errors show up, usually just the first error, and use pattern recognition techniques, aka 'experience', to fix the error in its context. Only if the fast track error recovery process fails, repeatedly, do developers spend minutes to carefully read the compiler error messages and deeply make sense of it. This matches the article thesis: the 'better' compiler messages are, the more likely is that pattern recognition techniques can use the error message as part of the fast track error fixing process.
- dathinab 5y ago> No, developers don't read compiler error messages, unless absolutely necessary. This depends fully on the ~language~ compiler. Some compilers create extremely long verbose but useless messages, other create use-full messages most times only as long as necessary with nice formatting making it easy to skip over the parts you don't care about. Like e.g. I always read error messages when programming rust, sure I fix a error at a time so I don't read all error messages at once, but I do go through them step by step.
- citrin_ru 5y agoClang (and modern GCC) warning are quite helpful, but they are ignored because by default they don't break a build (and only a small fraction of C/C++ projects use -Werror). If ignored they may cause problems later when an app will be used. Or may not cause any problems at all - that the case for subpar code in general - it is more likely, but not guaranteed to break.
- PaulDavisThe1st 5y ago-Werror can be hang up development at times, which is why we don't use it with Ardour. Sometimes, it can take a while to understand why some obscure message from the compiler is actually relevant and actionable, and that's not always the main focus while working on features or fixing a specific bug. However, we do have a policy of expecting that all compiler errors (except those in in-tree 3rd party libs) should be fixed. We take a very negative view of any commits that create new compiler warnings. Essentially, we prefer to have a development culture that treats warnings as errors rather than have the compiler enforce warnings as errors. When building Ardour, we this as a minimum: '-Wall', '-Wpointer-arith', '-Wcast-qual', '-Wcast-align', '-Wno-unused-parameter' but prefer to use this: '-Wall', '-Wpointer-arith', '-Wcast-qual', '-Wcast-align', '-Wno-unused-parameter', '-Wcast-align', '-Wextra', '-Wwrite-strings', '-Wunsafe-loop-optimizations', '-Wlogical-op', '-Wnon-virtual-dtor', '-Woverloaded-virtual', '-fstrict-overflow'
- throwawayboise 5y agoNo, they don't read them. Until they need to. Then they are valuable.
- dhosek 5y agoI will say that Rust error messages are absolutely wonderful. They generally accurately diagnose the problem and provide a suggestion for fixing it which is usually correct.
- unnouinceput 5y agoIf you are amazed by Rust messages then you'll experience out of body experience reading Delphi's error messages. Better pin point error messages I've never seen in any other IDE.
- estebank 5y agoGenuinely curious, where do you find Rust's messages lacking compared to Delphi's?
- ThePadawan 5y agoDo you still find that holds for somewhat ideomatic rust? I have made the same experience running tutorials and simple example code, but as soon as I took on anything more intricate, I got thrown right back into "GCC compiling a C++ template" territory. Basically, the compiler told me "constraint Foo<<<<trait::?>'<mut&>><other> does not meet bounds Bar<trait::dist>?<other>" and that was it.
- estebank 5y agoErrors involving multiple levels of trait bounds haven't historically been great, but we've made strides on them and there's still some ongoing work to make them better. Feel free to file a ticket next time you come across a case like these! For complex or uncommon cases we rely on people's reports to handle them better.
- ThePadawan 5y agoThanks for the info! I have to admit I was pretty discouraged and rage-deleted the code that made me give up, so I don't have it at hand any more. I also am really early in the process in learning about Rust, and dove in very deep by trying to use warp to write a web application. I'm usually the type to dive in by copy-pasting examples and modifying them to my needs, and I just hit a brick wall of compile errors here that discouraged me from continuing. When I gather the patience to continue, I will keep in mind what you said.
- spullara 5y agoWhen helping debug issues folks are having I have to non-ironically ask them if they read the error message. Often they haven't and it says what the issue is and sometimes how to fix it.
- not2b 5y agoYes, it is embarrassingly common that someone tells me something won't build, and I ask them what the error message was. They didn't know because they didn't read it, and reading the error message made the issue blindingly obvious.
- thewebcount 5y agoI think this is asking the wrong question. The first question should be, “Can developers understand compiler error messages?” I’ve had C++ errors where templates were involved and I didn’t even know if it was in my code because the template types in the error message were so long, I didn’t see any recognizable symbols in it. You think you’re declaring a std::foo<bar>, but the error message is talking about red-black trees and deques because those pieces happen to be used internally in either foo or bar, but you didn’t reference either of them directly. It’s maddening.
- fridif 5y agoNo because I don't have any. Thanks Java
- infogulch 5y agoNo and it's a tragedy. 95% of the time when I'm asked for help debugging a 'problem' I feel like my title should be professional out-loud error message reader.
- phist_mcgee 5y agoHow about not polluting build logs with stuff like this from the CoreJS developer: https://github.com/zloirock/core-js/issues/708 https://github.com/zloirock/core-js/issues/708 I'll read the compiler errors when I don't have to scroll past solicitations for employment.
- junon 5y agozloirock is probably my least favorite person on Github. I'm surprised he's not been banned yet for some of the things he's said to people. https://github.com/zloirock/core-js/issues/936#issuecomment-841085374 https://github.com/zloirock/core-js/issues/936#issuecomment-...
- imnotlost 5y agoNo academics or research papers needed... Double-click on the error-message, look at the code, fix the code, recompile. Can't understand what's wrong with the code? read the error message for clues. Maybe SDK or language docs, maybe google to figure it out.
- aaron695 5y agoThis needs either external pressure on the people writing them or an independent solution. Reality is it's only independent solution. I'd suggest a hash or a heuristic hash or a 3rd party algorithm run on the error linked to a crowed sourced location the user can chose. So the 3rd party can say ignore this it's stupid. Something the original error message writer could never tell their boss. Or when the error was written their info was wrong, now it's this. Or they can say use root permissions on the code. Or undo this security patch. And just like Stack, people could explain why and why not. Teach around the error. Their conclusion "that the difficulty of reading error messages is comparable to reading source code" can't be solved at the error message writers level. Politically or even for the ultra high users who need the hard core message. Even by allowing the user to rule out the easy solutions it might give, they can turn their brain back to source code mode to tackle it.
- whateveracct 5y agoHuge compiler errors activate our lizard brains. Overcome that and all of a sudden a big compiler message isn't any different than parsing a JSON blob or a chunk of code.
- deleted 5y ago[deleted]
- estebank 5y agoThere are multiple reasons that people might not pay as much attention to error messages: 1) there are too many irrelevant, knock-down errors caused by a single source 2) the errors are not clear in their description and attribution 3) the quality of errors varies a lot, meaning that ahead of time you don't know whether reading the error will be helpful or not, which leads to 4) you've been trained by experience to ignore errors in all tools These are a problem even if you do provide good diagnostics because people will not expect them to be of any use.
- smackeyacky 5y agoA while ago I sat down and fixed all the warnings coming out of a C embedded project. It was a real eye opener and not only did the code get more readable it also helped me clean up a few edge case errors I had missed. I have been programming professionally for over thirty years and most mentors used to shrug at warnings if the code worked. It inspired me to go back and clean up my C# and Java stuff and as a learning process I highly recommend it. It seems especially valuable in fast changing environments, generally you will upgrade a tool and get new warnings. I used to think they were just a pain but now I treat them like a way of gaining new insights.
- throw_m239339 5y agoTangential to that discussion: Please, compiler engineers, ALLOW the developer to output stack traces in a structured format. It doesn't matter if it's XML, JSON, YAML, whatever. Please allow the developer to configure the compiler so that when a compiled program yields an error, the error isn't some series of random tabbed lines, but something structured that can then be easily processed.
- jimktrains2 5y agoIt's that what the language server protocol is aiming to do?
- W-_-D 5y agoNot for standalone CLI utils - the LSP is more about powering IDE clients. You could author a CLI that invokes LSP under the hood, but there's no need to tie the output format to that dictated by the LSP.
- bluejekyll 5y agoI’m guessing most new languages are moving in this direction? rustc for example allows for different formatters, here’s the setting for enabling Json output: https://doc.rust-lang.org/rustc/json.html https://doc.rust-lang.org/rustc/json.html
- random_upvoter 5y agoAs someone who at one time wrote a parser for the error output of gcc, clang AND cl, I can only upvote this tenfold. It was a minor nightmare with a ton of special cases, and the final result was a heuristic rather than a formally sound method. Even determining which lines of the output belonged to the same error message was a challenge. Suffice to say that no fun was had.
- ghoward 5y agoThank you for this. I am writing a compiler, and I have never thought about this before. I'll make sure my compiler can do this. Any format you have a preference for?
- kesor 5y agono.
- nils-m-holm 5y agoI actually prefer PCC to GCC, because in most cases it just says, "line xxx, syntax error" and lets me figure out the rest. Reading GCC's or CLANG's multi-line error messages is a total waste of time (to me).
- mirker 5y agoI’d be interesting to study how people read Latex compiler errors, which I find to be unhelpful and an extreme case of bad error messages. I think part of the problem is an extensive use of macros, which expand out into gibberish with little line number information. I personally just bisect my commits rather than read the logs.
- foxyv 5y agoA big part of my mental checklist before committing code for a release is "Are all the warnings taken care of." If the build isn't clean then you are setting yourself up for failure. It may work for now, and maybe 99% of the messages are benign. But that 1% will bite you or your successors in the butt down the line.