6 ms·

How Should Compilers Explain Problems to Developers? (2018) [pdf]

52 points by zdw 4y ago

1-more 4y ago
I like what I see in the comparison table between Jikes and OpenJDK. Elm does a lot of this in its error messages, including hinting at possible typos in field names. What this lacks is a suggested next action for the developer.
nuancebydefault 4y ago
Today I was again amazed by the simplistic-ness of the description of a linking failure by a c++ compiler/linker. It was complaining about an undefined reference without stating where said reference resided. Compiler makers must hate us developers or think we are intelligent enough to have enough with just half a word.
- mhh__ 4y ago
  In general, the linker is written by people different to the people who write the compiler. I don't hate you I just don't control the whole stack. Also linkers can tell you where the symbol was referenced if that place has a location in the code. This isn't universal but it's practically always been there for me.
  - nuancebydefault 4y ago
    Thanks for not hating me :p What i also find hard to understand is that generally linkers are "single pass" i.e. you need to feed them objects and libs in the correct order (callers prior to callees) for them to solve all links. Not sure if this requirement still is always there.
    - JonChesterfield 4y ago
      That requirement vhas been gone for decades if it ever existed, though static library semantics look a bit like that if you don't pass extra flags around them. Linkers are multiple pass if doing compiler optimisation things. Otherwise they're pretty close to cat + patch some pieces, single pass seems fair. It's not immediately obvious to me why linkers don't say which functions were making the undefined symbol call. A suspicion is that the linker may not know what the function name is for the corresponding instruction without digging it out of the debug info and the linker probably doesn't have the corresponding parser ready to hand. It probably could narrow it down to a source object file, though libraries make that difficult.
    - WalterBright 4y ago
      I've written linkers. This one pass thing was gone before the 1980s.
      - nuancebydefault 4y ago
        I've seen it being a problem maybe 10 years ago but i'm happy if it is fully a thing of the past.
- tester756 4y ago
  >Compiler makers must hate us developers I think that C or C++ were just written by people who never gave a single damn about UX until competition appeared and then things started improving,
  - marginalia_nu 4y ago
    The abysmal compile time for these languages may also have encouraged this fairly bare-bones UX philosophy.
    - tester756 4y ago
      >fairly bare-bones UX philosophy. What do you mean?
  - marcosdumay 4y ago
    C was created by people that didn't have any extra resource to spend on UX. Just another day there was an article here about emulating the computer C was written for in a RP2040 (the Raspbery Pi "competitor" for the Arduino), and yes, it has plenty of space for that. C++ had more of a choice about it.
    - tester756 4y ago
      >C was created by people that didn't have any extra resource to spend on UX How many years have passed since that time?
      - marcosdumay 4y ago
        Oh, enough to improve things several times (to answer, 5 decades). People did some improvements, but clearly didn't keep-up with the times (and degraded some things too, UB being the elephant in the room). But large things almost never keep-up with the times.
        nuancebydefault 4y ago
DesiLurker 4y ago
explaining the compiler error messages should really be the first application of LLMs like GPT-4. in fact you can take all the knowledge crap out and just train it on programming info from gcc source code/tests & StackOverflow/GitHub. thereafter have it explain what went wrong. This would be especially useful if you are dealing with template instances and the user explains in a few sentances what they are trying to achieve. I have multi-decade experience in coding C/C++ but its shameful how much time still gets wasted on these stupid compiler errors.
- mhh__ 4y ago
  Or just fix the error message and add a hyperlink to an explanation... You don't need an AI you just need the right abstractions in the code. The AI could help you learn if you really don't understand but for just telling you what went wrong you don't need it. In fact the compiler can already fix a lot of these things for you if they bothered to implement it (see fixit in clang)
  - chrsig 4y ago
    Having a link to a detailed explanation would be amazing. Trying to even find the error message in a sea of c++ template notes is challenging...Then trying to actually parse it while examining the types is...painful.
  - remkop22 4y ago
    Exactly. Compiler developers wrote the logic for the errors you see, they (mostly) have the knowledge to explain the error, why do we need a language model trained on all this derived information, when it could be provided upfront together with context from your own source code.
effnorwood 4y ago
[dead]
ChancyChance 4y ago
I think RQ3 is interesting. It is the only proposition in the paper that references crowd-sourcing. I find that interesting because that immediately lends itself to ... god i hate myself ... ChatGPT. We've been asking GPT to write code, but has anyone asked it to debug code?
jimwhite42 4y ago
I thought this paper was very interesting on the topic: "Mind Your Language: On Novices' Interactions with Error Messages". One of the things that comes up, is that if the error message suggests a change, one of the risks is that such an edit can make the error go away, but the code will not be correct. By making a concrete suggestion, it tempts the programmer to just apply it without thinking. This is really interesting to me because it makes sense, and goes against many common ideas about how to make good error messages.
- hyperrail 4y ago
  > if the error message suggests a change, one of the risks is that such an edit can make the error go away, but the code will not be correct. [...] "Error messages: diagnostic is preferable to prescriptive" [1] is a riff on a similar theme, though the author of that blog post is focusing on error messages for Microsoft's C# compiler, which are more likely to be intended for experienced professional developers, so he is coming at the problem from a different perspective than the student-focused one of the paper you mentioned. [1] https://web.archive.org/web/20060720004516/http://blogs.msdn.com/ericlippert/archive/2006/07/07/659259.aspx https://web.archive.org/web/20060720004516/http://blogs.msdn...
TRiG_Ireland 4y ago
Languages do vary. PHP has some oddities, but its error messages mostly make good sense. It's much easier to debug than, for example, javascript.
- kccqzy 4y ago
  I suppose the oddities you mention include T_PAAMAYIM_NEKUDOTAYIM
mpd 4y ago
Having good error messages is one of the important priorities for me in my compiler, so I made the commitment early and am using codespan_reporting[0] to report the errors, which is going well so far. 0: https://github.com/brendanzab/codespan https://github.com/brendanzab/codespan
WalterBright 4y ago
The most useful advance in error messages is that advent of spell checkers for undefined symbols. The one I wrote used the existing symbol table as the dictionary. It works so well (and is so simple) it evokes the "why didn't I do that decades ago!" thought.
- JohnFen 4y ago
  > it evokes the "why didn't I do that decades ago!" thought. In my experience, that thought is a bright neon sign telling you that you have a potentially killer product on your hands.
DylanSp 4y ago
Interesting that this came up; I was reading https://arxiv.org/abs/2301.02308 https://arxiv.org/abs/2301.02308, on Rust's usability, earlier today. One of the interesting conclusions from that paper was that while Rust's error messages are generally high-quality, they can sometimes be too local, especially for helping get new Rust programmers used to the ownership model. Key quote: "rustc error messages not only describe the error in the code, but for certain error patterns, suggest edits that may fix the problem. However, these edits are always local and don’t provide any high-level design feedback which may be helpful in making the mindshift."
- Nathanba 4y ago
  I think Rust needs to show me a decision tree for why something on the lefthand side of a function call ends up being whatever vscode + lsp shows me. Because the compiler and rust-analyzer are going through various traits and auto-traits to ultimately land on one decision but if I want to understand how Rust works I need to know how it arrives at these conclusions. I don't want to have to jump deep down into language constructs and read the docs just to find out several levels deep that a trait was implemented by some other trait that then had an auto trait impl and that's why it ends up requiring me to write &. Rust being complex is fine but only if they go far beyond what has been standard tooling.

52 points by zdw 4y ago

I like what I see in the comparison table between Jikes and OpenJDK. Elm does a lot of this in its error messages, including hinting at possible typos in field names. What this lacks is a suggested next action for the developer.

nuancebydefault 4y ago

Today I was again amazed by the simplistic-ness of the description of a linking failure by a c++ compiler/linker. It was complaining about an undefined reference without stating where said reference resided. Compiler makers must hate us developers or think we are intelligent enough to have enough with just half a word.

mhh__ 4y ago
In general, the linker is written by people different to the people who write the compiler. I don't hate you I just don't control the whole stack. Also linkers can tell you where the symbol was referenced if that place has a location in the code. This isn't universal but it's practically always been there for me.
- nuancebydefault 4y ago
  Thanks for not hating me :p What i also find hard to understand is that generally linkers are "single pass" i.e. you need to feed them objects and libs in the correct order (callers prior to callees) for them to solve all links. Not sure if this requirement still is always there.
  - JonChesterfield 4y ago
    That requirement vhas been gone for decades if it ever existed, though static library semantics look a bit like that if you don't pass extra flags around them. Linkers are multiple pass if doing compiler optimisation things. Otherwise they're pretty close to cat + patch some pieces, single pass seems fair. It's not immediately obvious to me why linkers don't say which functions were making the undefined symbol call. A suspicion is that the linker may not know what the function name is for the corresponding instruction without digging it out of the debug info and the linker probably doesn't have the corresponding parser ready to hand. It probably could narrow it down to a source object file, though libraries make that difficult.
  - WalterBright 4y ago
    I've written linkers. This one pass thing was gone before the 1980s.
    - nuancebydefault 4y ago
      I've seen it being a problem maybe 10 years ago but i'm happy if it is fully a thing of the past.
tester756 4y ago
>Compiler makers must hate us developers I think that C or C++ were just written by people who never gave a single damn about UX until competition appeared and then things started improving,
- marginalia_nu 4y ago
  The abysmal compile time for these languages may also have encouraged this fairly bare-bones UX philosophy.
  - tester756 4y ago
    >fairly bare-bones UX philosophy. What do you mean?
- marcosdumay 4y ago
  C was created by people that didn't have any extra resource to spend on UX. Just another day there was an article here about emulating the computer C was written for in a RP2040 (the Raspbery Pi "competitor" for the Arduino), and yes, it has plenty of space for that. C++ had more of a choice about it.
  - tester756 4y ago
    >C was created by people that didn't have any extra resource to spend on UX How many years have passed since that time?
    - marcosdumay 4y ago
      Oh, enough to improve things several times (to answer, 5 decades). People did some improvements, but clearly didn't keep-up with the times (and degraded some things too, UB being the elephant in the room). But large things almost never keep-up with the times.
      - nuancebydefault 4y ago

DesiLurker 4y ago

explaining the compiler error messages should really be the first application of LLMs like GPT-4. in fact you can take all the knowledge crap out and just train it on programming info from gcc source code/tests & StackOverflow/GitHub. thereafter have it explain what went wrong. This would be especially useful if you are dealing with template instances and the user explains in a few sentances what they are trying to achieve. I have multi-decade experience in coding C/C++ but its shameful how much time still gets wasted on these stupid compiler errors.

mhh__ 4y ago
Or just fix the error message and add a hyperlink to an explanation... You don't need an AI you just need the right abstractions in the code. The AI could help you learn if you really don't understand but for just telling you what went wrong you don't need it. In fact the compiler can already fix a lot of these things for you if they bothered to implement it (see fixit in clang)
- chrsig 4y ago
  Having a link to a detailed explanation would be amazing. Trying to even find the error message in a sea of c++ template notes is challenging...Then trying to actually parse it while examining the types is...painful.
- remkop22 4y ago
  Exactly. Compiler developers wrote the logic for the errors you see, they (mostly) have the knowledge to explain the error, why do we need a language model trained on all this derived information, when it could be provided upfront together with context from your own source code.

effnorwood 4y ago

[dead]

ChancyChance 4y ago

I think RQ3 is interesting. It is the only proposition in the paper that references crowd-sourcing. I find that interesting because that immediately lends itself to ... god i hate myself ... ChatGPT. We've been asking GPT to write code, but has anyone asked it to debug code?

jimwhite42 4y ago

I thought this paper was very interesting on the topic: "Mind Your Language: On Novices' Interactions with Error Messages". One of the things that comes up, is that if the error message suggests a change, one of the risks is that such an edit can make the error go away, but the code will not be correct. By making a concrete suggestion, it tempts the programmer to just apply it without thinking. This is really interesting to me because it makes sense, and goes against many common ideas about how to make good error messages.

hyperrail 4y ago
> if the error message suggests a change, one of the risks is that such an edit can make the error go away, but the code will not be correct. [...] "Error messages: diagnostic is preferable to prescriptive" [1] is a riff on a similar theme, though the author of that blog post is focusing on error messages for Microsoft's C# compiler, which are more likely to be intended for experienced professional developers, so he is coming at the problem from a different perspective than the student-focused one of the paper you mentioned. [1] https://web.archive.org/web/20060720004516/http://blogs.msdn.com/ericlippert/archive/2006/07/07/659259.aspx https://web.archive.org/web/20060720004516/http://blogs.msdn...

TRiG_Ireland 4y ago

Languages do vary. PHP has some oddities, but its error messages mostly make good sense. It's much easier to debug than, for example, javascript.

kccqzy 4y ago
I suppose the oddities you mention include T_PAAMAYIM_NEKUDOTAYIM

mpd 4y ago

Having good error messages is one of the important priorities for me in my compiler, so I made the commitment early and am using codespan_reporting[0] to report the errors, which is going well so far. 0: https://github.com/brendanzab/codespan https://github.com/brendanzab/codespan

WalterBright 4y ago

The most useful advance in error messages is that advent of spell checkers for undefined symbols. The one I wrote used the existing symbol table as the dictionary. It works so well (and is so simple) it evokes the "why didn't I do that decades ago!" thought.

JohnFen 4y ago
> it evokes the "why didn't I do that decades ago!" thought. In my experience, that thought is a bright neon sign telling you that you have a potentially killer product on your hands.

DylanSp 4y ago

Interesting that this came up; I was reading https://arxiv.org/abs/2301.02308 https://arxiv.org/abs/2301.02308, on Rust's usability, earlier today. One of the interesting conclusions from that paper was that while Rust's error messages are generally high-quality, they can sometimes be too local, especially for helping get new Rust programmers used to the ownership model. Key quote: "rustc error messages not only describe the error in the code, but for certain error patterns, suggest edits that may fix the problem. However, these edits are always local and don’t provide any high-level design feedback which may be helpful in making the mindshift."

Nathanba 4y ago
I think Rust needs to show me a decision tree for why something on the lefthand side of a function call ends up being whatever vscode + lsp shows me. Because the compiler and rust-analyzer are going through various traits and auto-traits to ultimately land on one decision but if I want to understand how Rust works I need to know how it arrives at these conclusions. I don't want to have to jump deep down into language constructs and read the docs just to find out several levels deep that a trait was implemented by some other trait that then had an auto trait impl and that's why it ends up requiring me to write &. Rust being complex is fine but only if they go far beyond what has been standard tooling.