6 ms·

> Even if -Og was ubiquitous, it is still suboptimal to -O0: it can still inline code a bit too aggressively for an effective debugging session. IMO, the probl

> Even if -Og was ubiquitous, it is still suboptimal to -O0: it can still inline code a bit too aggressively for an effective debugging session. IMO, the problem is not so much the inlining, but the sad state of the optimization passes losing track of many things and producing useless debug info.

DannyBee 4y ago
Eons ago, i wrote a lot of the first version of debug location tracking/location list tracking in GCC, and the support for evaluating them in GDB (plenty of others came along and improved it nearly instantly - it thankfully did not last long alone ;P). I have spent, over the years, an amazing amount of time on trying to improve optimized-code debug info. You could and should improve the optimization passes, but at various points in life they were actually very good. It is always a dance between advancing optimization and debug info, and i certainly will not claim that debug info is thought about as much as it should be - it is often fixed later, as i'm sure you expect. But at the same time, what the author amounts to ends up reducing to the following: 1. The output executable is pretty well optimized 2. The output debug info is a recreation of the program as-written that can be executed by a debugger 3. The mapping between #2 and #1 has no holes. IE #2 is basically "you've output the entire program using dwarf5 as a target processor". This is never going to be a thing that regular users strive for, for lots of reasons. So if the game companies or whoever want this to be the case, they will actually have to contribute more than just bug reports - because it's not something that drives compiler usage for any language, whether it's C++ or anything else (IE it's hubris to believe that other languages won't end up the same way as they get better at optimizing. C++ just has a long head start) At the same time I am sure that just one good compiler person paid for a year working on any of the compilers to improve optimized debug info would get to a really good place. I know - i was once paid to do it, and when i stopped, it was, back then, in a pretty darn good place compared to where we started[1]. Rather than throwing away every abstraction, it's hard to believe this entire industry can't afford this. Yet I have not seen them try before (perhaps they did after my time, but my time covered 20 years, and it didn't really happen during then) So it is 100% reasonable to blame game developers or whoever as a group - because as a group, they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. [1] I'll note, in practice, the most effective route over time is likely actually to just store enough info to do deoptimize during debugging as-needed, which is the path various VM based languages take and works well. The traditional path to doing this is to JIT the language along the way (which is totally possible for C++), but it's not strictly necessary to do it that way. The harder part is the debugger/compiler cooperation, not the runtime mechanism.
- khuey 4y ago
  Thanks for doing that work. Even many years later gcc is in a noticeably better place than clang when it comes to how much debug info is preserved by the optimizer. https://robert.ocallahan.org/2018/11/comparing-quality-of-debug-information.html https://robert.ocallahan.org/2018/11/comparing-quality-of-de...
- dannas 4y ago
  [gcc/gdb/LLVM outsider here] If someone is interested in what's being done to improve LLVMs debuginfo, here's a talk by Djordje Todorovic from two years ago: https://www.youtube.com/watch?v=GpMLt1oecOk https://www.youtube.com/watch?v=GpMLt1oecOk. It's about tracking dwarf locations for variables by keeping around information from the parent frame. According to that presentation, at least that work has been progressing.
  - dannas 4y ago
    As for contributions from the gamedev industry, here's a EuroLLVM 2022 presentation from Orlando Cazalet-Hyams working at Sony: https://www.snsystems.com/technology/tech-blog/improving-debug-locations-for-variables-in-memory https://www.snsystems.com/technology/tech-blog/improving-deb...
- tialaramex 4y ago
  > they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. There are people from that industry or adjacent industries writing other languages. Most obviously Jai (Jonathan Blow's language, Jonathan did "Braid" and "The Witness") but you could also argue for Ginger Bill's Odin. I suppose in this context it will be interesting to see whether the debug story is any better in these languages (neither is yet finished).
- jmorse2 4y ago
  > they aren't (again, AFAIK) doing anything about it Person slightly doing things to LLVM checking in -- the three points you list are compelling, because it's always going to be difficult to describe one program (source) in terms of a vastly different (optimised) program. Do you think there's mileage in showing the developer a partially optimised program [0] instead of trying to perfectly describe the source? It'd be easier to describe in debug-info, and possibly easier for developers to identify unexpected changes to the code they wrote. [0] "Non-Transparent Debugging of Optimized Code", 1999
usefulcat 4y ago
It's not just inlining. With -Og (clang or gcc), the debugger frequently can't even tell me the value of function arguments because they've been "optimized out".
- adrian_b 4y ago
  While I agree with you and with the OP and I have also been sometimes annoyed by this kind of debugger messages, the more universal solution than wanting an improved debugger is to rely for debugging on some form of progress and error logging in your program, which will work at any optimization level. Adding such debugging aids in your program may need some extra work, but it also functions in all cases when using the debugger is hopeless, e.g. for concurrent programs or other programs that depend on real-time interactions which prevent the use of breakpoints.
  - usefulcat 4y ago
    Yes and no. I use logging too, but that's only useful when I know (or can guess) ahead of time what locations and what things at each location I'll need to know in order to diagnose the problem. Of course I can always go back and add some logging, but it's not a given that I'll want to keep it around after the problem is fixed. For example if it's in a performance-critical section of code, even the cost of checking to see whether logging is enabled may be too great. And in any case every log statement does add visual noise to the code.
- SuperV1234 4y ago
  Just want to remind you that `-Og` in Clang is not representative here, as it's exactly the same as `-O1` -- they implemented support for the flag just to ease transitions between GCC and Clang without having to filter out flags, but it doesn't provide any debugging-specific advantage.
  - not2b 4y ago
    Yes, those who think -Og is no good because they are clang users should perhaps file bug reports or contribute improvements instead.

DannyBee 4y ago

Eons ago, i wrote a lot of the first version of debug location tracking/location list tracking in GCC, and the support for evaluating them in GDB (plenty of others came along and improved it nearly instantly - it thankfully did not last long alone ;P). I have spent, over the years, an amazing amount of time on trying to improve optimized-code debug info. You could and should improve the optimization passes, but at various points in life they were actually very good. It is always a dance between advancing optimization and debug info, and i certainly will not claim that debug info is thought about as much as it should be - it is often fixed later, as i'm sure you expect. But at the same time, what the author amounts to ends up reducing to the following: 1. The output executable is pretty well optimized 2. The output debug info is a recreation of the program as-written that can be executed by a debugger 3. The mapping between #2 and #1 has no holes. IE #2 is basically "you've output the entire program using dwarf5 as a target processor". This is never going to be a thing that regular users strive for, for lots of reasons. So if the game companies or whoever want this to be the case, they will actually have to contribute more than just bug reports - because it's not something that drives compiler usage for any language, whether it's C++ or anything else (IE it's hubris to believe that other languages won't end up the same way as they get better at optimizing. C++ just has a long head start) At the same time I am sure that just one good compiler person paid for a year working on any of the compilers to improve optimized debug info would get to a really good place. I know - i was once paid to do it, and when i stopped, it was, back then, in a pretty darn good place compared to where we started[1]. Rather than throwing away every abstraction, it's hard to believe this entire industry can't afford this. Yet I have not seen them try before (perhaps they did after my time, but my time covered 20 years, and it didn't really happen during then) So it is 100% reasonable to blame game developers or whoever as a group - because as a group, they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. [1] I'll note, in practice, the most effective route over time is likely actually to just store enough info to do deoptimize during debugging as-needed, which is the path various VM based languages take and works well. The traditional path to doing this is to JIT the language along the way (which is totally possible for C++), but it's not strictly necessary to do it that way. The harder part is the debugger/compiler cooperation, not the runtime mechanism.

khuey 4y ago
Thanks for doing that work. Even many years later gcc is in a noticeably better place than clang when it comes to how much debug info is preserved by the optimizer. https://robert.ocallahan.org/2018/11/comparing-quality-of-debug-information.html https://robert.ocallahan.org/2018/11/comparing-quality-of-de...
dannas 4y ago
[gcc/gdb/LLVM outsider here] If someone is interested in what's being done to improve LLVMs debuginfo, here's a talk by Djordje Todorovic from two years ago: https://www.youtube.com/watch?v=GpMLt1oecOk https://www.youtube.com/watch?v=GpMLt1oecOk. It's about tracking dwarf locations for variables by keeping around information from the parent frame. According to that presentation, at least that work has been progressing.
- dannas 4y ago
  As for contributions from the gamedev industry, here's a EuroLLVM 2022 presentation from Orlando Cazalet-Hyams working at Sony: https://www.snsystems.com/technology/tech-blog/improving-debug-locations-for-variables-in-memory https://www.snsystems.com/technology/tech-blog/improving-deb...
tialaramex 4y ago
> they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them. There are people from that industry or adjacent industries writing other languages. Most obviously Jai (Jonathan Blow's language, Jonathan did "Braid" and "The Witness") but you could also argue for Ginger Bill's Odin. I suppose in this context it will be interesting to see whether the debug story is any better in these languages (neither is yet finished).
jmorse2 4y ago
> they aren't (again, AFAIK) doing anything about it Person slightly doing things to LLVM checking in -- the three points you list are compelling, because it's always going to be difficult to describe one program (source) in terms of a vastly different (optimised) program. Do you think there's mileage in showing the developer a partially optimised program [0] instead of trying to perfectly describe the source? It'd be easier to describe in debug-info, and possibly easier for developers to identify unexpected changes to the code they wrote. [0] "Non-Transparent Debugging of Optimized Code", 1999

usefulcat 4y ago

It's not just inlining. With -Og (clang or gcc), the debugger frequently can't even tell me the value of function arguments because they've been "optimized out".

adrian_b 4y ago
While I agree with you and with the OP and I have also been sometimes annoyed by this kind of debugger messages, the more universal solution than wanting an improved debugger is to rely for debugging on some form of progress and error logging in your program, which will work at any optimization level. Adding such debugging aids in your program may need some extra work, but it also functions in all cases when using the debugger is hopeless, e.g. for concurrent programs or other programs that depend on real-time interactions which prevent the use of breakpoints.
- usefulcat 4y ago
  Yes and no. I use logging too, but that's only useful when I know (or can guess) ahead of time what locations and what things at each location I'll need to know in order to diagnose the problem. Of course I can always go back and add some logging, but it's not a given that I'll want to keep it around after the problem is fixed. For example if it's in a performance-critical section of code, even the cost of checking to see whether logging is enabled may be too great. And in any case every log statement does add visual noise to the code.
SuperV1234 4y ago
Just want to remind you that `-Og` in Clang is not representative here, as it's exactly the same as `-O1` -- they implemented support for the flag just to ease transitions between GCC and Clang without having to filter out flags, but it doesn't provide any debugging-specific advantage.
- not2b 4y ago
  Yes, those who think -Og is no good because they are clang users should perhaps file bug reports or contribute improvements instead.