4 ms·

>Most compiler bugs are not in their interpretation of the specification, but in various optimization passes. I work on a large, open-source, industrial-grade

>Most compiler bugs are not in their interpretation of the specification, but in various optimization passes. I work on a large, open-source, industrial-grade compiler for a living and I disagree with this statement unless it is specifically referring to C compilers and not compilers for other languages.

tluyben2 5y ago
Which compiler do you work on?
- throwaway2037 5y ago
  "large, open-source, industrial-grade compiler". No kidding; I thought the same! How many can there really be?
  - tluyben2 5y ago
    Yeah and apparently a secret?
    - pombalit 5y ago
      I indeed prefer to avoid doxing myself.
- pombalit 5y ago
  I work on a GCC frontend, but not the C one.
pfdietz 5y ago
Or are bugs due to interpretations of the specification just easier to find? The point of compiler fuzzing like Csmith was that optimization bugs are really hard to find by the kinds of testing that would find more typical conformance bugs. Before these test generators were written one might have thought that optimization bugs were rare, simply because they hadn't been triggered much.
- pombalit 5y ago
  >Or are bugs due to interpretations of the specification just easier to find? It could be. Another possible explanation could be that I am biased due to mostly working on the frontend of the compiler. I don't think that's the case though, most of the tickets that our customers open seem to be related to the frontend doing something that doesn't match the spec.
  - pfdietz 5y ago
    That customers report front end bugs could also reflect how easy they are to trigger. I admit this could mean they are more important, even if they aren't more numerous. The relative frequency of different kinds of bugs can also change over time. Testing has a diminishing returns behavior. If front end bugs are easy to trigger, they will get expunged, and you'll be left with the harder to trigger bugs elsewhere in the compiler. I saw this in writing conformance tests for Common Lisp implementations: after implementations got the little direct tests right (function FOO implements its specified behavior), the remaining defects tended to be deeper and more subtle. Eventually random testing was churning out bizarre compiler interactions (but even that eventually burned through the reachable bugs.) It would be interesting to study various compiler testing approaches using mutation: introduce bugs to the compiler by mutating its code, then see what fraction the testing can detect. Fully mutating a compiler is likely infeasible, but one could create a random sample of mutations and make estimates.
    - moonchild 5y ago
      It's the pareto principle, I think: 80% of code that's out there will exercise 20% of the compiler's codepaths. It gets increasingly difficult to find meaningful code that tests those paths.

tluyben2 5y ago

Which compiler do you work on?

throwaway2037 5y ago
"large, open-source, industrial-grade compiler". No kidding; I thought the same! How many can there really be?
- tluyben2 5y ago
  Yeah and apparently a secret?
  - pombalit 5y ago
    I indeed prefer to avoid doxing myself.
pombalit 5y ago
I work on a GCC frontend, but not the C one.

pfdietz 5y ago

Or are bugs due to interpretations of the specification just easier to find? The point of compiler fuzzing like Csmith was that optimization bugs are really hard to find by the kinds of testing that would find more typical conformance bugs. Before these test generators were written one might have thought that optimization bugs were rare, simply because they hadn't been triggered much.

pombalit 5y ago
>Or are bugs due to interpretations of the specification just easier to find? It could be. Another possible explanation could be that I am biased due to mostly working on the frontend of the compiler. I don't think that's the case though, most of the tickets that our customers open seem to be related to the frontend doing something that doesn't match the spec.
- pfdietz 5y ago
  That customers report front end bugs could also reflect how easy they are to trigger. I admit this could mean they are more important, even if they aren't more numerous. The relative frequency of different kinds of bugs can also change over time. Testing has a diminishing returns behavior. If front end bugs are easy to trigger, they will get expunged, and you'll be left with the harder to trigger bugs elsewhere in the compiler. I saw this in writing conformance tests for Common Lisp implementations: after implementations got the little direct tests right (function FOO implements its specified behavior), the remaining defects tended to be deeper and more subtle. Eventually random testing was churning out bizarre compiler interactions (but even that eventually burned through the reachable bugs.) It would be interesting to study various compiler testing approaches using mutation: introduce bugs to the compiler by mutating its code, then see what fraction the testing can detect. Fully mutating a compiler is likely infeasible, but one could create a random sample of mutations and make estimates.
  - moonchild 5y ago
    It's the pareto principle, I think: 80% of code that's out there will exercise 20% of the compiler's codepaths. It gets increasingly difficult to find meaningful code that tests those paths.