4 ms·

> It is possible in theory that code that's less carefully optimized exhibits different behavior, or that the benchmarks chosen here are simply not as amenable

> It is possible in theory that code that's less carefully optimized exhibits different behavior, or that the benchmarks chosen here are simply not as amenable to compiler optimization as they could be This seems like a rather important point that's glossed over. Typical code is not often as optimized and meticulously written. It would be nice to see how much compilers have improved there.

josephg 6y ago
Yeah I agree. It would be really interesting to see the performance difference of a larger program - firefox, the linux kernel, postgres, or maybe clang itself. Unfortunately it might be hard to get the same program to compile with both compilers without a bit of work.
- bsaul 6y ago
  isn't llvm 11 able to build old llvm 2.7 software ?
  - josephg 6y ago
    It should! I'd love to read an analysis if anyone is interested in running this test and publishing a followup.
    - AlphaSite 6y ago
      I haven’t done it in years, but with some hacking early versions of clang could totally compile blender and that’s quite amiable to benchmarking (maybe too much?).
      - josephg 6y ago
        Blender is probably already pretty well optimized through SIMD and specialized high performance math libraries and whatnot. I think it would be more interesting to see the difference with "general purpose code".
bombcar 6y ago
Yeah, I would suspect that a code that has "algorithms" is already pretty well optimized before it hits the compiler; the place where gains would be seen would be in "enterprise" or business code - or something like OpenOffice. Harder to benchmark.
- mpweiher 6y ago
  > harder to benchmark And typically much harder to care about, as in: those applications are not stuck in algorithms that benefit from compiler optimisation. They're waiting for the database. Or the user. Or on the graphics library.
  - adrianN 6y ago
    Typically those programs are waiting for data to arrive from RAM because they have terrible data locality and chase pointers to do everything.
    - dahfizz 6y ago
      It takes 15 to 20 nanoseconds to fetch something cold from ram. That's slow if you do it in a tight loop, but that's absolutely nothing in a gui application.
Narishma 6y ago
It's not glossed over. It's mentioned multiple times in the article.
flohofwoe 6y ago
More work for the optimizer (because the code hasn't been manually "pre-optimized") most likely means longer compile times too though, so the relation between "twice the compile time for 15% speedup" might not improve much, and the optimizer might spend a lot of time on code that actually doesn't need optimizing (because it's not "hot" code). I agree though, it would be great to see the same experiment on other code bases.

josephg 6y ago

Yeah I agree. It would be really interesting to see the performance difference of a larger program - firefox, the linux kernel, postgres, or maybe clang itself. Unfortunately it might be hard to get the same program to compile with both compilers without a bit of work.

bsaul 6y ago
isn't llvm 11 able to build old llvm 2.7 software ?
- josephg 6y ago
  It should! I'd love to read an analysis if anyone is interested in running this test and publishing a followup.
  - AlphaSite 6y ago
    I haven’t done it in years, but with some hacking early versions of clang could totally compile blender and that’s quite amiable to benchmarking (maybe too much?).
    - josephg 6y ago
      Blender is probably already pretty well optimized through SIMD and specialized high performance math libraries and whatnot. I think it would be more interesting to see the difference with "general purpose code".

bombcar 6y ago

Yeah, I would suspect that a code that has "algorithms" is already pretty well optimized before it hits the compiler; the place where gains would be seen would be in "enterprise" or business code - or something like OpenOffice. Harder to benchmark.

mpweiher 6y ago
> harder to benchmark And typically much harder to care about, as in: those applications are not stuck in algorithms that benefit from compiler optimisation. They're waiting for the database. Or the user. Or on the graphics library.
- adrianN 6y ago
  Typically those programs are waiting for data to arrive from RAM because they have terrible data locality and chase pointers to do everything.
  - dahfizz 6y ago
    It takes 15 to 20 nanoseconds to fetch something cold from ram. That's slow if you do it in a tight loop, but that's absolutely nothing in a gui application.

Narishma 6y ago

It's not glossed over. It's mentioned multiple times in the article.

flohofwoe 6y ago

More work for the optimizer (because the code hasn't been manually "pre-optimized") most likely means longer compile times too though, so the relation between "twice the compile time for 15% speedup" might not improve much, and the optimizer might spend a lot of time on code that actually doesn't need optimizing (because it's not "hot" code). I agree though, it would be great to see the same experiment on other code bases.