5 ms·

What gets my goat about this paper is the amount of misleading information in it. Re-parsing files multiple times is a solved problem using the standard header

by cjensen 14y ago

What gets my goat about this paper is the amount of misleading information in it. Re-parsing files multiple times is a solved problem using the standard header guard macro pattern combined with a slightly intelligent C Preprocessor. Yet quotes about the horror of header blow-up pervade this paper as if this is an exciting new problem that has been solved. It isn't; it was solved more than 20 years ago.

enneff 14y ago
Yes, there are ways around it, and the paper discusses one way Google addressed it. But that's not the point. It's used as just one example to illustrate why C/C++ is not ideal for working at Google's scale.
- cjensen 14y ago
  I agree there's much more to the paper, and it's well worth the read. Bounds checking, avoiding pointer arithmetic, garbage collection, and concurrency all make sense at Google's scale. I just think the bashing was misguided in an attempt to better justify adding a subset of Ada to C
  - enneff 14y ago
    I think it's misguided to try and characterize Go as the addition of a subset of Ada to C. ;-)
akkartik 14y ago
Yes, the guard macro pattern works. But it's something programmers manage, so it's a band-aid rather than a solution. If you change a header you need to recompile all files that include it, even if the include was conditional. Now your build system starts to look inside files rather than just check timestamps. Similar tool bloat happens over and over again in large codebases. I'm not convinced Go has the right solution. But it's disingenuous to claim that it's competing with the C preprocessor. It's leagues better. You should enumerate all the information in the paper you consider to be misleading.
- jeremyjh 14y ago
  Can you name a language developed since the 80s that hasn't also solved this problem?
  - akkartik 14y ago
    From the article: "It's worth mentioning that this general approach to dependency management is not original; the ideas go back to the 1970s and flow through languages like Modula-2 and Ada. In the C family Java has elements of this approach."
  - enneff 14y ago
    Python? Ruby? JavaScript? There are many more, but I name these three as they are among most popular.
    - jfb 14y ago
      How has Javascript even addressed this problem?
- signa11 14y ago
  i am kind of surprised that no one has brought up "#pragma once" or even external-include-guards (as proposed by jon-lakos). both seem to drastically cut down build times...
chubot 14y ago
Why then does every nontrivial C or C++ code base I've ever worked on take 5 or 20 or 60 minutes to compile then? Do you honestly think it should take 5 minutes to compile 1M lines of code? The linker AFAIK also has bottlenecks, but with today's computers, and thinking about what a C compiler actually does, there's no reason that you shouldn't be download (say) Python, Perl, or Ruby and compile them in 3 or 4 seconds. All of those codebases are 1M lines or less.
- fdr 14y ago
  On compilation of C programs: On my unremarkable but not slouchy three-year-old workstation that possesses rotational disks and about 6GB of buffer cache. This cache has not been pre-warmed either, but I can't guarantee it is totally cold. There are with compiler optimizations on, too. $ cd ~/codes/postgres $ make distclean $ make -sj10 All of PostgreSQL successfully made. Ready to install. 247.83user 14.22system 0:46.26elapsed 566%CPU Depending on how you count it, this is 600,000 to 1M LOC. Let's try complete Linux compile without the drivers, via compiling user-mode linux. This is normally sufficient for debugging file systems or the memory management system (it's really a godsend for that), for example: $ cd ~/codes/linux env ARCH=um time make -sj16 LINK linux 321.04user 23.83system 1:02.49elapsed 551%CPU All in all, I have always found the complaints about performance and header files in C programs (note: not C++) either outmoded or unconvincing. There are plenty of legitimate problems -- including tricky issues with semantics -- involving the preprocessor, including something as simple as figuring out when it is safe to remove or reorder a header include! There's no need to imagine problems. This meme about how terrible compile times for large C programs is needs to be put out to pasture, or maybe I just need to be exposed to code bases about ten times the size and feel the need to not use incremental builds.
  - chubot 14y ago
    Thanks for actually posting some numbers, but to me this shows the problem and not the absence of it. It looks like you have 6 CPUs, and say you have 2e9 cycles per second on each of them. 46s * 6 * 2e9 / 1e6 LOC = 552,000 cycles to compile a single line of code on average. That seems 1-2 orders of magnitude off, no? When you had a 10Mhz computer, did you compile at 20 LOC per second? There's a scaling problem here. (That's ignoring disk access, but if you did it multiple times I doubt it would go more than twice as fast the second time. And to read 1M LOC from disk cold I'd guess should take on the order of 500ms anyway.) Is what you're saying is that those compilations are fast enough? You wouldn't want it to take 1 or 3 seconds and see no reason why it should? Also, Postgres and Linux I'm sure have a LOT better physical structure than your typical industry project. Most industrial projects are bound by talent, I would say. If choosing Go gets rid of this problem, then I would say that's a definite advantage on the side of Go, to be considered when choosing a language for a new project. And it's true that incremental builds are more important, but on projects I've worked on they take 10-60 seconds for a 1 line change in a .cc file and much worse when you're changing headers.
gsg 14y ago
In file A, #include "foo.h", parsed once. In file B, #include "foo.h", parsed once (again). In file C... Include guard optimisations do not solve this problem.

by cjensen 14y ago

enneff 14y ago

Yes, there are ways around it, and the paper discusses one way Google addressed it. But that's not the point. It's used as just one example to illustrate why C/C++ is not ideal for working at Google's scale.

cjensen 14y ago
I agree there's much more to the paper, and it's well worth the read. Bounds checking, avoiding pointer arithmetic, garbage collection, and concurrency all make sense at Google's scale. I just think the bashing was misguided in an attempt to better justify adding a subset of Ada to C
- enneff 14y ago
  I think it's misguided to try and characterize Go as the addition of a subset of Ada to C. ;-)

akkartik 14y ago

Yes, the guard macro pattern works. But it's something programmers manage, so it's a band-aid rather than a solution. If you change a header you need to recompile all files that include it, even if the include was conditional. Now your build system starts to look inside files rather than just check timestamps. Similar tool bloat happens over and over again in large codebases. I'm not convinced Go has the right solution. But it's disingenuous to claim that it's competing with the C preprocessor. It's leagues better. You should enumerate all the information in the paper you consider to be misleading.

jeremyjh 14y ago
Can you name a language developed since the 80s that hasn't also solved this problem?
- akkartik 14y ago
  From the article: "It's worth mentioning that this general approach to dependency management is not original; the ideas go back to the 1970s and flow through languages like Modula-2 and Ada. In the C family Java has elements of this approach."
- enneff 14y ago
  Python? Ruby? JavaScript? There are many more, but I name these three as they are among most popular.
  - jfb 14y ago
    How has Javascript even addressed this problem?
signa11 14y ago
i am kind of surprised that no one has brought up "#pragma once" or even external-include-guards (as proposed by jon-lakos). both seem to drastically cut down build times...

chubot 14y ago

Why then does every nontrivial C or C++ code base I've ever worked on take 5 or 20 or 60 minutes to compile then? Do you honestly think it should take 5 minutes to compile 1M lines of code? The linker AFAIK also has bottlenecks, but with today's computers, and thinking about what a C compiler actually does, there's no reason that you shouldn't be download (say) Python, Perl, or Ruby and compile them in 3 or 4 seconds. All of those codebases are 1M lines or less.

fdr 14y ago
On compilation of C programs: On my unremarkable but not slouchy three-year-old workstation that possesses rotational disks and about 6GB of buffer cache. This cache has not been pre-warmed either, but I can't guarantee it is totally cold. There are with compiler optimizations on, too. $ cd ~/codes/postgres $ make distclean $ make -sj10 All of PostgreSQL successfully made. Ready to install. 247.83user 14.22system 0:46.26elapsed 566%CPU Depending on how you count it, this is 600,000 to 1M LOC. Let's try complete Linux compile without the drivers, via compiling user-mode linux. This is normally sufficient for debugging file systems or the memory management system (it's really a godsend for that), for example: $ cd ~/codes/linux env ARCH=um time make -sj16 LINK linux 321.04user 23.83system 1:02.49elapsed 551%CPU All in all, I have always found the complaints about performance and header files in C programs (note: not C++) either outmoded or unconvincing. There are plenty of legitimate problems -- including tricky issues with semantics -- involving the preprocessor, including something as simple as figuring out when it is safe to remove or reorder a header include! There's no need to imagine problems. This meme about how terrible compile times for large C programs is needs to be put out to pasture, or maybe I just need to be exposed to code bases about ten times the size and feel the need to not use incremental builds.
- chubot 14y ago
  Thanks for actually posting some numbers, but to me this shows the problem and not the absence of it. It looks like you have 6 CPUs, and say you have 2e9 cycles per second on each of them. 46s * 6 * 2e9 / 1e6 LOC = 552,000 cycles to compile a single line of code on average. That seems 1-2 orders of magnitude off, no? When you had a 10Mhz computer, did you compile at 20 LOC per second? There's a scaling problem here. (That's ignoring disk access, but if you did it multiple times I doubt it would go more than twice as fast the second time. And to read 1M LOC from disk cold I'd guess should take on the order of 500ms anyway.) Is what you're saying is that those compilations are fast enough? You wouldn't want it to take 1 or 3 seconds and see no reason why it should? Also, Postgres and Linux I'm sure have a LOT better physical structure than your typical industry project. Most industrial projects are bound by talent, I would say. If choosing Go gets rid of this problem, then I would say that's a definite advantage on the side of Go, to be considered when choosing a language for a new project. And it's true that incremental builds are more important, but on projects I've worked on they take 10-60 seconds for a 1 line change in a .cc file and much worse when you're changing headers.

gsg 14y ago

In file A, #include "foo.h", parsed once. In file B, #include "foo.h", parsed once (again). In file C... Include guard optimisations do not solve this problem.