7 ms·

7 seconds slow.. I was happy when I decreased the 32 hours build to 4 hours.

jstimpfle 8y ago
7 seconds for a rebuild after a small edit is already a little unergonomic when you are iterating heavily. But it all depends on the project. Simple C-style code should compile at ~1M lines of code per second on today's machines without heavy optimization, and maybe compiling on 4 cores or so. (Not possible with common infrastructure like C, gcc/clang/msvc, large headers per compilation unit, many small files, object file style linking! and the awfully slow linkers (I wonder if they could be faster))
- pif 8y ago
  > ... should compile at ~1M lines of code per second ... > Not possible with common infrastructure If it is not possible with common infrastructure, what do base your estimate on?
  - jstimpfle 8y ago
    My own experiments. I have recently bought a new PC with an i5-7600 and my own pascal parser, written in straightforward unoptimized C, parses about 2M lines of code per second on a single thread. Also I'm following Jonathan Blow's streams (Jai language). With his own trivial x64 backend he has/had a very real compilation speed of 80K lines/s, and only the parser was parallelized. I think he indends to improve that to 800K lines/s. Note that his language is quite a bit more work to compile than a basic C-like language (which is about as easy as you can get if you discount the parsing model).
- Sir_Cmpwn 8y ago
  I'm not sure what you're talking about with respect to the C compiler. C compilers are the fastest around even before being paralllelized - and they can be.
  - jstimpfle 8y ago
    C parsing is rather slow due to the preprocessor (also: huuuge include files) and due to context-dependent syntax (need symbol table). And compiling itself need not be slow, but the popular compilers are not exactly super fast even with -O0. LLVM in particular has a reputation of being somewhat bloated. And finally, due to common practice of using a standard object file format, which is slow to link (at least with common tools), the whole build experience is not exactly blazingly fast. With clang/gcc/msvc, I think I'm more in the ballpark of 3K-30K lines/sec (-O0, 100-1000 lines/file, not counting basic std* includes).
    - Sir_Cmpwn 8y ago
      gold is pretty fast. I don't have a source but those numbers seem very off to me. C compilers are fast. C++ much less so, but C++ is a different language.
      - jstimpfle 8y ago
        I tried to follow gold's author's series on linkers once but gave up. Now that I've written an ELF-64 writer and know a little more about linking I should give it another shot. In any case, I think I've heard of still faster linkers than gold, and at the time I also found gold's object oriented architecture (as it was described in the series) questionable.
        Orphis 8y ago
        gold is sure fast compared to bfd.ld, but then you should try lld from the LLVM project which is much faster in general.
- vbezhenar 8y ago
  I knew one guy who used C++ but hated STL and most of standard library. He usually implemented data structures in-place as he need them (linked list, growable array, etc). While it seems strange to reimplement similar data structure over and over again, it worked for him. And one particularly wonderful thing was how fast his projects compiled. He had one project with few libraries, around 100k lines and it compiled very fast, something like second from the scratch and it was 10 years ago.
  - jki275 8y ago
    That's the way I learned C++. While I mostly use the STL now as it's much more convenient to write, there is an elegance to writing one's own tools.
    - sam0x17 8y ago
      you'd also be shocked, SHOCKED at how inefficient some of the STL implementations of basic data structure algorithms are
      - chillee 8y ago
        Of special note is the unordered_map/unordered_set implementations. https://news.ycombinator.com/item?id=7849213#7849607 https://news.ycombinator.com/item?id=7849213#7849607
        sam0x17 8y ago
        imagine all the CS research that relies on the STL expecting it to behave optimally
      - jki275 8y ago
        I've almost purposely avoided studying what the STL does because of that. I have some of my own notions on how some of the data structures ought to be built in order to be efficient, and sometimes I implement them myself to be sure I get the behavior I want.
- nicoburns 8y ago
  The linkers could definitely be faster! I've seen reports of Rust debug builds going from ~6 seconds to ~2 seconds by moving from gold to lld
- hinkley 8y ago
  Seven seconds is already pretty fast. Might have warranted some profiling and a brief attempt to speed up the parts of the compiler that your usage hits, since that could potentially be useful to everybody and thus magnify the value of cutting one second off a build. But this? Was a cool learning experience, no more, no less. 7 seconds is plenty fast enough to make “watch” the cheapest and most reliable solution to the speed problem.
SSLy 8y ago
Do not measure other people problems by your scale.

7 seconds for a rebuild after a small edit is already a little unergonomic when you are iterating heavily. But it all depends on the project. Simple C-style code should compile at ~1M lines of code per second on today's machines without heavy optimization, and maybe compiling on 4 cores or so. (Not possible with common infrastructure like C, gcc/clang/msvc, large headers per compilation unit, many small files, object file style linking! and the awfully slow linkers (I wonder if they could be faster))

pif 8y ago
> ... should compile at ~1M lines of code per second ... > Not possible with common infrastructure If it is not possible with common infrastructure, what do base your estimate on?
- jstimpfle 8y ago
  My own experiments. I have recently bought a new PC with an i5-7600 and my own pascal parser, written in straightforward unoptimized C, parses about 2M lines of code per second on a single thread. Also I'm following Jonathan Blow's streams (Jai language). With his own trivial x64 backend he has/had a very real compilation speed of 80K lines/s, and only the parser was parallelized. I think he indends to improve that to 800K lines/s. Note that his language is quite a bit more work to compile than a basic C-like language (which is about as easy as you can get if you discount the parsing model).
Sir_Cmpwn 8y ago
I'm not sure what you're talking about with respect to the C compiler. C compilers are the fastest around even before being paralllelized - and they can be.
- jstimpfle 8y ago
  C parsing is rather slow due to the preprocessor (also: huuuge include files) and due to context-dependent syntax (need symbol table). And compiling itself need not be slow, but the popular compilers are not exactly super fast even with -O0. LLVM in particular has a reputation of being somewhat bloated. And finally, due to common practice of using a standard object file format, which is slow to link (at least with common tools), the whole build experience is not exactly blazingly fast. With clang/gcc/msvc, I think I'm more in the ballpark of 3K-30K lines/sec (-O0, 100-1000 lines/file, not counting basic std* includes).
  - Sir_Cmpwn 8y ago
    gold is pretty fast. I don't have a source but those numbers seem very off to me. C compilers are fast. C++ much less so, but C++ is a different language.
    - jstimpfle 8y ago
      I tried to follow gold's author's series on linkers once but gave up. Now that I've written an ELF-64 writer and know a little more about linking I should give it another shot. In any case, I think I've heard of still faster linkers than gold, and at the time I also found gold's object oriented architecture (as it was described in the series) questionable.
      - Orphis 8y ago
        gold is sure fast compared to bfd.ld, but then you should try lld from the LLVM project which is much faster in general.
vbezhenar 8y ago
I knew one guy who used C++ but hated STL and most of standard library. He usually implemented data structures in-place as he need them (linked list, growable array, etc). While it seems strange to reimplement similar data structure over and over again, it worked for him. And one particularly wonderful thing was how fast his projects compiled. He had one project with few libraries, around 100k lines and it compiled very fast, something like second from the scratch and it was 10 years ago.
- jki275 8y ago
  That's the way I learned C++. While I mostly use the STL now as it's much more convenient to write, there is an elegance to writing one's own tools.
  - sam0x17 8y ago
    you'd also be shocked, SHOCKED at how inefficient some of the STL implementations of basic data structure algorithms are
    - chillee 8y ago
      Of special note is the unordered_map/unordered_set implementations. https://news.ycombinator.com/item?id=7849213#7849607 https://news.ycombinator.com/item?id=7849213#7849607
      - sam0x17 8y ago
        imagine all the CS research that relies on the STL expecting it to behave optimally
    - jki275 8y ago
      I've almost purposely avoided studying what the STL does because of that. I have some of my own notions on how some of the data structures ought to be built in order to be efficient, and sometimes I implement them myself to be sure I get the behavior I want.
nicoburns 8y ago
The linkers could definitely be faster! I've seen reports of Rust debug builds going from ~6 seconds to ~2 seconds by moving from gold to lld
hinkley 8y ago
Seven seconds is already pretty fast. Might have warranted some profiling and a brief attempt to speed up the parts of the compiler that your usage hits, since that could potentially be useful to everybody and thus magnify the value of cutting one second off a build. But this? Was a cool learning experience, no more, no less. 7 seconds is plenty fast enough to make “watch” the cheapest and most reliable solution to the speed problem.

SSLy 8y ago

Do not measure other people problems by your scale.