4 ms·

At the highest level of performance needs, if you want to parallelize the build across machines, you're going to need some kind of storage abstraction for inter

by jchw 4y ago

At the highest level of performance needs, if you want to parallelize the build across machines, you're going to need some kind of storage abstraction for intermediate files since translation units are the easiest place to split builds at. In some places, where builds are particularly optimized, there are special distributed filesystems just for object files. In this case, it's not necessarily true that the object files are backed by disk even. Being backed by disk locally mainly helps for incrementally building so that you can change one file and only recompile intermediate files for translation units that depend on this file. Disk/FS caching presumably helps a lot with redundant I/O, and I think most of the benefit of building with tmpfs winds up being putting the source itself in RAM. Edit: also it's worth noting that many compilers can output object files which can be linked by other linkers, allowing you to mix output from different compilers in some circumstances.

yazaddaruvala 4y ago
Thank you the distributed compile usecase is a fun one to think about. I can see why it would require some intermediate set of bytes to shuffle across the network. > Being backed by disk locally mainly helps for incrementally building so that you can change one file and only recompile intermediate files for translation units that depend on this file. For the local incremental usecase I’d love to see a more state-full compiler instead. One that could change the bytes of a binary instead. e.g. give all functions some additional “empty padding”. Then any modifications could directly fit into the binary as needed until some defragmentation process which creates the final output binary.
- jchw 4y ago
  Don't quote me on this, but I do believe MSVC does exactly this by default. The option is /INCREMENTAL. That said, it accomplishes this still using an on-disk datastore.

by jchw 4y ago

yazaddaruvala 4y ago

Thank you the distributed compile usecase is a fun one to think about. I can see why it would require some intermediate set of bytes to shuffle across the network. > Being backed by disk locally mainly helps for incrementally building so that you can change one file and only recompile intermediate files for translation units that depend on this file. For the local incremental usecase I’d love to see a more state-full compiler instead. One that could change the bytes of a binary instead. e.g. give all functions some additional “empty padding”. Then any modifications could directly fit into the binary as needed until some defragmentation process which creates the final output binary.

jchw 4y ago
Don't quote me on this, but I do believe MSVC does exactly this by default. The option is /INCREMENTAL. That said, it accomplishes this still using an on-disk datastore.