5 ms·

It seems odd to me that we still require .o files at all. Given the final program needs to be storable in RAM + Virtual Memory it surprises me that we still ne

It seems odd to me that we still require .o files at all. Given the final program needs to be storable in RAM + Virtual Memory it surprises me that we still need the intermediate step of pushing to the file system only to then immediately reopen and merge those files. Does someone have more info on this? Or is the reason just legacy? Or is the reason just some ideological “single responsibility” thing?

barbegal 4y ago
Builds can be much faster if you don't need to recompile all the modules into .o files.
forrestthewoods 4y ago
What problem are you trying to solve by keeping .o files in a processes’s memory without spilling to disk?
- yazaddaruvala 4y ago
  To reduce the latency of compile-link-test
  - forrestthewoods 4y ago
    I don’t think writing to and reading from disk is a significant portion of this loop. You’d want to measure the cost and thus maximum benefit. SSDs are fast. There isn’t much upside here.
    - yazaddaruvala 4y ago
      The problem may be disk speed, but I agree it may not. The problem I see is the dropping of state. Specifically, to produce a .o file the compiler has already read through and created indexes of module::function/struct names, their layouts, dependencies, etc. My understanding is that the .o files need to be re-parsed by the linker to create these indexes, layouts, and dependencies. Especially with LTOs I'd imagine there would be additional inlining work (stuff the compiler is already good at). This is all just wasted time, including the IO bottlenecks - even if those are marginal.
jchw 4y ago
At the highest level of performance needs, if you want to parallelize the build across machines, you're going to need some kind of storage abstraction for intermediate files since translation units are the easiest place to split builds at. In some places, where builds are particularly optimized, there are special distributed filesystems just for object files. In this case, it's not necessarily true that the object files are backed by disk even. Being backed by disk locally mainly helps for incrementally building so that you can change one file and only recompile intermediate files for translation units that depend on this file. Disk/FS caching presumably helps a lot with redundant I/O, and I think most of the benefit of building with tmpfs winds up being putting the source itself in RAM. Edit: also it's worth noting that many compilers can output object files which can be linked by other linkers, allowing you to mix output from different compilers in some circumstances.
- yazaddaruvala 4y ago
  Thank you the distributed compile usecase is a fun one to think about. I can see why it would require some intermediate set of bytes to shuffle across the network. > Being backed by disk locally mainly helps for incrementally building so that you can change one file and only recompile intermediate files for translation units that depend on this file. For the local incremental usecase I’d love to see a more state-full compiler instead. One that could change the bytes of a binary instead. e.g. give all functions some additional “empty padding”. Then any modifications could directly fit into the binary as needed until some defragmentation process which creates the final output binary.
  - jchw 4y ago
    Don't quote me on this, but I do believe MSVC does exactly this by default. The option is /INCREMENTAL. That said, it accomplishes this still using an on-disk datastore.
olig15 4y ago
We make use of lots of distributed compiles at work which .o(bj) files are useful for, but to compile something locally it might be useful to keep things in memory - if all of the objects fit. The code base’s objects I work on won’t fit in ram, so will get paged to disk at some point during linking anyway
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
rcme 4y ago
Seems like it’d be useful to have .o files on disk for incremental compilation.
guenthert 4y ago
Well, you might know that all those source files belong to exactly one executable (or library), but how would the compiler know?
- wmf 4y ago
  You could combine the build system (e.g. make), the linker, and the compiler into one app that knows everything and does it all in memory. It could be slightly faster but much less flexible.
  - colejohnson66 4y ago
    Isn't that what msbuild does for .NET (Core and 5+, not Framework)? The `obj` directory contains some intermediate stuff, but not object files.
sargstuff 4y ago
What about the case of different compiler generated object files from different languages / object file formats being combined? (without the need to have access to each unique compiler generating the .o file)
JonChesterfield 4y ago
Build directory in a tmpfs ram disk works really well.
jcranmer 4y ago
There's actually quite a few reasons. The first thing to point out is that reading a recently-written file isn't all that expensive: it's stored in the filesystem cache anyways, so you bypass the disk for the reads. Moreover, the filesystem is actually a decent database for multiprocess communication. If every translation unit is compiled into an independent file, which is then combined into a single final output, there is no need to build any complex locking mechanisms or the like and you still get to take advantage of the embarrassingly parallel nature of compiling. Incremental compilation is an incredibly important tool. If you make a small change to one file, it's frequently not necessary to rebuild most of the code. Making the output of individual file compilations work in a way that allows incremental compilation to happen requires basically building .o files--and there's very little savings to be had by not emitting them to disk. Finally, I'll note that very frequently, debug builds of large applications cannot fit in RAM. Debugging symbols bloat builds tremendously, especially in intermediate object form (since many symbols end up needing to be duplicated in every single .o file). A debug build of a large application may take up 80GB in disk space, to build a binary that (without debug symbols) would be perhaps 100MB in size. Keeping everything in RAM just isn't feasible at scale.
simplotek 4y ago
> Given the final program needs to be storable in RAM + Virtual Memory it surprises me that we still need the intermediate step of pushing to the file system only to then immediately reopen and merge those files. You don't. I recommend you learn about unity builds, and how major build systems support toggling them at the project and subproject level. The main reason why most people haven't heard about the concept and those who did the majority doesn't bother with it is that a) you have little to nothing to gain by them, b) you throw incremental builds out if the window, c) you ruin internal linkage and thus can introduce hard to track errors. Also, it makes no sense at all to argue how the released software needs to run in memory to justify aspects related to how the software is built. At most you have arguments over code bloat and premature optimization, but at what cost?

barbegal 4y ago

Builds can be much faster if you don't need to recompile all the modules into .o files.

forrestthewoods 4y ago

What problem are you trying to solve by keeping .o files in a processes’s memory without spilling to disk?

yazaddaruvala 4y ago
To reduce the latency of compile-link-test
- forrestthewoods 4y ago
  I don’t think writing to and reading from disk is a significant portion of this loop. You’d want to measure the cost and thus maximum benefit. SSDs are fast. There isn’t much upside here.
  - yazaddaruvala 4y ago
    The problem may be disk speed, but I agree it may not. The problem I see is the dropping of state. Specifically, to produce a .o file the compiler has already read through and created indexes of module::function/struct names, their layouts, dependencies, etc. My understanding is that the .o files need to be re-parsed by the linker to create these indexes, layouts, and dependencies. Especially with LTOs I'd imagine there would be additional inlining work (stuff the compiler is already good at). This is all just wasted time, including the IO bottlenecks - even if those are marginal.

jchw 4y ago

At the highest level of performance needs, if you want to parallelize the build across machines, you're going to need some kind of storage abstraction for intermediate files since translation units are the easiest place to split builds at. In some places, where builds are particularly optimized, there are special distributed filesystems just for object files. In this case, it's not necessarily true that the object files are backed by disk even. Being backed by disk locally mainly helps for incrementally building so that you can change one file and only recompile intermediate files for translation units that depend on this file. Disk/FS caching presumably helps a lot with redundant I/O, and I think most of the benefit of building with tmpfs winds up being putting the source itself in RAM. Edit: also it's worth noting that many compilers can output object files which can be linked by other linkers, allowing you to mix output from different compilers in some circumstances.

yazaddaruvala 4y ago
Thank you the distributed compile usecase is a fun one to think about. I can see why it would require some intermediate set of bytes to shuffle across the network. > Being backed by disk locally mainly helps for incrementally building so that you can change one file and only recompile intermediate files for translation units that depend on this file. For the local incremental usecase I’d love to see a more state-full compiler instead. One that could change the bytes of a binary instead. e.g. give all functions some additional “empty padding”. Then any modifications could directly fit into the binary as needed until some defragmentation process which creates the final output binary.
- jchw 4y ago
  Don't quote me on this, but I do believe MSVC does exactly this by default. The option is /INCREMENTAL. That said, it accomplishes this still using an on-disk datastore.

olig15 4y ago

We make use of lots of distributed compiles at work which .o(bj) files are useful for, but to compile something locally it might be useful to keep things in memory - if all of the objects fit. The code base’s objects I work on won’t fit in ram, so will get paged to disk at some point during linking anyway

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

rcme 4y ago

Seems like it’d be useful to have .o files on disk for incremental compilation.

guenthert 4y ago

Well, you might know that all those source files belong to exactly one executable (or library), but how would the compiler know?

wmf 4y ago
You could combine the build system (e.g. make), the linker, and the compiler into one app that knows everything and does it all in memory. It could be slightly faster but much less flexible.
- colejohnson66 4y ago
  Isn't that what msbuild does for .NET (Core and 5+, not Framework)? The `obj` directory contains some intermediate stuff, but not object files.

sargstuff 4y ago

What about the case of different compiler generated object files from different languages / object file formats being combined? (without the need to have access to each unique compiler generating the .o file)

JonChesterfield 4y ago

Build directory in a tmpfs ram disk works really well.

jcranmer 4y ago

There's actually quite a few reasons. The first thing to point out is that reading a recently-written file isn't all that expensive: it's stored in the filesystem cache anyways, so you bypass the disk for the reads. Moreover, the filesystem is actually a decent database for multiprocess communication. If every translation unit is compiled into an independent file, which is then combined into a single final output, there is no need to build any complex locking mechanisms or the like and you still get to take advantage of the embarrassingly parallel nature of compiling. Incremental compilation is an incredibly important tool. If you make a small change to one file, it's frequently not necessary to rebuild most of the code. Making the output of individual file compilations work in a way that allows incremental compilation to happen requires basically building .o files--and there's very little savings to be had by not emitting them to disk. Finally, I'll note that very frequently, debug builds of large applications cannot fit in RAM. Debugging symbols bloat builds tremendously, especially in intermediate object form (since many symbols end up needing to be duplicated in every single .o file). A debug build of a large application may take up 80GB in disk space, to build a binary that (without debug symbols) would be perhaps 100MB in size. Keeping everything in RAM just isn't feasible at scale.

simplotek 4y ago

> Given the final program needs to be storable in RAM + Virtual Memory it surprises me that we still need the intermediate step of pushing to the file system only to then immediately reopen and merge those files. You don't. I recommend you learn about unity builds, and how major build systems support toggling them at the project and subproject level. The main reason why most people haven't heard about the concept and those who did the majority doesn't bother with it is that a) you have little to nothing to gain by them, b) you throw incremental builds out if the window, c) you ruin internal linkage and thus can introduce hard to track errors. Also, it makes no sense at all to argue how the released software needs to run in memory to justify aspects related to how the software is built. At most you have arguments over code bloat and premature optimization, but at what cost?