3 ms·
I'm an old Delphi/Pascal application programmer here, trying to get the lay of the land in the C/C++ world. Why is there a forest of files in Cosmopolitan Libc
by mikewarot 6y ago
I'm an old Delphi/Pascal application programmer here, trying to get the lay of the land in the C/C++ world.
Why is there a forest of files in Cosmopolitan Libc? I tried looking at it on Github to see how things were done, and there were a lot of .h and .S files, I couldn't actually find the C source, though I'm sure somewhere in there is a .c file.
Doesn't fragmenting the source into thousands of files make compilation far slower than it needs to be?
Also, I wonder if/how functions that aren't called in a program get trimmed away by the linker, and thus don't make the executable larger.
- jart 6y agoIt takes about 18 seconds to build the entire Cosmopolitan repo and run all its unit tests on my PC. During that time the make command builds 14,376 .o files, 66 .a archives, and 421 .com executables. It's an exceedingly fast build config. Having lots of objects is a good thing because it helps static linking work better. When the Unix linker loads a symbol from a static archive, it pulls in the whole module that defines that symbol. For example, if you define memcpy() and memset() in the same .c file and then your app only needs one of them, they'll both go towards bloating your binary. Workarounds exist like -ffunction-sections and -Wl,--gc-sections but a C library should make assumptions about the fewest flags feasible. What's the end result? We're able to build executables that are 12kb in size which run on seven different operating systems. The big codebase is what made tiny binaries possible: https://justine.lol/cosmopolitan/howfat.html https://justine.lol/cosmopolitan/howfat.html
- krab 6y agoMay I ask - what was the reason for creating APE/Cosmopolitan? I read your post about actually portable executable format but I wonder if it's something that you found immediately helpful for some project or if you work on it just out of curiosity.
- matheusmoreira 6y agoDo you build on tmpfs? I started doing that and it makes builds ridiculously fast. $XDG_RUNTIME_DIR is perfect for this, I just make a directory there and symlink the build directory to it. Even added some makefile logic to ensure the link target exists: build_directory_link := $(shell readlink $(build_directory)) $(if $(build_directory_link),$(shell mkdir -p $(build_directory_link)))
- ed25519FUUU 6y agoGood optimization, thanks for the pointer!
- matheusmoreira 6y agoIt also reduces wear on storage devices. Why write temporary build artifacts to permanent memory after all? Gotta use those 32 GB of RAM for something. Linux will automatically cache all source files in RAM because of the page table cache. Writes to slow storage devices are eliminated through tmpfs. Amazing really.
- wyldfire 6y agoAlso, if you are using BFD ld, you should consider gold or lld. Significantly faster if the link is the slow step. Also much lower peak memory usage for lld, which is great if you have multiple concurrent link steps.
- simias 6y agoCompiling ASM and even C is very fast these days. No generics, small compilation units etc... Even with optims it's usually a non issue in my experience. For a datapoint it takes me ~2 minutes to build a reasonably fully featured ARM linux kernel with a cold cache on a 5 year old i5. wc counts 1399 CC, 65 AS and 411 LD calls. And of course incremental rebuilds are only a fraction of that.
- moonchild 6y ago> Doesn't fragmenting the source into thousands of files make compilation far slower than it needs to be? This allows incremental compilation to work much better than it otherwise would. Your first clean build may be slightly slower, but after that, you only have to recompile the files you change.