6 ms·
Feedback of someone who is used to manage large (>1500) software stack in C / C++ / Fortran / Python / Rust / etc: - (1) Provide a way to compile without inter
by adev_ 6mo ago
Feedback of someone who is used to manage large (>1500) software stack in C / C++ / Fortran / Python / Rust / etc:
- (1) Provide a way to compile without internet access and specify the associated dependencies path manually. This is absolutely critical.
Most 'serious' multi-language package managers and integration systems are building in a sandbox without internet access for security reasons and reproducibility reasons.
If your build system does not allow to build offline and with manually specified dependencies, you will make life of integrators and package managers miserable and they will avoid your project.
(2) Never ever build in '-03 -march=native' by default. This is always a red flag and a sign of immaturity. People expect code to be portable and shippable.
Good default options should be CMake equivalent of "RelWithDebInfo" (meaning: -O2 -g -DNDEBUG ).
-O3 can be argued. -march=native is always always a mistake.
- (3) Allow your build tool to be built by an other build tool (e.g CMake).
Anybody caring about reproducibility will want to start from sources, not from a pre-compiled binary. This also matter for cross compilation.
- (4) Please offer a compatibility with pkg-config (https://en.wikipedia.org/wiki/Pkg-config https://en.wikipedia.org/wiki/Pkg-config) and if possible CPS (https://cps-org.github.io/cps/overview.html https://cps-org.github.io/cps/overview.html) for both consumption and generation.
They are what will allow interoperability between your system and other build systems.
- (5) last but not least: Consider seriously the cross-compilation use case.
It is common in the world of embedded systems to cross compile. Any build system that does not support cross-compilation will be de facto banned from the embedded domain.
- moralestapia 6mo ago>15000 15000 what?
- adev_ 6mo ago1500 C/C++ individual software components. The 15000 was a typo on my side. Fixed.
- moralestapia 6mo agoI see, thanks. I didn't mind the number it just wasn't clear what was it about.
- deleted 6mo ago[deleted]
- tgma 6mo ago> -march=native is always always a mistake Gentoo user: hold my beer.
- jjmarr 6mo agoIt's also an option on NixOS but I haven't managed to get it working unlike Gentoo.
- CarVac 6mo agoGentoo binaries aren't shipped that way
- greenavocado 6mo agoGentoo..... distributes binaries?
- rascul 6mo agoYes https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart
- digitalPhonix 6mo agoBut not with march=native? The distirbuted binaries use two standard instruction sets for x86-64 and one for arm like “march=x86-64-v3” https://wiki.gentoo.org/wiki/Gentoo_binhost/Available_packages_and_configurations#amd64_x86-64-v3_variant https://wiki.gentoo.org/wiki/Gentoo_binhost/Available_packag...
- account42 6mo agoYou can have your own binary host or even just compile packages on another host on demand. -march=native is a concern in both cases.
- account42 6mo ago
- CoastalCoder 6mo ago> Never ever build in '-03 -march=native' by default. This is always a red flag and a sign of immaturity. Perhaps you can see how there are some assumptions baked into that statement.
- eqvinox 6mo agoWhat assumptions would that be? Shipping anything built with -march=native is a horrible idea. Even on homogeneous targets like one of the clouds, you never know if they'll e.g. switch CPU vendors. The correct thing to do is use microarch levels (e.g. x86-64-v2) or build fully generic if the target architecture doesn't have MA levels.
- tempest_ 6mo agoI build on the exact hardware I intend to deploy my software to and ship it to another machine with the same specs as the one it was built on. I am willing to hear arguments for other approaches.
- eqvinox 6mo agoI'm willing to hear arguments for your approach? it certainly has scale issues when you need to support larger deployments. [P.S.: the way I understand the words, "shipping" means "passing it off to someone else, likely across org boundaries" whereas what you're doing I'd call "deploying"]
- teo_zero 6mo agoSo, do you see now the assumptions baked in your argument? > when you need to support larger deployments > shipping > passing it off to someone else
- zahllos 6mo agoNot the OP, but: -march says the compiler can assume that the features of that particular CPU architecture family, which is broken out by generation, can be relied upon. In the worst case the compiler could in theory generate code that does not run on older CPUs of the same family or from different vendors. -mtune says "generate code that is optimised for this architecture" but it doesn't trigger arch specific features. Whether these are right or not depends on what you are doing. If you are building gentoo on your laptop you should absolutely -mtune=native and -march=native. That's the whole point: you get the most optimised code you can for your hardware. If you are shipping code for a wide variety of architectures and crucially the method of shipping is binary form then you want to think more about what you might want to support. You could do either: if you're shipping standard software pick a reasonable baseline (check what your distribution uses in its cflags). If however you're shipping compute-intensive software perhaps you load a shared object per CPU family or build your engine in place for best performance. The Intel compiler quite famously optimised per family, included all the copies in the output and selected the worst one on AMD ;) (https://medium.com/codex/fixing-intel-compilers-unfair-cpu-dispatcher-part-1-2-4a4a367c8919 https://medium.com/codex/fixing-intel-compilers-unfair-cpu-d...)
- Teknoman117 6mo agoAs someone who has also spent two decades wrangling C/C++ codebases, I wholeheartedly agree with every statement here. I have an even stronger sentiment regarding cross compilation though - In any build system, I think the distinction between “cross” and “non-cross” compilation is an anti-pattern. Always design build systems assuming cross compilation. It hurts nothing if it just so happens that your host and target platform/architecture end up being the same, and saves you everything down the line if you need to also build binaries for something else.
- bsder 6mo ago> In any build system, I think the distinction between “cross” and “non-cross” compilation is an anti-pattern. This is one of the huge wins of Zig. Any Zig host compiler can produce output for any supported target. Cross compiling becomes straightforward.
- sebastos 6mo agoAmen. It always baffled me that cross compiling was ever considered a special, weird, off-nominal thing. I’d love to understand the history of that better, because it seems like it should have been obvious from the start that building for the exact same computer you’re compiling from is a special case.
- Teknoman117 6mo agoA few things come to mind, but I wasn't even alive then so what do I know XD. On one hand, it seems rather strange, because back in the early days of C (and later C++) there were far more CPU architectures in play. Every big Unix hardware vendor had their own CPU architecture, whereas today we only have about six. (In my mind: x86, arm, mips, risc-v, ppc, and s390x) But it might be that in the early days of C/C++, development involved connecting to large shared Unix environments where the machine you developed on what always the machine (or at least the same type of machine) the program would run on, and also that those vendors weren't exactly incentivized to make developing for competitor's architectures easy.
- pjmlp 6mo agoAgree with the feedback. Also the problem isn't creating a cargo like tool for C and C++, that is the easy part, the problem is getting more userbase than vcpkg or conan for it to matter for those communities.
- criticalfault 6mo agosince you have a lot of experience, can I ask what do you think about this: - skipping cmake completely? would this be feasible? - integration of other languages in the project? - how to handle qt?
- adev_ 6mo ago> skipping cmake completely? would this be feasible? Feasible but difficult. CMake has a tremendous user mass, so you do want to be able to use a CMake-based project as a dependency. The CMake Target/Config export system expose CMake internals and make that difficult to consume a CMake built project without CMake. The cleanest way to do that is probably what xmake is doing: Calling cmake and extract targets information from CMake to your own build system with some scripting. It is flaky but xmake has proven it is doable. That's said: CPS should make that easier on the longer term. Please also consider that CMake is doing a lot of work under the hood to contains compiler quirks that you will have to do manually. > integration of other languages in the project? Trying to integrate higher level languages (Python, JS) in package managers of lower level languages (C, C++) is generally a bad idea. The dependency relation is inverted and interoperability betweens package managers is always poor. Diamond dependency and conflicting versions will become quickly a problem. I would advise to just expose properly your build system with the properties I described and use a multi-language package manager (e.g Nix) or, at default, the higher level language package manager (e.g uv with a scikit-build-core equivalent) on top of that. This will be one order of magnitude easier to do. > how to handle qt? Qt is nothing special to handle. Qt is a multi language framework (C++, MOC, QML, JS and even python for PySide) and need to be handle as such.