4 ms·

I just spent 3 days to statically link a third party dependency to my C++ project, in a way that works with linux/windows/macos. The package wasn't available on

by lairv 2y ago

I just spent 3 days to statically link a third party dependency to my C++ project, in a way that works with linux/windows/macos. The package wasn't available on conan/vcpkg, it was just a github repo with a weird combination of Makefile/cmake file. While I'm far from a cmake/cpp expert, this is a non-issue with most modern languages: you just pip install, cargo add, npm install, go get etc. You can read front-to-cover books about every details of C++ semantics and it'll still be a PITA to work on real world projects

77pt77 2y ago
The javascript toolchain for dependencies and build is as arcane as the sloppiest Makefile. It's amazing that we ended up with the javascript equivalente of autotools.
- agumonkey 2y ago
  yeah, dealing with npm really made me feel like the best part of autotools
- dannymcgee 2y ago
  Created an account just to say: I am a 10-year JavaScript veteran who knows the ecosystem at least as well as any sane person could possibly be expected to, and this is 100% correct. When I tell people that JS build tooling is the second-worst headache to C/C++ build tooling, they laugh, but stick a toe outside the happy path of `npm create <some-framework-project> && npm start` and you will know what pain feels like. I did get some similar vibes trying to work with Gradle and Maven for a project a while back, but I don't really live in that world so it's hard to say whether my experience was typical or just symptomatic of my inexperience.
  - smarkov 2y ago
    And it'll continue to get worse because JS devs love coming up with their own abstractions over everything and piling complexity. You don't write JS, you write TS. You don't write HTML, you write JSX. You don't write CSS, you write Tailwind. And of course, React has its own compiler as of recently. Now you have to figure out in what order to run these parsers, bundle everything together including your hundreds of dependencies, minimize, obfuscate, tree shake, and whatever else fairy dust magic you want to sprinkle on top. Meanwhile, the default build tool of choice changes about once every 2 years, and now you also have a choice of runtime - node vs bun vs deno. Can't wait to see what the next revolutionary idea would be to contribute to this madness.
    - claytongulick 2y ago
      There are those of us out there that have a beautiful experience with JavaScript. Vanilla JS, web components, a lightweight rendering library like lit-html, class props as reactive state... NodeJS + express + vanilla web components is the most graceful and productive stack I've ever used in 30+ years of development.
    - dannymcgee 2y ago
      You forgot to mention the 3 or 4 different TypeScript compilers! I do have to admit at least some of the pain is self-inflicted on my part. I _want_ those nice abstractions, so I will write a damn Rollup or Babel plugin if it means I can use Sass stylesheets in my TypeScript+JSX components that compile to vanilla Custom Elements (so I don't have to depend on a huge framework runtime). But as nice as the Babel/Rollup/Vite plugin APIs are, when you start doing stuff like that you then have to deal with all the deep-in-the-weeds bullshit that comes with it, like the fact that Webpack and Rollup and Node.js all have subtly different (and mostly poorly documented) module resolution rules and that they all differ wildly from the official ES-module spec that the standards committee finally shipped like 20 years too late, so trying to get your unit-testing framework and your bundle toolchain and your browser to all agree about how to digest the mess you've made becomes this giant clusterfucktastrophe that makes you question every life decision and formative event that led you to the moment you thought this might be a neat idea. But yeah just like build a React app or whatever and you'll probably be fine.
    - singhrac 2y ago
      Good luck trying to use these tools with anything other than the blessed path of node on the backend. The JS ecosystem absolutely has the problem of "too many compilers", and of course as I'm typing this there's someone out there writing a "one compiler to unify them all".
npalli 2y ago
But to flip this around, let's say you want to use this third party in some other "modern" language how would this become better?. Clearly if someone hadn't done the work to package in vcpkg/conan you can expect it wouldn't exist in pip/cargo/npm either. So if you had a bad time in C++, you would have a dramatically worse time in other languages. Goes to show what everyone else is saying here, there are parts of historic C++ which are in very bad shape particularly the build story. However, if it's in vcpkg it is exactly as easy to install as the other ones.
- kouteiheika 2y ago
  > But to flip this around, let's say you want to use this third party in some other "modern" language how would this become better?. Clearly if someone hadn't done the work to package in vcpkg/conan you can expect it wouldn't exist in pip/cargo/npm either. So if you had a bad time in C++, you would have a dramatically worse time in other languages. Err, no? In case of Rust usually there is really no work that's necessary to package it and upload. You just type `cargo publish` and you're done (after filling out metadata in Cargo.toml). Even if someone wouldn't upload it to crates.io as a Rust crate you can just add a single line to your Cargo.toml: package_name = { git = "https://github.com/user/repo", rev = "revision" } and you're done, you can use it as a dependency.
  - dgfitz 2y ago
    I think you moved the goalposts. I can configure a CMake file to pull a GitHub repo, which seems similar enough.
    - Spivak 2y ago
      I think the difference is that you can pip install a github url and it will work, it will even work if the upstream has done nothing to package it. I think the issue could be broadly defined as how difficult it is how difficult it is to take someone else's code and use it as part of your project. For C/C++ the answer can be completely trivial to a massive pain in the ass.
carom 2y ago
What was particularly difficult about it? Sometimes modifying someone else's code to build statically or on another platform is tricky. Linking is generally just include dir, lib path, and lib though.
- lairv 2y ago
  The library was duckdb (https://duckdb.org/docs/installation/ https://duckdb.org/docs/installation/), and here are some of the issues I ran into: - trying to compile against the prebuilt duckdb_static.so file got me a ton of undefined reference error, I asked about this on their discord and it seems to be a knwon issue, there are additional dependencies that are not part of the release so I had to build it myself - the library uses CMake but it contains a Makefile and they recommend using that to build it - It seems like on windows, if you create a dynamic library you have to add __declspec(dllexport) before each function. DuckDB has a #ifdef DUCKDB_STATIC_BUILD to toggle/disable it which isn't documented anywhere, you have to read the code I'm sure someone will tell me that this is very standard and shouldn't take me 3 days to figure it out, but with basic knowledge of cmake and build systems that's a bit of a pain, and there's no other language where you have to do that
  - jpc0 2y ago
    CMake, the defacto if not standard build system, when correctly setup, does all this for you. Your argument is along the lines of "the maintainer of X rust package didn't setup cargo correctly and it's impossible to include in my project, rust is a horrible language". You can absolutely make an argument that the lack of a standard build system is a pain point, and well it's C++ you get to be a special snowflake and use a makefile instead of CMake but then the onus is on you to provide correct documentation.
    - oconnor663 2y ago
      > the maintainer of X rust package didn't setup cargo correctly and it's impossible to include in my project That basically never happens, which is the point.
    - 2y ago

by lairv 2y ago

77pt77 2y ago

The javascript toolchain for dependencies and build is as arcane as the sloppiest Makefile. It's amazing that we ended up with the javascript equivalente of autotools.

agumonkey 2y ago
yeah, dealing with npm really made me feel like the best part of autotools
dannymcgee 2y ago
Created an account just to say: I am a 10-year JavaScript veteran who knows the ecosystem at least as well as any sane person could possibly be expected to, and this is 100% correct. When I tell people that JS build tooling is the second-worst headache to C/C++ build tooling, they laugh, but stick a toe outside the happy path of `npm create <some-framework-project> && npm start` and you will know what pain feels like. I did get some similar vibes trying to work with Gradle and Maven for a project a while back, but I don't really live in that world so it's hard to say whether my experience was typical or just symptomatic of my inexperience.
- smarkov 2y ago
  And it'll continue to get worse because JS devs love coming up with their own abstractions over everything and piling complexity. You don't write JS, you write TS. You don't write HTML, you write JSX. You don't write CSS, you write Tailwind. And of course, React has its own compiler as of recently. Now you have to figure out in what order to run these parsers, bundle everything together including your hundreds of dependencies, minimize, obfuscate, tree shake, and whatever else fairy dust magic you want to sprinkle on top. Meanwhile, the default build tool of choice changes about once every 2 years, and now you also have a choice of runtime - node vs bun vs deno. Can't wait to see what the next revolutionary idea would be to contribute to this madness.
  - claytongulick 2y ago
    There are those of us out there that have a beautiful experience with JavaScript. Vanilla JS, web components, a lightweight rendering library like lit-html, class props as reactive state... NodeJS + express + vanilla web components is the most graceful and productive stack I've ever used in 30+ years of development.
  - dannymcgee 2y ago
    You forgot to mention the 3 or 4 different TypeScript compilers! I do have to admit at least some of the pain is self-inflicted on my part. I _want_ those nice abstractions, so I will write a damn Rollup or Babel plugin if it means I can use Sass stylesheets in my TypeScript+JSX components that compile to vanilla Custom Elements (so I don't have to depend on a huge framework runtime). But as nice as the Babel/Rollup/Vite plugin APIs are, when you start doing stuff like that you then have to deal with all the deep-in-the-weeds bullshit that comes with it, like the fact that Webpack and Rollup and Node.js all have subtly different (and mostly poorly documented) module resolution rules and that they all differ wildly from the official ES-module spec that the standards committee finally shipped like 20 years too late, so trying to get your unit-testing framework and your bundle toolchain and your browser to all agree about how to digest the mess you've made becomes this giant clusterfucktastrophe that makes you question every life decision and formative event that led you to the moment you thought this might be a neat idea. But yeah just like build a React app or whatever and you'll probably be fine.
  - singhrac 2y ago
    Good luck trying to use these tools with anything other than the blessed path of node on the backend. The JS ecosystem absolutely has the problem of "too many compilers", and of course as I'm typing this there's someone out there writing a "one compiler to unify them all".

npalli 2y ago

But to flip this around, let's say you want to use this third party in some other "modern" language how would this become better?. Clearly if someone hadn't done the work to package in vcpkg/conan you can expect it wouldn't exist in pip/cargo/npm either. So if you had a bad time in C++, you would have a dramatically worse time in other languages. Goes to show what everyone else is saying here, there are parts of historic C++ which are in very bad shape particularly the build story. However, if it's in vcpkg it is exactly as easy to install as the other ones.

kouteiheika 2y ago
> But to flip this around, let's say you want to use this third party in some other "modern" language how would this become better?. Clearly if someone hadn't done the work to package in vcpkg/conan you can expect it wouldn't exist in pip/cargo/npm either. So if you had a bad time in C++, you would have a dramatically worse time in other languages. Err, no? In case of Rust usually there is really no work that's necessary to package it and upload. You just type `cargo publish` and you're done (after filling out metadata in Cargo.toml). Even if someone wouldn't upload it to crates.io as a Rust crate you can just add a single line to your Cargo.toml: package_name = { git = "https://github.com/user/repo", rev = "revision" } and you're done, you can use it as a dependency.
- dgfitz 2y ago
  I think you moved the goalposts. I can configure a CMake file to pull a GitHub repo, which seems similar enough.
  - Spivak 2y ago
    I think the difference is that you can pip install a github url and it will work, it will even work if the upstream has done nothing to package it. I think the issue could be broadly defined as how difficult it is how difficult it is to take someone else's code and use it as part of your project. For C/C++ the answer can be completely trivial to a massive pain in the ass.

carom 2y ago

What was particularly difficult about it? Sometimes modifying someone else's code to build statically or on another platform is tricky. Linking is generally just include dir, lib path, and lib though.

lairv 2y ago
The library was duckdb (https://duckdb.org/docs/installation/ https://duckdb.org/docs/installation/), and here are some of the issues I ran into: - trying to compile against the prebuilt duckdb_static.so file got me a ton of undefined reference error, I asked about this on their discord and it seems to be a knwon issue, there are additional dependencies that are not part of the release so I had to build it myself - the library uses CMake but it contains a Makefile and they recommend using that to build it - It seems like on windows, if you create a dynamic library you have to add __declspec(dllexport) before each function. DuckDB has a #ifdef DUCKDB_STATIC_BUILD to toggle/disable it which isn't documented anywhere, you have to read the code I'm sure someone will tell me that this is very standard and shouldn't take me 3 days to figure it out, but with basic knowledge of cmake and build systems that's a bit of a pain, and there's no other language where you have to do that
- jpc0 2y ago
  CMake, the defacto if not standard build system, when correctly setup, does all this for you. Your argument is along the lines of "the maintainer of X rust package didn't setup cargo correctly and it's impossible to include in my project, rust is a horrible language". You can absolutely make an argument that the lack of a standard build system is a pain point, and well it's C++ you get to be a special snowflake and use a makefile instead of CMake but then the onus is on you to provide correct documentation.
  - oconnor663 2y ago
    > the maintainer of X rust package didn't setup cargo correctly and it's impossible to include in my project That basically never happens, which is the point.
  - 2y ago