6 ms·

Maybe it's something about being a C++ Dev that lends your output to extra verbosity, but I feel like this reads like a recipe on a blogspam site and I had to s

Maybe it's something about being a C++ Dev that lends your output to extra verbosity, but I feel like this reads like a recipe on a blogspam site and I had to scroll down really far to get to the point. And the point is, he's making npm/maven for C++. Did C++ not already have a fully featured package manager until now? Why not?

tastroder 7y ago
For those wondering, the content seems to start about halfway down: https://vector-of-bool.github.io/2020/01/06/new-decade.html#a-new-tool https://vector-of-bool.github.io/2020/01/06/new-decade.html#... The docs similarly don't seem to include a concise explanation but at least are a bit more readable: https://vector-of-bool.github.io/docs/dds/guide/packages.html https://vector-of-bool.github.io/docs/dds/guide/packages.htm... I don't really see it either, my typical C++ project is okay with the library management the different *make tools offer. But C++ is not my go to language these days, I'm usually rather careful on which dependencies take on in those projects and where they come from, not really sure a npm/maven dependency model is something I would want there.
usrnm 7y ago
> Did C++ not already have a fully featured package manager until now? There are several competing projects, but no clear winner yet. And a lot of people a companies prefer to build their stuff themselves for additional control. > Why not? It's complicated. In a few words, it's practically impossible to build a general tool that could suit everybody's needs.
- capableweb 7y ago
  > it's practically impossible to build a general tool that could suit everybody's needs. Indeed and that's not what npm or maven aims for either, no tool fits everybody's need, not even npm! Probably there is a deeper issue than "it's hard", or if it's "just hard", why is so much harder for C++ than for the JavaScript folks?
  - usrnm 7y ago
    There is no compilation step in the js world, or any other interpreted language for that matter. Even in case of JIT, it happens completely transparently and has no effect on the package you download from npm. Nobody distributes JIT'ed code. C++, on the other hand, doesn't work like that. Source code needs to be compiled to be usable and the exact way it is supposed to be compiled varies enormously between different projects. Not to mention that there are usually lots of compiler options and project-specific macro definitions that you might want to change to get the exact binary you need. Compilation matters, it matters a lot, and it cannot be easily abstracted away without sacrifising performance and flexibility.
    - yunyu 7y ago
      Plenty of npm/pypi modules require building platform-specific native code (see node-gyp, wheel), and a lot of JS packages require transpilation. I don't find this argument particularly compelling.
  - shakna 7y ago
    > why is so much harder for C++ than for the JavaScript folks? There are C++ package managers that exist and work - they tend to be your OS' package manager, because of the problems they will have to work around. Because C++ has a lot more flexibility in core components than JavaScript does, which aren't easy to resolve at the package manager level. Things projects may _require_ to build correctly: * Multiple compilers that don't always have compatible extensions that sometimes share the same flags, and can falsely report to be each other when queried. * Multiple simultaneous differing versions of clib to link against. They may requiring differing specific versions of specific implementations. * Specific kernel libraries to link against. You can't easily build and link against a specific OS kernel in isolation. * Projects may expect specific paths, and ignore or break attempts to change these. Not being able to compile a project that has a history of twenty years isn't an option. * Arbitrary execution with intentional side-effects. It's bad, but there is a very long history of projects that you're dealing with here. * CPATH and CXXPATH are globals. Relative path linking is possible, but difficult and a pile of hacks. You can force a node_modules style to exist, and some projects do, but most are a mix of relative and global, and the global can override your local and break things.
jasode 7y ago
>, he's making npm/maven for C++. Well, based on what I read, he's not actually making the equivalent of npm. Instead, he's proposing a new tool ("dds") and new syntax of config files to integrate build and package management. This distinction is key because the "package manager" for javascript implies npm the the tool & package specification but also a public repository ("https://www.npmjs.com" https://www.npmjs.com") that is funded by the Nodejs/NPM Inc entity. Likewise, Rust's package manager points to "crates.io" repository which I believe is funded by Mozilla. This blog article is not talking about making a C++ repository like "npmjs.com" and "crates.io". Only a syntax specification. Arguably, the destination repository (server) is more important than the command-line tool (client) because it's easier for a funded website with diskspace+bandwidth to justify the existence of a tool rather than the other way around. > Did C++ not already have a fully featured package manager until now? Why not? It's a seemingly simple question but it the "answer" which hasn't happened yet is complex and has multiple parts. The 2 big parts are (1) social inertia and (2) technical complexity (1) social inertia: C++ created ~1986 has 2+ decades of usage before the idea of "let's have a process _automatically_ reach out to the internet to download dependencies" became popular. Thus, there is no well-known repository that became a Schelling Point[1] for C++ programmers to upload their C++ libraries. Consider popular C/C++ libraries like SQLite, Zlib, OpenSSL, etc. Why don't those authors upload their libraries to a package manager repository? Because nobody made an influential website destination for it that everybody adopted. E.g. If Google Inc funded a C++ repo with disk space and bandwidth, would corporate politics prevent Microsoft and Apple from uploading their code to it? In contrast, the Javascript quickly adopted NPM that was funded by Nodejs and it was the already default baked into Nodejs downloads. Which entity in the C++ world is analogous to NPM inc to fund a repository that Google/MS/Apple/Adobe/etc would all adopt? It needs to be an entity that cuts across all those competitors. Maybe Intel? Or maybe Epic Games (they make Unreal Engine)? Or all the big competitors agree to contribute to a non-profit entity? Nobody has stepped up to the plate like NPM Inc did. (2) technical challenges: C++ has a compile and link steps that interpreted languages like Javascript don't have. In npm, the assets are "text" (i.e. .js files like leftpad()). In C++, many library publishers (proprietary) don't offer source code (the "text") and only binary precompiled .lib files. These libraries are binary blobs that are incompatible across cpu architectures and compilers (Intel vs ARM, Microsoft MSVC vs gcc, etc). Because C++ is low-level, there's also a different set of build configs such as 32bit vs 64bit, static link vs dynamic link, and DEBUG vs RELEASE, etc. This means C++ needs to implement a much more complicated "dependency" syntax way beyond just "versions" and nobody has created that specification that everybody has agreed to use. Since Javascript doesn't have separate compile & link phases, something like "npm" is much easier to create. If Rust also has "compile & link" step, how did they successfully get people to adopt the "cargo" tool and the "crates.io" repository? Probably because Rust's ecosystem of programmers has more emphasis on shared "open source code" than the C++ programmers of the 1980s and 1990s. Also, we emphasize again that Rust's patron (Mozilla) also funded "crates.io" and Rust programmers just adopted it so the "cargo" tool has an http destination to point to download artifacts. There's no equivalent influential entity in C++ yet. In terms of package spec complexity, I haven't studied Rust's TOML configuration spec to know if it's flexible enough to handle all of the different ways that C++ practitioners build files. A lot of C++ build flexibility is put into "./configure.sh" scripts (e.g. ffmpeg, Qt, etc) so I don't know if toml spec is a superset of all that. [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)
pjc50 7y ago
> Did C++ not already have a fully featured package manager until now? Why not? No, it doesn't. I'd say it was due to age and fragmentation; most of the more recent languages have, in effect, a single "vendor" who can coordinate something like that. Whereas C++ has historically one big proprietary and one open vendor who fought each other (Microsoft vs. GNU), plus a long tail of proprietary implementations (e.g. Intel); more recently there is clang. The language itself doesn't have packages. In practice, the role taken by a package manager in the Free world was occupied by autoconf, which does a good job of building across a huge variety of systems but is a pain to use. There was also much more of a tradition of "this is what you're getting, deal with it": rather than a program specifying precise version requirements, autoconf probes the capabilities and then adapts accordingly. Build system in the Free world is make, with occasional jam if you're a big boost user. Microsoft of course do something completely different with MSBuild. Visual studio does have an integrated package manager, nuget, but it's orientated at C# use.
- brandmeyer 7y ago
  > No, it doesn't. I'd say it was due to age and fragmentation; most of the more recent languages have, in effect, a single "vendor" who can coordinate something like that. This is a widely held meme, but I think its flat wrong. The reality is that most C++ project's have a very wide spectrum of types of dependencies and build steps. The history of competing vendors certainly adds to the non-homogeneity, but it isn't the only portion. Even if all of our C and C++ dependencies were built and installed by a common package manager, we would still have dependencies on a variety of packages in Fortran, Python, Bash, Tcl, Verilator, NGSPICE, Octave, and so on. Running the compiler and linker represents only a handful of the many steps in the process. C++'s niche lies at an intersection of performance and complexity. If the project wasn't demanding enough for C++, folks wouldn't pick it in the first place.
- zamalek 7y ago
  > but it's orientated at C# use. It's oriented for MSBuild. The package creation tooling is aimed at .Net. It's certainly possible to create high quality C++ Nuget packages (that are also huge because you have to provide multiple binaries).

tastroder 7y ago

For those wondering, the content seems to start about halfway down: https://vector-of-bool.github.io/2020/01/06/new-decade.html#a-new-tool https://vector-of-bool.github.io/2020/01/06/new-decade.html#... The docs similarly don't seem to include a concise explanation but at least are a bit more readable: https://vector-of-bool.github.io/docs/dds/guide/packages.html https://vector-of-bool.github.io/docs/dds/guide/packages.htm... I don't really see it either, my typical C++ project is okay with the library management the different *make tools offer. But C++ is not my go to language these days, I'm usually rather careful on which dependencies take on in those projects and where they come from, not really sure a npm/maven dependency model is something I would want there.

usrnm 7y ago

> Did C++ not already have a fully featured package manager until now? There are several competing projects, but no clear winner yet. And a lot of people a companies prefer to build their stuff themselves for additional control. > Why not? It's complicated. In a few words, it's practically impossible to build a general tool that could suit everybody's needs.

capableweb 7y ago
> it's practically impossible to build a general tool that could suit everybody's needs. Indeed and that's not what npm or maven aims for either, no tool fits everybody's need, not even npm! Probably there is a deeper issue than "it's hard", or if it's "just hard", why is so much harder for C++ than for the JavaScript folks?
- usrnm 7y ago
  There is no compilation step in the js world, or any other interpreted language for that matter. Even in case of JIT, it happens completely transparently and has no effect on the package you download from npm. Nobody distributes JIT'ed code. C++, on the other hand, doesn't work like that. Source code needs to be compiled to be usable and the exact way it is supposed to be compiled varies enormously between different projects. Not to mention that there are usually lots of compiler options and project-specific macro definitions that you might want to change to get the exact binary you need. Compilation matters, it matters a lot, and it cannot be easily abstracted away without sacrifising performance and flexibility.
  - yunyu 7y ago
    Plenty of npm/pypi modules require building platform-specific native code (see node-gyp, wheel), and a lot of JS packages require transpilation. I don't find this argument particularly compelling.
- shakna 7y ago
  > why is so much harder for C++ than for the JavaScript folks? There are C++ package managers that exist and work - they tend to be your OS' package manager, because of the problems they will have to work around. Because C++ has a lot more flexibility in core components than JavaScript does, which aren't easy to resolve at the package manager level. Things projects may _require_ to build correctly: * Multiple compilers that don't always have compatible extensions that sometimes share the same flags, and can falsely report to be each other when queried. * Multiple simultaneous differing versions of clib to link against. They may requiring differing specific versions of specific implementations. * Specific kernel libraries to link against. You can't easily build and link against a specific OS kernel in isolation. * Projects may expect specific paths, and ignore or break attempts to change these. Not being able to compile a project that has a history of twenty years isn't an option. * Arbitrary execution with intentional side-effects. It's bad, but there is a very long history of projects that you're dealing with here. * CPATH and CXXPATH are globals. Relative path linking is possible, but difficult and a pile of hacks. You can force a node_modules style to exist, and some projects do, but most are a mix of relative and global, and the global can override your local and break things.

jasode 7y ago

>, he's making npm/maven for C++. Well, based on what I read, he's not actually making the equivalent of npm. Instead, he's proposing a new tool ("dds") and new syntax of config files to integrate build and package management. This distinction is key because the "package manager" for javascript implies npm the the tool & package specification but also a public repository ("https://www.npmjs.com" https://www.npmjs.com") that is funded by the Nodejs/NPM Inc entity. Likewise, Rust's package manager points to "crates.io" repository which I believe is funded by Mozilla. This blog article is not talking about making a C++ repository like "npmjs.com" and "crates.io". Only a syntax specification. Arguably, the destination repository (server) is more important than the command-line tool (client) because it's easier for a funded website with diskspace+bandwidth to justify the existence of a tool rather than the other way around. > Did C++ not already have a fully featured package manager until now? Why not? It's a seemingly simple question but it the "answer" which hasn't happened yet is complex and has multiple parts. The 2 big parts are (1) social inertia and (2) technical complexity (1) social inertia: C++ created ~1986 has 2+ decades of usage before the idea of "let's have a process _automatically_ reach out to the internet to download dependencies" became popular. Thus, there is no well-known repository that became a Schelling Point[1] for C++ programmers to upload their C++ libraries. Consider popular C/C++ libraries like SQLite, Zlib, OpenSSL, etc. Why don't those authors upload their libraries to a package manager repository? Because nobody made an influential website destination for it that everybody adopted. E.g. If Google Inc funded a C++ repo with disk space and bandwidth, would corporate politics prevent Microsoft and Apple from uploading their code to it? In contrast, the Javascript quickly adopted NPM that was funded by Nodejs and it was the already default baked into Nodejs downloads. Which entity in the C++ world is analogous to NPM inc to fund a repository that Google/MS/Apple/Adobe/etc would all adopt? It needs to be an entity that cuts across all those competitors. Maybe Intel? Or maybe Epic Games (they make Unreal Engine)? Or all the big competitors agree to contribute to a non-profit entity? Nobody has stepped up to the plate like NPM Inc did. (2) technical challenges: C++ has a compile and link steps that interpreted languages like Javascript don't have. In npm, the assets are "text" (i.e. .js files like leftpad()). In C++, many library publishers (proprietary) don't offer source code (the "text") and only binary precompiled .lib files. These libraries are binary blobs that are incompatible across cpu architectures and compilers (Intel vs ARM, Microsoft MSVC vs gcc, etc). Because C++ is low-level, there's also a different set of build configs such as 32bit vs 64bit, static link vs dynamic link, and DEBUG vs RELEASE, etc. This means C++ needs to implement a much more complicated "dependency" syntax way beyond just "versions" and nobody has created that specification that everybody has agreed to use. Since Javascript doesn't have separate compile & link phases, something like "npm" is much easier to create. If Rust also has "compile & link" step, how did they successfully get people to adopt the "cargo" tool and the "crates.io" repository? Probably because Rust's ecosystem of programmers has more emphasis on shared "open source code" than the C++ programmers of the 1980s and 1990s. Also, we emphasize again that Rust's patron (Mozilla) also funded "crates.io" and Rust programmers just adopted it so the "cargo" tool has an http destination to point to download artifacts. There's no equivalent influential entity in C++ yet. In terms of package spec complexity, I haven't studied Rust's TOML configuration spec to know if it's flexible enough to handle all of the different ways that C++ practitioners build files. A lot of C++ build flexibility is put into "./configure.sh" scripts (e.g. ffmpeg, Qt, etc) so I don't know if toml spec is a superset of all that. [1] https://en.wikipedia.org/wiki/Focal_point_(game_theory) https://en.wikipedia.org/wiki/Focal_point_(game_theory)

pjc50 7y ago

> Did C++ not already have a fully featured package manager until now? Why not? No, it doesn't. I'd say it was due to age and fragmentation; most of the more recent languages have, in effect, a single "vendor" who can coordinate something like that. Whereas C++ has historically one big proprietary and one open vendor who fought each other (Microsoft vs. GNU), plus a long tail of proprietary implementations (e.g. Intel); more recently there is clang. The language itself doesn't have packages. In practice, the role taken by a package manager in the Free world was occupied by autoconf, which does a good job of building across a huge variety of systems but is a pain to use. There was also much more of a tradition of "this is what you're getting, deal with it": rather than a program specifying precise version requirements, autoconf probes the capabilities and then adapts accordingly. Build system in the Free world is make, with occasional jam if you're a big boost user. Microsoft of course do something completely different with MSBuild. Visual studio does have an integrated package manager, nuget, but it's orientated at C# use.

brandmeyer 7y ago
> No, it doesn't. I'd say it was due to age and fragmentation; most of the more recent languages have, in effect, a single "vendor" who can coordinate something like that. This is a widely held meme, but I think its flat wrong. The reality is that most C++ project's have a very wide spectrum of types of dependencies and build steps. The history of competing vendors certainly adds to the non-homogeneity, but it isn't the only portion. Even if all of our C and C++ dependencies were built and installed by a common package manager, we would still have dependencies on a variety of packages in Fortran, Python, Bash, Tcl, Verilator, NGSPICE, Octave, and so on. Running the compiler and linker represents only a handful of the many steps in the process. C++'s niche lies at an intersection of performance and complexity. If the project wasn't demanding enough for C++, folks wouldn't pick it in the first place.
zamalek 7y ago
> but it's orientated at C# use. It's oriented for MSBuild. The package creation tooling is aimed at .Net. It's certainly possible to create high quality C++ Nuget packages (that are also huge because you have to provide multiple binaries).