4 ms·

I don't believe we can get to thousands of packages by packaging a substantial part ourselves -- we will just get bogged down in this instead of building a dece

by boris 8y ago

I don't believe we can get to thousands of packages by packaging a substantial part ourselves -- we will just get bogged down in this instead of building a decent toolchain. Instead, the "secret" I believe is to make creating, testing, and publishing C/C++ packages as easy and frictionless as possible. This is how you get to thousands of packages and this is what we are trying to do.

izacus 8y ago
Is it at least able to consume CMake projects so we can mix and match until more packages are available?
- boris 8y ago
  Yes, see https://build2.org/build2-toolchain/doc/build2-toolchain-intro.xhtml#guide-unpackaged-deps https://build2.org/build2-toolchain/doc/build2-toolchain-int...
ploxiln 8y ago
This is exactly what linux distributions do. Big ones with lots of contributors package around 10k libraries themselves. Small one-man shows package at least a few hundred common c / c++ libraries. This is also why I'm not terribly interested in these language-ecosystem package managers. Good C libraries, such as I use in my own C projects, keep a stable backwards-compatible interface for years.
- pjmlp 8y ago
  Which is all fine and dandy if one only cares about Linux distributions, the world is bit bigger than that.
- orbifold 8y ago
  Any sufficiently large software project can't rely on the distribution to provide anything for them. The macOS and windows application distribution models are much saner in that regard. The goal should be a self contained, easily reproducible build with minimal external system dependencies. This is why build systems like bazel are so great: They force you to be very explicit about what you depend on and give you a stable and reliable way to reproduce your build once you've specified that.
  - umanwizard 8y ago
    > Any sufficiently large software project can't rely on the distribution to provide anything for them. This is factually wrong. I'm able to compile things as complicated as clang/llvm, firefox, and LibreOffice relying mostly on dependencies installed through the distribution's package manager.
    - exDM69 8y ago
      The distro package manager is there to provide your day to day apps and the libraries to support it. Sooner or later you will run into an issue where your development work will depend on a (version) of a library that will break an app you depend on. Or you will need two versions of a library for your previous and next release which both need maintenance. Nor do they provide reproducible builds and other features that are needed for serious work. Distro package managers are not suited for development and dependency management. They might work most of the time, but that is not good enough.
      - erik_seaberg 8y ago
        The distro package manager is there to offer a menu of compatible known-good dependencies that you can readily adopt. It's a ton of work you don't need to do. If instead you bundle random stuff committed last month, you should have a compelling reason for beginning to maintain a half-baked distro all by yourself.
    - takluyver 8y ago
      Clang, Firefox and LibreOffice are all likely to be in distro packages themselves, so their build dependencies have to be there as well. Building something of similar complexity that isn't already packaged would be a fairer test.
    - orbifold 8y ago
      It is not, just take a look at the third party directory of chromium, where it tracks all its third party dependencies https://github.com/chromium/chromium/tree/master/third_party https://github.com/chromium/chromium/tree/master/third_party the main point of their build system gn (similar to bazel) is to have explicit and distribution independent control of all dependencies (typically tied to specific versions). LLVM basically has no system dependencies as well, instead it implements a lot of platform abstractions within the repository (https://github.com/llvm-mirror/llvm/tree/master/lib/Support https://github.com/llvm-mirror/llvm/tree/master/lib/Support), the remaining dependencies are mostly build dependencies such as python 2.7 and cmake. Same story for Cryengine (https://github.com/CRYTEK/CRYENGINE/tree/release/Code/Libs https://github.com/CRYTEK/CRYENGINE/tree/release/Code/Libs) and Unreal Engine (https://github.com/EpicGames/UnrealEngine/tree/release/Engine/Source/ThirdParty https://github.com/EpicGames/UnrealEngine/tree/release/Engin...). Linux distributions simply duplicate a lot of work because they are based on a flawed and outdated distribution model, outdated because the main argument for shared libraries disappears with > 1 TB commodity hard drives. The approach taken by the projects above is also the only way to do cross platform development, simply because both Windows, macOS and Consoles don't have the concept of an operating system package manager. (If you are using homebrew for development on mac, you will run into a lot of trouble sooner or later).
chacham15 8y ago
I leave this feedback so that hopefully it may be useful to you, but when I was investigating rewriting my build infra a while ago, I looked at build2 and didnt use it for this very reason. I know that I'm one person, but the fact that the gp is the #1 comment tells me that other people had similar thoughts as well. Something that might help kickstart this is if a few of the big packages were up there: openssl, libprotobuf, opencv, google test, etc. If those really big ones were there, that would draw people to add in the other packages that go with those thereby expanding the available packages. But now, since theres nothing, you would have to do everything by yourself in a new system which is daunting for anyone. (Also, by maintaining those packages, you learn what the pains of maintaining the packages are and so as you "get bogged down" you can solve those problems making it easier for others to maintain their own packages)
- boris 8y ago
  Yes, all valid/good points, thanks. We are packaging external libraries that we need ourselves (libmysqlclinet was "fun" -- it is actually a C-library with C++ implementation details). And getting some "critical mass" of packages done by us is also something we are considering.

by boris 8y ago

izacus 8y ago

Is it at least able to consume CMake projects so we can mix and match until more packages are available?

boris 8y ago
Yes, see https://build2.org/build2-toolchain/doc/build2-toolchain-intro.xhtml#guide-unpackaged-deps https://build2.org/build2-toolchain/doc/build2-toolchain-int...

ploxiln 8y ago

This is exactly what linux distributions do. Big ones with lots of contributors package around 10k libraries themselves. Small one-man shows package at least a few hundred common c / c++ libraries. This is also why I'm not terribly interested in these language-ecosystem package managers. Good C libraries, such as I use in my own C projects, keep a stable backwards-compatible interface for years.

pjmlp 8y ago
Which is all fine and dandy if one only cares about Linux distributions, the world is bit bigger than that.
orbifold 8y ago
Any sufficiently large software project can't rely on the distribution to provide anything for them. The macOS and windows application distribution models are much saner in that regard. The goal should be a self contained, easily reproducible build with minimal external system dependencies. This is why build systems like bazel are so great: They force you to be very explicit about what you depend on and give you a stable and reliable way to reproduce your build once you've specified that.
- umanwizard 8y ago
  > Any sufficiently large software project can't rely on the distribution to provide anything for them. This is factually wrong. I'm able to compile things as complicated as clang/llvm, firefox, and LibreOffice relying mostly on dependencies installed through the distribution's package manager.
  - exDM69 8y ago
    The distro package manager is there to provide your day to day apps and the libraries to support it. Sooner or later you will run into an issue where your development work will depend on a (version) of a library that will break an app you depend on. Or you will need two versions of a library for your previous and next release which both need maintenance. Nor do they provide reproducible builds and other features that are needed for serious work. Distro package managers are not suited for development and dependency management. They might work most of the time, but that is not good enough.
    - erik_seaberg 8y ago
      The distro package manager is there to offer a menu of compatible known-good dependencies that you can readily adopt. It's a ton of work you don't need to do. If instead you bundle random stuff committed last month, you should have a compelling reason for beginning to maintain a half-baked distro all by yourself.
  - takluyver 8y ago
    Clang, Firefox and LibreOffice are all likely to be in distro packages themselves, so their build dependencies have to be there as well. Building something of similar complexity that isn't already packaged would be a fairer test.
  - orbifold 8y ago
    It is not, just take a look at the third party directory of chromium, where it tracks all its third party dependencies https://github.com/chromium/chromium/tree/master/third_party https://github.com/chromium/chromium/tree/master/third_party the main point of their build system gn (similar to bazel) is to have explicit and distribution independent control of all dependencies (typically tied to specific versions). LLVM basically has no system dependencies as well, instead it implements a lot of platform abstractions within the repository (https://github.com/llvm-mirror/llvm/tree/master/lib/Support https://github.com/llvm-mirror/llvm/tree/master/lib/Support), the remaining dependencies are mostly build dependencies such as python 2.7 and cmake. Same story for Cryengine (https://github.com/CRYTEK/CRYENGINE/tree/release/Code/Libs https://github.com/CRYTEK/CRYENGINE/tree/release/Code/Libs) and Unreal Engine (https://github.com/EpicGames/UnrealEngine/tree/release/Engine/Source/ThirdParty https://github.com/EpicGames/UnrealEngine/tree/release/Engin...). Linux distributions simply duplicate a lot of work because they are based on a flawed and outdated distribution model, outdated because the main argument for shared libraries disappears with > 1 TB commodity hard drives. The approach taken by the projects above is also the only way to do cross platform development, simply because both Windows, macOS and Consoles don't have the concept of an operating system package manager. (If you are using homebrew for development on mac, you will run into a lot of trouble sooner or later).

chacham15 8y ago

I leave this feedback so that hopefully it may be useful to you, but when I was investigating rewriting my build infra a while ago, I looked at build2 and didnt use it for this very reason. I know that I'm one person, but the fact that the gp is the #1 comment tells me that other people had similar thoughts as well. Something that might help kickstart this is if a few of the big packages were up there: openssl, libprotobuf, opencv, google test, etc. If those really big ones were there, that would draw people to add in the other packages that go with those thereby expanding the available packages. But now, since theres nothing, you would have to do everything by yourself in a new system which is daunting for anyone. (Also, by maintaining those packages, you learn what the pains of maintaining the packages are and so as you "get bogged down" you can solve those problems making it easier for others to maintain their own packages)

boris 8y ago
Yes, all valid/good points, thanks. We are packaging external libraries that we need ourselves (libmysqlclinet was "fun" -- it is actually a C-library with C++ implementation details). And getting some "critical mass" of packages done by us is also something we are considering.