19 ms·

Clib: Package manager for C

83 points by petercooper 13y ago

oofabz 13y ago
I love this if only because it highlights that package management is a hard problem, still unsolved in even the most mature languages. Package management is something that sounds really easy on the surface but once you get into it you are overwhelmed with details, tradeoffs, and drudgery. If we're ever going to solve it, we first need to acknowledge that it's a very difficult problem.
cmbaus 13y ago
This is a good idea. I could see this changing C development for the better.
pmahoney 13y ago
Nice, but can anything challenge the "every language gets its own unique package manager"? (And own unique build system for that matter).
- andrewl-hn 13y ago
  James Halliday a while ago wrote a package system for C based on npm - a node package system: https://github.com/substack/dotc#dotc https://github.com/substack/dotc#dotc The machinery behind it is surprisingly simple. You have 2 pre-processor commands: #require and #exports and you use them to build up a hierarchy of dependencies.
  - e12e 13y ago
    Interesting. Not sure how I feel about the requirment to use c++ features for a c package manager though...
- cleverjake 13y ago
  its a pain, but realistically, the needs of each community are often best served by that community.
- yxhuvud 13y ago
  While it would be very nice if one of the existing language package managers stepped up their game by supporting multiple languages and cross-language dependencies, one still has to remember that these fulfill a need - namely to make it easy to share code within a coding community. This is something distro package systems have failed to achieve - their systems have too much bureucratic overhead to make a package available for all and require that a library to be packaged multiple times to be used in more than one distro.
  - zimbatm 13y ago
    It's because we need a user-space package manager. System package managers require root access to be handled and tend to provide only a single stable version of each package. While they are good to provide the base system, on the application level it's desirable to use specific version of libraries.
    - Hello71 13y ago
      > It's because we need a user-space package manager. that word doesn't mean what you think it means > System package managers require root access to be handled nope, Gentoo Prefix can be installed by anyone, as long as you have somewhere that's u+w and exec. > and tend to provide only a single stable version of each package. nope, Gentoo allows any versions of every package to be mixed and matched, as long as the underlying platform and package supports it.
      - zimbatm 13y ago
        Is Gentoo Prefix installable on other platforms ? If you want to match the language-specific package managers you also have to be able to support at least the common POSIX platforms and allow per-project dependencies management.
    - Svenstaro 13y ago
      Shouldn't software optimally always use the newest version of all the libraries it depends on? It's a security issue if they don't. And as far as I know, only Linux distro package repositories even have backports, not application package repos.
- mitchty 13y ago
  I honestly find pkg-config to be the most sane "c package manager" As long as a ./configure && make && make install installs package config files all I need to do is set PACKAGE_CONFIG_PATH and be done with it. I don't think c needs its own package manager, or at least another one.
  - jevinskie 13y ago
    I really like pkg-config but it has some issues with static linking. I had to create a GCC wrapper script to properly build and link the libimobiledevice suite into 100% static executables. pkg-config config files have fields to handle transitive shared linking but not one for static linking. P.S. Apologies for the accidental downvote. I compensated by up voting another comment.
    - mitchty 13y ago
      Yeah I love pkg-config and static linking is annoying. Generally what I do to get around that is in configure.ac something like: libuvdir=`pkg-config --variable=libdir libuv` LIBS="$LIBS $libuvdir/libuv.a" That tends to work well enough for my needs. I gate it with a --build-static flag so that I can run off shared if I want to or not. I'd rather take pkg-config+autotools over any of these newfangled options. Going to take a lot to sway me over from a tried and true option across osx/linux/solaris/eh fine hpux too but just cause i'm in a good mood.
  - deleted 13y ago
    [deleted]
- patmcguire 13y ago
  It's not too hard to make a meta-manager, is it? Wrap the ten or so people use in a script, have dependencies like "clib": { // whatever }, "gem": { // whatever } So you could do something like install GSL in the C package and then Ruby GSL in your gems. Saves the trouble if installing things to bind to, lets the things manage what they manage best. Install for the meta-manager would install the relevant package managers, then you've got it all.
  - StefanKarpinski 13y ago
    I think you significantly underestimate how difficult a problem correctly installing and configuring software is. Even just specifying and resolving the dependencies and relationships between libraries and packages is quite hard.
    - chubot 13y ago
      One thing I didn't realize until recently is that it's not just "hard" engineering-wise, it's hard theoretically, i.e. NP-hard. With version constraints, package managers actually have to solve a problem equivalent to SAT solving. All the "practical" package managers rely on relatively terrifying hacks and heuristics to get around this. I have been looking through Debian metadata, and I am very surprised at how unsound it is. The metadata is very tightly coupled to apt-get ("virtual" package handling, etc.) Not to mention that all the algorithms basically only work in one direction (going forward, not backward). It appears that some academic/niche package managers are incorporating SAT solvers. "Managing the complexity of large free and open source package-based software distributions" http://scholar.google.com/scholar?cluster=16249102547364024825&hl=en&as_sdt=0,5 http://scholar.google.com/scholar?cluster=162491025473640248... More references: http://www.mancoosi.org/edos/references/ http://www.mancoosi.org/edos/references/ Anyway the root problem is that C headers and other language-specific interfaces are not always a good way to interface software components... developing more stable binary protocols would greatly ameliorate the versioning problem.
      - StefanKarpinski 12y ago
- StefanKarpinski 13y ago
  There is a natural break between system-level package management and language-level package management. If you install language-level packages with system-level package managers, then each language package needs to support every single system-level package manager. E.g. you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc. This is a completely ridiculous situation to put language developers in, especially since most language-level packages don't care what system they run on. Instead, the best practice seems to be to install programming languages with the system-level package manager and install the language's packages with the language-level package manager. That way you only need one system-level package per language, and each language-level package only needs to deal with that language's package manager. The system package manager is responsible for dealing with system-specific things while the language package manager is responsible for language-specific things. It's a natural division. It is conceivable that there could be a single cross-language, language-level package manager. But what language would it be written in? And how would it deal with the fact that each language uses different technologies and has different approaches for installing and configuring libraries and packages? When you think about it for a bit, I don't think it's so insensible for each language to have its own package manager.
  - Hello71 13y ago
    > you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc wrong. that's the package maintainer's job. moreover, sensible package management systems have templates or libraries (eclasses in Gentoo) to properly handle all kinds of packages. all we ask for is that developers don't actively break package management by bundling everything as is done here.
    - voltagex_ 13y ago
      Ir seems like there is often a breakdown in communication between upstream and distro package maintainers. It may also be that authors don't have time to talk with ~3 (Debian, Fedora, ???) distros, each with different packaging processes and rules.
    - drewcrawford 13y ago
- mmerickel 13y ago
  nix (http://nixos.org/nix/ http://nixos.org/nix/) seems to be the solution with the most potential in this area, solving all dependencies at the project level and removing the need for language specific build environments.
  - Nullabillity 13y ago
    Also, per-user installs sweetens the deal even further.
  - pmahoney 13y ago
    I've actually been exploring nix myself recently. It has some interesting features. I'm finding it quite nice to maintain some C-based packages across both Linux and Mac OS X (where it is somewhat of a competitor to Homebrew). I've been trying to get it to manage some Ruby applications, complete with gems, private github gems, etc. This has been difficult, in part because some areas of nix are a bit rough, and in part because Rubygems has incomplete dependency information (i.e. there's no standard way to know that, say typhoeus gem depends on C library libcurl, and I'm still unclear on how/if dynamic loading can be made to load the nix packaged libcurl). Something like nix is what I had in mind with my original question. I'd like to see new languages at least design their custom package managers with these more generic systems in mind. Rubygems has some quirks, and Bundler just includes everything and the kitchen sink (sourcing gems from local files, github, various other features) that complicate matters. Nix is good for keeping things more or less isolated and tracking complete dependencies, though an alternate way to accomplish similar is something like Docker and then just scripting all the native package managers...
null_ptr 13y ago
I like this! Is it a sort of staging ground for libraries to eventually graduate to various distributions' official repositories, or is it just random bite-sized utilities for quick weekend projects?
Sir_Cmpwn 13y ago
What ever happened to "apt-get" as the C package manager?
- JetSpiegel 13y ago
  $ apt-get bash: apt-get: command not found
- pjmlp 13y ago
  Doesn't work on AS/400, Windows, VxWorks, QNX, ...
  - panzi 13y ago
    And Fedora Linux.
gxc40 13y ago
There is some overlap with CCAN (http://ccodearchive.net/ http://ccodearchive.net/).
chj 13y ago
Can be a big time saver. For any package manager to work painlessly and to scale, the repository must be able to detect name conflicts on the code level.
giovannibajo1 13y ago
I think this is missing half of the problem, that is connecting the package with the build system. I know there is no "standard" build system for C, but still. Go for instance does both; once you "go get" a package, it not only downloads and pre-builds it, but makes sure that it is available for transparent usage in your program. I personally bias towards CMake, so I would appreciate a C package manager which also makes sure that the package is included in my CMakeList.txt somehow.
- shoo 13y ago
  what if these are in fact two separate problems? then perhaps it is simpler to solve them with separate tools instead of conflating them.
  - giovannibajo1 13y ago
    I can't see how it's simpler, because then you need to write N*M bridges/connectors/plugins to let N package managers interact with M build systems. And what if the library you download is meant to be built with a build system which is different from the one you chose for your application? What if you then want to globally turn on debugging or change a compilation flag, or a preprocessor define to enable/disable a feature? I don't know of any language-specific package manager that doesn't also make the package immediately available to the programming environment without further fiddling. Solving half of the problem... well, it's a half solution :)
michaelmior 13y ago
It's a little scary for me that the source of available package is a wiki page which is editable by anyone with a GitHub account.
- lcampbell 13y ago
  Not only that, but there appear to be potential buffer overflows which involve the data pulled from the wiki page[1]. I'm not sure why asprintf wasn't used here (and in other places). The partial re-implementation of libc[2, 3] is also somewhat frightening. As are potential unchecked integer overflows when calculating a size to pass to malloc[4]. C is hard. -- [1] https://github.com/clibs/clib/blob/master/src/clib-install.c#L86 https://github.com/clibs/clib/blob/master/src/clib-install.c... [2] https://github.com/clibs/clib/blob/master/deps/fs/fs.c https://github.com/clibs/clib/blob/master/deps/fs/fs.c [3] https://github.com/clibs/clib/blob/master/deps/str-copy/str-copy.c https://github.com/clibs/clib/blob/master/deps/str-copy/str-... [4] https://github.com/clibs/clib/blob/master/deps/http-get/http-get.c#L19 https://github.com/clibs/clib/blob/master/deps/http-get/http...
  - gone35 12y ago
    Oh dear... I'm just sight-linting so this might be wrong, but I'm certain the function executable in src/clib-install.c [1] can be easily overflown by providing a maliciously long package name or version --neither of which are validated at all [2] despite coming straight from the wiki page, as you point out. The function allocates 256-byte buffers to store a url and file name, respectively, but blindly fills them up using sprintf with non-validated user-supplied data, and without checking the resulting number of characters written, eg: char *url = malloc(256); if(NULL == file) goto e1; sprintf(file,"%s-%s.tar.gz",pkg->name,pkg->version); C is unforgiving indeed. Also what's up with all those (0 == count) yoda conditionals all over? Is that becoming a thing again? [1] https://github.com/clibs/clib/blob/master/src/clib-install.c https://github.com/clibs/clib/blob/master/src/clib-install.c [2] As far as I can tell, the package info from the wiki is dealt with by functions in deps/clib-package.c that, despite promising names like json_object_get_string_safe, do not ever validate the provided name or version string: https://github.com/clibs/clib/blob/master/deps/clib-package/clib-package.c https://github.com/clibs/clib/blob/master/deps/clib-package/...

83 points by petercooper 13y ago

I love this if only because it highlights that package management is a hard problem, still unsolved in even the most mature languages. Package management is something that sounds really easy on the surface but once you get into it you are overwhelmed with details, tradeoffs, and drudgery. If we're ever going to solve it, we first need to acknowledge that it's a very difficult problem.

cmbaus 13y ago

This is a good idea. I could see this changing C development for the better.

pmahoney 13y ago

Nice, but can anything challenge the "every language gets its own unique package manager"? (And own unique build system for that matter).

andrewl-hn 13y ago
James Halliday a while ago wrote a package system for C based on npm - a node package system: https://github.com/substack/dotc#dotc https://github.com/substack/dotc#dotc The machinery behind it is surprisingly simple. You have 2 pre-processor commands: #require and #exports and you use them to build up a hierarchy of dependencies.
- e12e 13y ago
  Interesting. Not sure how I feel about the requirment to use c++ features for a c package manager though...
cleverjake 13y ago
its a pain, but realistically, the needs of each community are often best served by that community.
yxhuvud 13y ago
While it would be very nice if one of the existing language package managers stepped up their game by supporting multiple languages and cross-language dependencies, one still has to remember that these fulfill a need - namely to make it easy to share code within a coding community. This is something distro package systems have failed to achieve - their systems have too much bureucratic overhead to make a package available for all and require that a library to be packaged multiple times to be used in more than one distro.
- zimbatm 13y ago
  It's because we need a user-space package manager. System package managers require root access to be handled and tend to provide only a single stable version of each package. While they are good to provide the base system, on the application level it's desirable to use specific version of libraries.
  - Hello71 13y ago
    > It's because we need a user-space package manager. that word doesn't mean what you think it means > System package managers require root access to be handled nope, Gentoo Prefix can be installed by anyone, as long as you have somewhere that's u+w and exec. > and tend to provide only a single stable version of each package. nope, Gentoo allows any versions of every package to be mixed and matched, as long as the underlying platform and package supports it.
    - zimbatm 13y ago
      Is Gentoo Prefix installable on other platforms ? If you want to match the language-specific package managers you also have to be able to support at least the common POSIX platforms and allow per-project dependencies management.
  - Svenstaro 13y ago
    Shouldn't software optimally always use the newest version of all the libraries it depends on? It's a security issue if they don't. And as far as I know, only Linux distro package repositories even have backports, not application package repos.
mitchty 13y ago
I honestly find pkg-config to be the most sane "c package manager" As long as a ./configure && make && make install installs package config files all I need to do is set PACKAGE_CONFIG_PATH and be done with it. I don't think c needs its own package manager, or at least another one.
- jevinskie 13y ago
  I really like pkg-config but it has some issues with static linking. I had to create a GCC wrapper script to properly build and link the libimobiledevice suite into 100% static executables. pkg-config config files have fields to handle transitive shared linking but not one for static linking. P.S. Apologies for the accidental downvote. I compensated by up voting another comment.
  - mitchty 13y ago
    Yeah I love pkg-config and static linking is annoying. Generally what I do to get around that is in configure.ac something like: libuvdir=`pkg-config --variable=libdir libuv` LIBS="$LIBS $libuvdir/libuv.a" That tends to work well enough for my needs. I gate it with a --build-static flag so that I can run off shared if I want to or not. I'd rather take pkg-config+autotools over any of these newfangled options. Going to take a lot to sway me over from a tried and true option across osx/linux/solaris/eh fine hpux too but just cause i'm in a good mood.
- deleted 13y ago
  [deleted]
patmcguire 13y ago
It's not too hard to make a meta-manager, is it? Wrap the ten or so people use in a script, have dependencies like "clib": { // whatever }, "gem": { // whatever } So you could do something like install GSL in the C package and then Ruby GSL in your gems. Saves the trouble if installing things to bind to, lets the things manage what they manage best. Install for the meta-manager would install the relevant package managers, then you've got it all.
- StefanKarpinski 13y ago
  I think you significantly underestimate how difficult a problem correctly installing and configuring software is. Even just specifying and resolving the dependencies and relationships between libraries and packages is quite hard.
  - chubot 13y ago
    One thing I didn't realize until recently is that it's not just "hard" engineering-wise, it's hard theoretically, i.e. NP-hard. With version constraints, package managers actually have to solve a problem equivalent to SAT solving. All the "practical" package managers rely on relatively terrifying hacks and heuristics to get around this. I have been looking through Debian metadata, and I am very surprised at how unsound it is. The metadata is very tightly coupled to apt-get ("virtual" package handling, etc.) Not to mention that all the algorithms basically only work in one direction (going forward, not backward). It appears that some academic/niche package managers are incorporating SAT solvers. "Managing the complexity of large free and open source package-based software distributions" http://scholar.google.com/scholar?cluster=16249102547364024825&hl=en&as_sdt=0,5 http://scholar.google.com/scholar?cluster=162491025473640248... More references: http://www.mancoosi.org/edos/references/ http://www.mancoosi.org/edos/references/ Anyway the root problem is that C headers and other language-specific interfaces are not always a good way to interface software components... developing more stable binary protocols would greatly ameliorate the versioning problem.
    - StefanKarpinski 12y ago
StefanKarpinski 13y ago
There is a natural break between system-level package management and language-level package management. If you install language-level packages with system-level package managers, then each language package needs to support every single system-level package manager. E.g. you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc. This is a completely ridiculous situation to put language developers in, especially since most language-level packages don't care what system they run on. Instead, the best practice seems to be to install programming languages with the system-level package manager and install the language's packages with the language-level package manager. That way you only need one system-level package per language, and each language-level package only needs to deal with that language's package manager. The system package manager is responsible for dealing with system-specific things while the language package manager is responsible for language-specific things. It's a natural division. It is conceivable that there could be a single cross-language, language-level package manager. But what language would it be written in? And how would it deal with the fact that each language uses different technologies and has different approaches for installing and configuring libraries and packages? When you think about it for a bit, I don't think it's so insensible for each language to have its own package manager.
- Hello71 13y ago
  > you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc wrong. that's the package maintainer's job. moreover, sensible package management systems have templates or libraries (eclasses in Gentoo) to properly handle all kinds of packages. all we ask for is that developers don't actively break package management by bundling everything as is done here.
  - voltagex_ 13y ago
    Ir seems like there is often a breakdown in communication between upstream and distro package maintainers. It may also be that authors don't have time to talk with ~3 (Debian, Fedora, ???) distros, each with different packaging processes and rules.
  - drewcrawford 13y ago
mmerickel 13y ago
nix (http://nixos.org/nix/ http://nixos.org/nix/) seems to be the solution with the most potential in this area, solving all dependencies at the project level and removing the need for language specific build environments.
- Nullabillity 13y ago
  Also, per-user installs sweetens the deal even further.
- pmahoney 13y ago
  I've actually been exploring nix myself recently. It has some interesting features. I'm finding it quite nice to maintain some C-based packages across both Linux and Mac OS X (where it is somewhat of a competitor to Homebrew). I've been trying to get it to manage some Ruby applications, complete with gems, private github gems, etc. This has been difficult, in part because some areas of nix are a bit rough, and in part because Rubygems has incomplete dependency information (i.e. there's no standard way to know that, say typhoeus gem depends on C library libcurl, and I'm still unclear on how/if dynamic loading can be made to load the nix packaged libcurl). Something like nix is what I had in mind with my original question. I'd like to see new languages at least design their custom package managers with these more generic systems in mind. Rubygems has some quirks, and Bundler just includes everything and the kitchen sink (sourcing gems from local files, github, various other features) that complicate matters. Nix is good for keeping things more or less isolated and tracking complete dependencies, though an alternate way to accomplish similar is something like Docker and then just scripting all the native package managers...

null_ptr 13y ago

I like this! Is it a sort of staging ground for libraries to eventually graduate to various distributions' official repositories, or is it just random bite-sized utilities for quick weekend projects?

Sir_Cmpwn 13y ago

What ever happened to "apt-get" as the C package manager?

JetSpiegel 13y ago
$ apt-get bash: apt-get: command not found
pjmlp 13y ago
Doesn't work on AS/400, Windows, VxWorks, QNX, ...
- panzi 13y ago
  And Fedora Linux.

gxc40 13y ago

There is some overlap with CCAN (http://ccodearchive.net/ http://ccodearchive.net/).

chj 13y ago

Can be a big time saver. For any package manager to work painlessly and to scale, the repository must be able to detect name conflicts on the code level.

giovannibajo1 13y ago

I think this is missing half of the problem, that is connecting the package with the build system. I know there is no "standard" build system for C, but still. Go for instance does both; once you "go get" a package, it not only downloads and pre-builds it, but makes sure that it is available for transparent usage in your program. I personally bias towards CMake, so I would appreciate a C package manager which also makes sure that the package is included in my CMakeList.txt somehow.

shoo 13y ago
what if these are in fact two separate problems? then perhaps it is simpler to solve them with separate tools instead of conflating them.
- giovannibajo1 13y ago
  I can't see how it's simpler, because then you need to write N*M bridges/connectors/plugins to let N package managers interact with M build systems. And what if the library you download is meant to be built with a build system which is different from the one you chose for your application? What if you then want to globally turn on debugging or change a compilation flag, or a preprocessor define to enable/disable a feature? I don't know of any language-specific package manager that doesn't also make the package immediately available to the programming environment without further fiddling. Solving half of the problem... well, it's a half solution :)

michaelmior 13y ago

It's a little scary for me that the source of available package is a wiki page which is editable by anyone with a GitHub account.

lcampbell 13y ago
Not only that, but there appear to be potential buffer overflows which involve the data pulled from the wiki page[1]. I'm not sure why asprintf wasn't used here (and in other places). The partial re-implementation of libc[2, 3] is also somewhat frightening. As are potential unchecked integer overflows when calculating a size to pass to malloc[4]. C is hard. -- [1] https://github.com/clibs/clib/blob/master/src/clib-install.c#L86 https://github.com/clibs/clib/blob/master/src/clib-install.c... [2] https://github.com/clibs/clib/blob/master/deps/fs/fs.c https://github.com/clibs/clib/blob/master/deps/fs/fs.c [3] https://github.com/clibs/clib/blob/master/deps/str-copy/str-copy.c https://github.com/clibs/clib/blob/master/deps/str-copy/str-... [4] https://github.com/clibs/clib/blob/master/deps/http-get/http-get.c#L19 https://github.com/clibs/clib/blob/master/deps/http-get/http...
- gone35 12y ago
  Oh dear... I'm just sight-linting so this might be wrong, but I'm certain the function executable in src/clib-install.c [1] can be easily overflown by providing a maliciously long package name or version --neither of which are validated at all [2] despite coming straight from the wiki page, as you point out. The function allocates 256-byte buffers to store a url and file name, respectively, but blindly fills them up using sprintf with non-validated user-supplied data, and without checking the resulting number of characters written, eg: char *url = malloc(256); if(NULL == file) goto e1; sprintf(file,"%s-%s.tar.gz",pkg->name,pkg->version); C is unforgiving indeed. Also what's up with all those (0 == count) yoda conditionals all over? Is that becoming a thing again? [1] https://github.com/clibs/clib/blob/master/src/clib-install.c https://github.com/clibs/clib/blob/master/src/clib-install.c [2] As far as I can tell, the package info from the wiki is dealt with by functions in deps/clib-package.c that, despite promising names like json_object_get_string_safe, do not ever validate the provided name or version string: https://github.com/clibs/clib/blob/master/deps/clib-package/clib-package.c https://github.com/clibs/clib/blob/master/deps/clib-package/...