6 ms·

CDE: Easily run a program on any linux without thinking about dependencies

132 points by gregwebs 16y ago

rbanffy 16y ago
Common Desktop Environment? I have a couple old RISC boxes running it... ;-) Sorry. Couldn't resist... Now, seriously, this goes a bit beyond package management by bringing along what seemed to be the specific versions of libraries. The goal is to do reproducible results for scientific data but, in this case, I would prefer not to have the original libraries packed along but having the dependencies flagged in a way that any version-specific bug could be easily pointed out.
- js4all 16y ago
  Common Desktop Environment. That was also my first thought. We had it on our Sun E250.
  - rbanffy 16y ago
    Nice machine. Wicked fast on its day. I bet it still can hold its own despite the 2GB RAM limit.
    - js4all 16y ago
      It was and served well. But now it's not in use anymore.
- pyre 16y ago
  I also though Common Desktop Environment at first. At least from a 'marketing' perspective, it might be a good idea to not use an acronym that is already in use in the Linux/Unix world.
  - mattew 16y ago
    This is a very good point. I can't see CDE and not think about all the environment on all the Sun workstations that I have had in the past.
- darkandbrooding 16y ago
  That was my first thought as well. My initial exposure to nix was a purchased copy of Red Hat Linux 4.2, running on an HP Vectra tower. and I remember paying extra for the CDE install CDs. An absolute noob when I started, by the time I'd learned enough to make my modem work I felt like I'd climbed a mountain. sigh* Good times.
mayanklahiri 16y ago
This is fantastic. I haven't tried it out yet, but I'll give it a shot for the paper I'm writing. BTW, even with a "uniform" Linux environment, I've been finding so many differences in the behavior of libraries that it's driving me nuts. This will be a nice way to get past, for example, inconsistencies in wxWidgets.
- pgbovine 16y ago
  what research field are you working in? please email me personally if you'd like to chat more about how cde can help your workflow. i'm interested in collecting real-life case studies and anecdotes
  - mayanklahiri 16y ago
    I replied to you via email, but thought I'd post it here as well in case anyone else had suggestions. I've been thinking about a general purpose experimentation framework for research. Over the last 5 years, I've been re-writing parts of this framework for different projects and wish I had spent some time generalizing and packaging it. The workflow is: dataset -> experimental parameters -> code+libraries (this is where the trouble often is) -> result files (plain text, hundreds!) -> analysis (generally in R) -> selectively plotting results
pierrefar 16y ago
From my superficial knowledge, this is quite similar to the PC-BSD PBI format: http://www.pcbsd.org/content/view/20/26/ http://www.pcbsd.org/content/view/20/26/ . They both allow you to distribute software in an easy non-hellish way on Linux.
pgbovine 16y ago
hi everyone, i'm the author of CDE, so please feel free to ping me if you have questions, complaints, or bug reports. i'm in the midst of fixing a bunch of silly bugs now and improving the documentation.
- rbanffy 16y ago
  I find the dependency-capture mechanism very interesting. I would love to see it used to help generating dependency lists for package management systems too. I will probably have to build my first .rpm packages (I am yet to read the first docs) in the near future and CDE could be a lovely addition to my toolbox. Thanks for doing it.
- Goladus 16y ago
  This looks awesome. My biggest initial fear (not having used it yet), is that I would inadvertently package up something like my private SSH key.
  - pgbovine 16y ago
    oh good point. the contents of the cde-package is simply a directory hierarchy, so you can feel free to use 'find' to skim thru it. also, i'm now adding a feature where you can tell cde to ignore certain directories (e.g., ~/.ssh/) so that it won't pick up those files.
    - bostonvaulter2 16y ago
      Maybe you could output a warning on files with perms like 600?
- partition 16y ago
  Something like this has been sorely needed. I'm wondering if it's possible to generalize this over different OSes, if the code itself is multi-platform, or there is some way to specify hints for it (or should there be). I also go to Stanford so would be easy to discuss this in depth if you want.
InclinedPlane 16y ago
If anyone wonders why C/C++ programming is still so popular, here's a perfect case study. Most new fangled languages don't offer the option of static linking. This single piece of uncompromising idiological doctrine has led to billions of dollars in effort expended in engineering systems to workaround that choice and to effectively reproduce static linking in different form.
- j_baker 16y ago
  Most "newfangled" languages don't use linking in the same way as C and C++ do anyway (unless of course you're interfacing with C or C++). For instance, you can't statically link to a python source file. That's not because of ideology. It's because static linking doesn't even have any meaning.
  - d0mine 16y ago
    Analog of static linking for a pure python library is including it within the library package. Another example: deploying python application in its own virtualenv environment.
- mkramlich 16y ago
  dynamic linking is generally a functional superset of static linking because you can bundle the needed libraries with the app, as say loose files in the app's dir tree, for example. This lets you avoid Shared DLL Hell while still allowing you to surgically deploy a critical bugfix to one of those libraries.
mkramlich 16y ago
awesome tool, great job! no like the name though. sounds like a Unix desktop system.
nitrogen 16y ago
Cool idea, and a worthy cause. Is there any provision for copyleft license compliance? Perhaps a distant-future version could use dpkg -S or the rpm equivalent to find which package owns a file, then apt-get source or something to optionally link or bundle a source package with the sandbox.
chbarts 16y ago
I think the potted example on the web page is somewhat poor, as it seems to be a problem existing package management systems already solve. The more interesting aspect is the sandboxing, and how this is great for one-off software runs where it isn't worth it to install the needed packages on your base system because they're logically just a unit with the software you're testing. You mention this, but I think you'd do well to make it more obvious. This is also, and this is something you mention, a way of preserving historic software and data. Bundle a full CDE package with an emulator and you can keep running that program for even longer; in fact, it seems a CDE package could gradually grow layers as time demands, from moving a (user-mode) kernel in, to adding device emulation, to going the final step and making the package more-or-less a ship in a bottle running in full hardware emulation. This could also grow into a way to run untrusted (or minimally-trusted) software in an iron box, not just a sandbox. But that is harder and probably not your preferred direction.
sfk 16y ago
Reposting the comment of chbarts, which has been killed despite being entirely appropriate. Moderating has gone completely out of hand lately: chbarts wrote: " I think the potted example on the web page is somewhat poor, as it seems to be a problem existing package management systems already solve. The more interesting aspect is the sandboxing, and how this is great for one-off software runs where it isn't worth it to install the needed packages on your base system because they're logically just a unit with the software you're testing. You mention this, but I think you'd do well to make it more obvious. This is also, and this is something you mention, a way of preserving historic software and data. Bundle a full CDE package with an emulator and you can keep running that program for even longer; in fact, it seems a CDE package could gradually grow layers as time demands, from moving a (user-mode) kernel in, to adding device emulation, to going the final step and making the package more-or-less a ship in a bottle running in full hardware emulation. This could also grow into a way to run untrusted (or minimally-trusted) software in an iron box, not just a sandbox. But that is harder and probably not your preferred direction."
- pgbovine 16y ago
  thanks for reposting, sfk. i don't know why the original comment would've been killed ... chbarts brought up a great critique about my presentation of cde.
  - blasdel 16y ago
    It looks like they got capriciously hellbanned for comments in this thread: http://news.ycombinator.com/item?id=1320949 http://news.ycombinator.com/item?id=1320949 — everything after that was auto-killed, several dozen productive comments over the course of six months. Items can be fully deleted or [dead]-ed, and users can be silently hellbanned so that all of their contributions are [dead] on arrival for everyone but themselves. AFAIK there are no other metamoderation mechanisms.
    - chbarts 16y ago
      Well, thank you. I suspected as much but I wasn't sure. I am honestly happy to hear this discussed openly, and I'm happy to hear any of my comments characterized as 'productive'. (I'm male, by the way, if that makes it any easier to continue this discussion.)
pixdamix 16y ago
It reminds me of the Ermine Statifier. http://www.magicermine.com/sesl.html http://www.magicermine.com/sesl.html. This tools is so useful when you have to for example run tcpdump or a QT app on an embedded system without installing a TON of libraries or messing with LD_LIBRARY_PATH.
bbulkow 16y ago
Wouldn't this cause severe problems with copywrite and licensing? Just because you have a copy of some library on a machine doesn't mean you have the ability to duplicate it to another machine, let alone archive it for future researchers. I could also see the possible issue of pulling in GPL licensed libraries, and the general theory is if you ship a package with GPL licensed code in it, your code must be GPL. There's a fine line between "sending along someone else's libraries" and becoming a monolithic executable containing someone else's libraries - and between those points is where GPL gets you.
pilif 16y ago
There needs to be some kind of solution for the problem of really confidential files to be packaged. The second example in the video packaged the machines /etc/passwd for example (granted, it doesn't contain the passwords, but still). The third example was showing some real bloat in that it also packaged the kernel and the initramd. I really don't see how these two can be of any use on the target machine. Still. Very interesting idea and impressively easy to use.

132 points by gregwebs 16y ago

Common Desktop Environment? I have a couple old RISC boxes running it... ;-) Sorry. Couldn't resist... Now, seriously, this goes a bit beyond package management by bringing along what seemed to be the specific versions of libraries. The goal is to do reproducible results for scientific data but, in this case, I would prefer not to have the original libraries packed along but having the dependencies flagged in a way that any version-specific bug could be easily pointed out.

js4all 16y ago
Common Desktop Environment. That was also my first thought. We had it on our Sun E250.
- rbanffy 16y ago
  Nice machine. Wicked fast on its day. I bet it still can hold its own despite the 2GB RAM limit.
  - js4all 16y ago
    It was and served well. But now it's not in use anymore.
pyre 16y ago
I also though Common Desktop Environment at first. At least from a 'marketing' perspective, it might be a good idea to not use an acronym that is already in use in the Linux/Unix world.
- mattew 16y ago
  This is a very good point. I can't see CDE and not think about all the environment on all the Sun workstations that I have had in the past.
darkandbrooding 16y ago
That was my first thought as well. My initial exposure to nix was a purchased copy of Red Hat Linux 4.2, running on an HP Vectra tower. and I remember paying extra for the CDE install CDs. An absolute noob when I started, by the time I'd learned enough to make my modem work I felt like I'd climbed a mountain. sigh* Good times.

mayanklahiri 16y ago

This is fantastic. I haven't tried it out yet, but I'll give it a shot for the paper I'm writing. BTW, even with a "uniform" Linux environment, I've been finding so many differences in the behavior of libraries that it's driving me nuts. This will be a nice way to get past, for example, inconsistencies in wxWidgets.

pgbovine 16y ago
what research field are you working in? please email me personally if you'd like to chat more about how cde can help your workflow. i'm interested in collecting real-life case studies and anecdotes
- mayanklahiri 16y ago
  I replied to you via email, but thought I'd post it here as well in case anyone else had suggestions. I've been thinking about a general purpose experimentation framework for research. Over the last 5 years, I've been re-writing parts of this framework for different projects and wish I had spent some time generalizing and packaging it. The workflow is: dataset -> experimental parameters -> code+libraries (this is where the trouble often is) -> result files (plain text, hundreds!) -> analysis (generally in R) -> selectively plotting results

pierrefar 16y ago

From my superficial knowledge, this is quite similar to the PC-BSD PBI format: http://www.pcbsd.org/content/view/20/26/ http://www.pcbsd.org/content/view/20/26/ . They both allow you to distribute software in an easy non-hellish way on Linux.

pgbovine 16y ago

hi everyone, i'm the author of CDE, so please feel free to ping me if you have questions, complaints, or bug reports. i'm in the midst of fixing a bunch of silly bugs now and improving the documentation.

rbanffy 16y ago
I find the dependency-capture mechanism very interesting. I would love to see it used to help generating dependency lists for package management systems too. I will probably have to build my first .rpm packages (I am yet to read the first docs) in the near future and CDE could be a lovely addition to my toolbox. Thanks for doing it.
Goladus 16y ago
This looks awesome. My biggest initial fear (not having used it yet), is that I would inadvertently package up something like my private SSH key.
- pgbovine 16y ago
  oh good point. the contents of the cde-package is simply a directory hierarchy, so you can feel free to use 'find' to skim thru it. also, i'm now adding a feature where you can tell cde to ignore certain directories (e.g., ~/.ssh/) so that it won't pick up those files.
  - bostonvaulter2 16y ago
    Maybe you could output a warning on files with perms like 600?
partition 16y ago
Something like this has been sorely needed. I'm wondering if it's possible to generalize this over different OSes, if the code itself is multi-platform, or there is some way to specify hints for it (or should there be). I also go to Stanford so would be easy to discuss this in depth if you want.

InclinedPlane 16y ago

If anyone wonders why C/C++ programming is still so popular, here's a perfect case study. Most new fangled languages don't offer the option of static linking. This single piece of uncompromising idiological doctrine has led to billions of dollars in effort expended in engineering systems to workaround that choice and to effectively reproduce static linking in different form.

j_baker 16y ago
Most "newfangled" languages don't use linking in the same way as C and C++ do anyway (unless of course you're interfacing with C or C++). For instance, you can't statically link to a python source file. That's not because of ideology. It's because static linking doesn't even have any meaning.
- d0mine 16y ago
  Analog of static linking for a pure python library is including it within the library package. Another example: deploying python application in its own virtualenv environment.
mkramlich 16y ago
dynamic linking is generally a functional superset of static linking because you can bundle the needed libraries with the app, as say loose files in the app's dir tree, for example. This lets you avoid Shared DLL Hell while still allowing you to surgically deploy a critical bugfix to one of those libraries.

mkramlich 16y ago

awesome tool, great job! no like the name though. sounds like a Unix desktop system.

nitrogen 16y ago

Cool idea, and a worthy cause. Is there any provision for copyleft license compliance? Perhaps a distant-future version could use dpkg -S or the rpm equivalent to find which package owns a file, then apt-get source or something to optionally link or bundle a source package with the sandbox.

chbarts 16y ago

I think the potted example on the web page is somewhat poor, as it seems to be a problem existing package management systems already solve. The more interesting aspect is the sandboxing, and how this is great for one-off software runs where it isn't worth it to install the needed packages on your base system because they're logically just a unit with the software you're testing. You mention this, but I think you'd do well to make it more obvious. This is also, and this is something you mention, a way of preserving historic software and data. Bundle a full CDE package with an emulator and you can keep running that program for even longer; in fact, it seems a CDE package could gradually grow layers as time demands, from moving a (user-mode) kernel in, to adding device emulation, to going the final step and making the package more-or-less a ship in a bottle running in full hardware emulation. This could also grow into a way to run untrusted (or minimally-trusted) software in an iron box, not just a sandbox. But that is harder and probably not your preferred direction.

sfk 16y ago

Reposting the comment of chbarts, which has been killed despite being entirely appropriate. Moderating has gone completely out of hand lately: chbarts wrote: " I think the potted example on the web page is somewhat poor, as it seems to be a problem existing package management systems already solve. The more interesting aspect is the sandboxing, and how this is great for one-off software runs where it isn't worth it to install the needed packages on your base system because they're logically just a unit with the software you're testing. You mention this, but I think you'd do well to make it more obvious. This is also, and this is something you mention, a way of preserving historic software and data. Bundle a full CDE package with an emulator and you can keep running that program for even longer; in fact, it seems a CDE package could gradually grow layers as time demands, from moving a (user-mode) kernel in, to adding device emulation, to going the final step and making the package more-or-less a ship in a bottle running in full hardware emulation. This could also grow into a way to run untrusted (or minimally-trusted) software in an iron box, not just a sandbox. But that is harder and probably not your preferred direction."

pgbovine 16y ago
thanks for reposting, sfk. i don't know why the original comment would've been killed ... chbarts brought up a great critique about my presentation of cde.
- blasdel 16y ago
  It looks like they got capriciously hellbanned for comments in this thread: http://news.ycombinator.com/item?id=1320949 http://news.ycombinator.com/item?id=1320949 — everything after that was auto-killed, several dozen productive comments over the course of six months. Items can be fully deleted or [dead]-ed, and users can be silently hellbanned so that all of their contributions are [dead] on arrival for everyone but themselves. AFAIK there are no other metamoderation mechanisms.
  - chbarts 16y ago
    Well, thank you. I suspected as much but I wasn't sure. I am honestly happy to hear this discussed openly, and I'm happy to hear any of my comments characterized as 'productive'. (I'm male, by the way, if that makes it any easier to continue this discussion.)

pixdamix 16y ago

It reminds me of the Ermine Statifier. http://www.magicermine.com/sesl.html http://www.magicermine.com/sesl.html. This tools is so useful when you have to for example run tcpdump or a QT app on an embedded system without installing a TON of libraries or messing with LD_LIBRARY_PATH.

bbulkow 16y ago

Wouldn't this cause severe problems with copywrite and licensing? Just because you have a copy of some library on a machine doesn't mean you have the ability to duplicate it to another machine, let alone archive it for future researchers. I could also see the possible issue of pulling in GPL licensed libraries, and the general theory is if you ship a package with GPL licensed code in it, your code must be GPL. There's a fine line between "sending along someone else's libraries" and becoming a monolithic executable containing someone else's libraries - and between those points is where GPL gets you.

pilif 16y ago

There needs to be some kind of solution for the problem of really confidential files to be packaged. The second example in the video packaged the machines /etc/passwd for example (granted, it doesn't contain the passwords, but still). The third example was showing some real bloat in that it also packaged the kernel and the initramd. I really don't see how these two can be of any use on the target machine. Still. Very interesting idea and impressively easy to use.