15 ms·

GNU Shepherd 0.5.0 released

148 points by mbakke 8y ago

rhabarba 8y ago
> The GNU Shepherd is developed jointly with the GNU Guix project Wasn't it made for the hurd?
- CamTin 8y ago
  Shepherd should run on Hurd, since Guile does, but it was created as part of the GNU system, which is in reality usually run on the Linux kernel.
- equalunique 8y ago
  Yes it was, but more activity with Shepherd is occurring in the Guix SD project. Guix has also been ported to Hurd, so it's all going to be one big happy family (one day when Hurd finally gets USB and 64-bit support).
jonathanstrange 8y ago
Interesting,after all this criticism of systemd. This makes me wonder about the GuixSD distribution. Is this distro ready for prime-time. Suppose I switched from Linux Mint to GuixSD, what would be the advantages and disadvantages? I like Lisp and program in Racket, so Guile wouldn't be far from that. I just wonder about experiences of other. How much does switching to such a lesser known free distro as the main OS "constrain" you?
- rhabarba 8y ago
  > Suppose I switched from Linux Mint to GuixSD, what would be the advantages and disadvantages? You'd lose a lot of software choices. Actually, the restriction to "GNU-friendly software" was what made me skip GuixSD so far. I guess that my WiFi wouldn't work.
  - serhart 8y ago
    Not true. You can write your own packages in the same high level interface. Including the mainline Linux kernel and broadcom drivers. I have GuixSd running on my 2013 MBP
    - tacolizard 8y ago
      can you tell me how you got that running in some more detail? I also have a 2013 MBP and would like to try out GuixSD
      - serhart 8y ago
        I don't have access to it now. When I do I can put the package definitions and OS config up somewhere. When you first boot GuixSD most of the core stuff should work fine. Wireless will not work but you should have ethernet. I created mainline linux kernel and firmware packages [1]. Then you need to get the broadcom b43 firmware to get wireless support. I think I made 2 packages. One for b43-fwcutter and one the broadcom-wl firmware which uses b43-fwcutter to cut and install it. I think just about everything on my MBP was working after that. [1] example linux package https://github.com/wingo/guix-nonfree/blob/master/gnu/packages/linux-nonfree.scm https://github.com/wingo/guix-nonfree/blob/master/gnu/packag... [2] http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/ http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/
      - serhart 8y ago
        Here is a gist of the relevant packages and configs I have for my MBP. https://gist.github.com/steve-ayerhart/81c9dfc773472e08002cd890216de481 https://gist.github.com/steve-ayerhart/81c9dfc773472e08002cd...
    - noobermin 8y ago
      Is there a...non official repository of these somewhere?
      - serhart 8y ago
- rufugee 8y ago
  I use nixos, not GuixSD, but I can speak to "what would be the advantages and disadvantages", particularly since guix is a reimplementation of nix. The benefits to me are immense, because I travel between around six machines on a daily or weekly basis. Prior to moving to NixOS, I'd have to manually keep those machines in sync in terms of software installs, etc. However, after switching to NixOS, I have a set of files I version control which will declaratively build the system the way I want it. So, if I install a new package and I'd like it be the same across all machines, I simply configure it, commit to git, and then pull down the config on the other machines and do "nixos-rebuild switch". Everything becomes exactly the same. It's saved me so much time in terms of managing my computing life. I really couldn't see going back to a regular distro after using one which is declarative. The drawbacks? Well, for one, Nix and Guix have various tricks with symlinking, etc, to make it appear to applications as if it's a typical Linux filesystem, but it's not. E.g. there's no /lib, etc. It's a little weird getting used to. And with nix, there's the nix language, which isn't that difficult to learn but it's not my favorite. If the underlying language needs to be functional, I'd prefer a lisp over nix. Also, with nixos, there are too many different commands required to learn when you get deeper into it, but work is underway to improve that story. At least for me...the benefits far outweigh the risks. I don't think I can ever go back. I think there's a real opportunity for a commercially-backed declarative Linux os...it really is a game changer and would do well in the enterprise.
  - xfer 8y ago
    For me, the problem isn't so much with the nix language(it's very similar to haskell), but the lack of discoverability for docs of library. The manual is mostly incomplete and i often have to look at sources. My most favourite thing is that i can just override the packages to compile and install a new version/extra configure flags of a package that's not yet upstream.
    - rufugee 8y ago
      Interesting... I've found the manuals to be ok so far. Can you share what you've had problems finding?
peatmoss 8y ago
Man, I’ve really wanted to take the plunge and dedicate some play time to GUIX for a while now. I feel like between the GUIX package system and Shepherd, they’re creating an authentically new and exciting thing in the universe of Linux distributions. The fact that they’re using Scheme for so much of it is also pretty cool, as I think this sort of high level systems programming is a great potential use case for showcasing meta programming as well as the unique power of lisps for code-as-data.
- rufugee 8y ago
  Same here...as a NixOS user, I really wish it used scheme instead of the nix language. However, I do like the pragmatic nature of NixOS (you're able to easily install non-free packages). GUIX doesn't make this easy at all from what I understand. I appreciate those virtues, but practically speaking, I have to use non-free software on the desktop all the time.
  - _emacsomancer_ 8y ago
    Guix does have the capacity to add 'channels', e.g. https://gitlab.com/mbakke/guix-chromium https://gitlab.com/mbakke/guix-chromium , so while non-free software while never be part of the distribution itself, it could be done unofficially. Also, you can run the Nix standalone on top of GuixSD, so you could run GuixSD and then use Nix to install whatever's missing.
    - yorwba 8y ago
      > you could run GuixSD and then use Nix to install whatever's missing. How much cooperation between the two is necessary? If I wanted to install CUDA drivers with Nix would it "just work" (i.e. if it works on NixOS it works on GuixSD) or could GuixSD make problems? I found a comment from a year ago claiming that the kernel wouldn't load blobs: https://news.ycombinator.com/item?id=14630581 https://news.ycombinator.com/item?id=14630581
      - rufugee 8y ago
bjoli 8y ago
For anyone interested in guile, guile 3.0 will have a JIT, making it even faster than 2.2 (by quite a large margin). Guile is becoming really nice, and I find myself reaching for other lisps less and less.
- logicchains 8y ago
  How far along is it towards becoming the underlying runtime for Emacs lisp?
  - bjoli 8y ago
    With the bikeshedding we see for much less controversial changes in the Emacs codebase, I suspect it will be released around the release of Duke Nukem forever II. Elisp code runs fine under guile, but it has seen virtually no optimisation work and does a lot of unnecessary work. I suspect there is an lot of low hanging fruit, but since there is virtually no interest in guile-emacs apart from in comments on HN and Reddit I doubt the situation will get much better. The guile VM is getting better and better though. Maybe they'll come around by guile 4 :)
    - yepguy 8y ago
      I've always thought someone should use guile to add support for elisp plugins to vim & neovim using their rpc interfaces. Then maybe if something like that existed the org-mode authors could be convinced to cleanly separate out the emacs-specific code, and we could have org-mode on vim using the same codebase as emacs. I have no idea how viable that is, but it would be pretty sweet. Too much for me to take on right now, but the idea has been rattling around my brain for a few years.
  - equalunique 8y ago
    https://www.emacswiki.org/emacs/GuileEmacs https://www.emacswiki.org/emacs/GuileEmacs
- zeveb 8y ago
  Scheme is a neat little language, but it's really not ideal for writing full software systems. Indeed, Guile & Racket (back when it was PLT Scheme) are perfect exemplars of Greenspun's Tenth Rule: Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. E.g. GOOPS is similar to CLOS, but missing a bit and unsupported other than in Guile. Meanwhile, implementations like SBCL are free, compile to machine code and are competitive on some benchmarks with C/C++. Common Lisp — while it does have some warts — is intended for building long-lived, durable, full-featured systems. It's a really good language, and it standardises quite a lot. Often, what look like warts are actually useful features in large systems. It really saddens me that so much effort has been expended trying to make Scheme a reasonable language for large systems, when there already exists one which is. That doesn't even get to the bits of Scheme which I believe are mistakes (e.g. a single namespace, making NIL, () & #f different).
arminiusreturns 8y ago
This is awesome! I have really been impressed with gnu's use of guile, and I really like how much more quickly iteration is than many other gnu projects. For those heavy cron users, mcron (guile based cron replacement) has been my favorite so far. With a good deamon manager, we're one step closer to HURD!
tomkat0789 8y ago
Pardon me, I'm sort of a casual Linux user and I have no idea what people are talking about. What are GNU Guix and Shepherd? A cursory glance at Wikipedia makes it seem like they're GNU alternatives to the apt-get and init/systemD I've been on for awhile. I'm outside of my Debian comfort zone, apologies for my ignorance! What's significant about this?
- pjmlp 8y ago
  Basically they follow similar ideas to NixOS, but using Scheme instead of a custom script language.
theamk 8y ago
Does anyone know if they solved their "trust init script too much" problems? Last time I checked, they were loading user-defined scripts right into PID 1, so a badly written script could bring down the whole system. People blame systemd for pid 1 instability, but at least systemd crashes get fixed eventually. With Shepherd, any package on the system can cause PID 1 to crash, and this just seems to crazy for me.
- lfam 8y ago
  Can you give any more detail about this? Or a reference to a bug report or mailing list thread?
  - theamk 8y ago
    The manual [1] starts by saying: "When shepherd gets started, it reads and evaluates a configuration file." That config is full-featured, seemingly unrestricted, lisp file. While I saw some mentions of handling syntax errors on the mailing list, I saw no mentions of what happens if there is an infinite loop or if a script just reads a huge files and OOM's the process. Now, one could imagine that maybe configs are parsed in a separate process, and then serialized and shipped to the main process for safe execution. But it does not seem to be the case -- the manual explicitly says you can use any Guile code to start processes, not just a safe subset. Another option is that maybe there is a thread or a forked process for parsing config. But this does not seem to be the case either -- the manual does not mention that, and searching of the source code for "fork" or "thread" or "timeout" does not seem to indicate anything related. The only mention of timeness expectation seem to be a single sentence "Actions are expected to be non-blocking." in one of the mailing lists [2]. So together, it seems to me that any bad script will just cause the daemon, and by extension the whole system, to not start. [1] "https://www.gnu.org/software/shepherd/manual/shepherd.html#Services" https://www.gnu.org/software/shepherd/manual/shepherd.html#S... [2] http://lists.gnu.org/archive/html/guix-devel/2016-11/msg00945.html http://lists.gnu.org/archive/html/guix-devel/2016-11/msg0094...
_emacsomancer_ 8y ago
There's a quote from the GNU Shepherd manual I really appreciated: > I decided to not make it too sophisticated by trying to guess what the user might want just to theoretically fulfill the request we are processing. If something goes wrong, it is usually better to tell the user about the problem and let her fix it, taking care to make finding solutions or workarounds for problems (like a misconfigured service) easy. This way, the user is in control of what happens and we can keep the implementation simple. To make a long story short, we don't try to be too clever, which is usually a good idea in developing software.

148 points by mbakke 8y ago

rhabarba 8y ago

> The GNU Shepherd is developed jointly with the GNU Guix project Wasn't it made for the hurd?

CamTin 8y ago
Shepherd should run on Hurd, since Guile does, but it was created as part of the GNU system, which is in reality usually run on the Linux kernel.
equalunique 8y ago
Yes it was, but more activity with Shepherd is occurring in the Guix SD project. Guix has also been ported to Hurd, so it's all going to be one big happy family (one day when Hurd finally gets USB and 64-bit support).

jonathanstrange 8y ago

Interesting,after all this criticism of systemd. This makes me wonder about the GuixSD distribution. Is this distro ready for prime-time. Suppose I switched from Linux Mint to GuixSD, what would be the advantages and disadvantages? I like Lisp and program in Racket, so Guile wouldn't be far from that. I just wonder about experiences of other. How much does switching to such a lesser known free distro as the main OS "constrain" you?

rhabarba 8y ago
> Suppose I switched from Linux Mint to GuixSD, what would be the advantages and disadvantages? You'd lose a lot of software choices. Actually, the restriction to "GNU-friendly software" was what made me skip GuixSD so far. I guess that my WiFi wouldn't work.
- serhart 8y ago
  Not true. You can write your own packages in the same high level interface. Including the mainline Linux kernel and broadcom drivers. I have GuixSd running on my 2013 MBP
  - tacolizard 8y ago
    can you tell me how you got that running in some more detail? I also have a 2013 MBP and would like to try out GuixSD
    - serhart 8y ago
      I don't have access to it now. When I do I can put the package definitions and OS config up somewhere. When you first boot GuixSD most of the core stuff should work fine. Wireless will not work but you should have ethernet. I created mainline linux kernel and firmware packages [1]. Then you need to get the broadcom b43 firmware to get wireless support. I think I made 2 packages. One for b43-fwcutter and one the broadcom-wl firmware which uses b43-fwcutter to cut and install it. I think just about everything on my MBP was working after that. [1] example linux package https://github.com/wingo/guix-nonfree/blob/master/gnu/packages/linux-nonfree.scm https://github.com/wingo/guix-nonfree/blob/master/gnu/packag... [2] http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/ http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/
    - serhart 8y ago
      Here is a gist of the relevant packages and configs I have for my MBP. https://gist.github.com/steve-ayerhart/81c9dfc773472e08002cd890216de481 https://gist.github.com/steve-ayerhart/81c9dfc773472e08002cd...
  - noobermin 8y ago
    Is there a...non official repository of these somewhere?
    - serhart 8y ago
rufugee 8y ago
I use nixos, not GuixSD, but I can speak to "what would be the advantages and disadvantages", particularly since guix is a reimplementation of nix. The benefits to me are immense, because I travel between around six machines on a daily or weekly basis. Prior to moving to NixOS, I'd have to manually keep those machines in sync in terms of software installs, etc. However, after switching to NixOS, I have a set of files I version control which will declaratively build the system the way I want it. So, if I install a new package and I'd like it be the same across all machines, I simply configure it, commit to git, and then pull down the config on the other machines and do "nixos-rebuild switch". Everything becomes exactly the same. It's saved me so much time in terms of managing my computing life. I really couldn't see going back to a regular distro after using one which is declarative. The drawbacks? Well, for one, Nix and Guix have various tricks with symlinking, etc, to make it appear to applications as if it's a typical Linux filesystem, but it's not. E.g. there's no /lib, etc. It's a little weird getting used to. And with nix, there's the nix language, which isn't that difficult to learn but it's not my favorite. If the underlying language needs to be functional, I'd prefer a lisp over nix. Also, with nixos, there are too many different commands required to learn when you get deeper into it, but work is underway to improve that story. At least for me...the benefits far outweigh the risks. I don't think I can ever go back. I think there's a real opportunity for a commercially-backed declarative Linux os...it really is a game changer and would do well in the enterprise.
- xfer 8y ago
  For me, the problem isn't so much with the nix language(it's very similar to haskell), but the lack of discoverability for docs of library. The manual is mostly incomplete and i often have to look at sources. My most favourite thing is that i can just override the packages to compile and install a new version/extra configure flags of a package that's not yet upstream.
  - rufugee 8y ago
    Interesting... I've found the manuals to be ok so far. Can you share what you've had problems finding?

peatmoss 8y ago

Man, I’ve really wanted to take the plunge and dedicate some play time to GUIX for a while now. I feel like between the GUIX package system and Shepherd, they’re creating an authentically new and exciting thing in the universe of Linux distributions. The fact that they’re using Scheme for so much of it is also pretty cool, as I think this sort of high level systems programming is a great potential use case for showcasing meta programming as well as the unique power of lisps for code-as-data.

rufugee 8y ago
Same here...as a NixOS user, I really wish it used scheme instead of the nix language. However, I do like the pragmatic nature of NixOS (you're able to easily install non-free packages). GUIX doesn't make this easy at all from what I understand. I appreciate those virtues, but practically speaking, I have to use non-free software on the desktop all the time.
- _emacsomancer_ 8y ago
  Guix does have the capacity to add 'channels', e.g. https://gitlab.com/mbakke/guix-chromium https://gitlab.com/mbakke/guix-chromium , so while non-free software while never be part of the distribution itself, it could be done unofficially. Also, you can run the Nix standalone on top of GuixSD, so you could run GuixSD and then use Nix to install whatever's missing.
  - yorwba 8y ago
    > you could run GuixSD and then use Nix to install whatever's missing. How much cooperation between the two is necessary? If I wanted to install CUDA drivers with Nix would it "just work" (i.e. if it works on NixOS it works on GuixSD) or could GuixSD make problems? I found a comment from a year ago claiming that the kernel wouldn't load blobs: https://news.ycombinator.com/item?id=14630581 https://news.ycombinator.com/item?id=14630581
    - rufugee 8y ago

bjoli 8y ago

For anyone interested in guile, guile 3.0 will have a JIT, making it even faster than 2.2 (by quite a large margin). Guile is becoming really nice, and I find myself reaching for other lisps less and less.

logicchains 8y ago
How far along is it towards becoming the underlying runtime for Emacs lisp?
- bjoli 8y ago
  With the bikeshedding we see for much less controversial changes in the Emacs codebase, I suspect it will be released around the release of Duke Nukem forever II. Elisp code runs fine under guile, but it has seen virtually no optimisation work and does a lot of unnecessary work. I suspect there is an lot of low hanging fruit, but since there is virtually no interest in guile-emacs apart from in comments on HN and Reddit I doubt the situation will get much better. The guile VM is getting better and better though. Maybe they'll come around by guile 4 :)
  - yepguy 8y ago
    I've always thought someone should use guile to add support for elisp plugins to vim & neovim using their rpc interfaces. Then maybe if something like that existed the org-mode authors could be convinced to cleanly separate out the emacs-specific code, and we could have org-mode on vim using the same codebase as emacs. I have no idea how viable that is, but it would be pretty sweet. Too much for me to take on right now, but the idea has been rattling around my brain for a few years.
- equalunique 8y ago
  https://www.emacswiki.org/emacs/GuileEmacs https://www.emacswiki.org/emacs/GuileEmacs
zeveb 8y ago
Scheme is a neat little language, but it's really not ideal for writing full software systems. Indeed, Guile & Racket (back when it was PLT Scheme) are perfect exemplars of Greenspun's Tenth Rule: Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. E.g. GOOPS is similar to CLOS, but missing a bit and unsupported other than in Guile. Meanwhile, implementations like SBCL are free, compile to machine code and are competitive on some benchmarks with C/C++. Common Lisp — while it does have some warts — is intended for building long-lived, durable, full-featured systems. It's a really good language, and it standardises quite a lot. Often, what look like warts are actually useful features in large systems. It really saddens me that so much effort has been expended trying to make Scheme a reasonable language for large systems, when there already exists one which is. That doesn't even get to the bits of Scheme which I believe are mistakes (e.g. a single namespace, making NIL, () & #f different).

arminiusreturns 8y ago

This is awesome! I have really been impressed with gnu's use of guile, and I really like how much more quickly iteration is than many other gnu projects. For those heavy cron users, mcron (guile based cron replacement) has been my favorite so far. With a good deamon manager, we're one step closer to HURD!

tomkat0789 8y ago

Pardon me, I'm sort of a casual Linux user and I have no idea what people are talking about. What are GNU Guix and Shepherd? A cursory glance at Wikipedia makes it seem like they're GNU alternatives to the apt-get and init/systemD I've been on for awhile. I'm outside of my Debian comfort zone, apologies for my ignorance! What's significant about this?

pjmlp 8y ago
Basically they follow similar ideas to NixOS, but using Scheme instead of a custom script language.

theamk 8y ago

Does anyone know if they solved their "trust init script too much" problems? Last time I checked, they were loading user-defined scripts right into PID 1, so a badly written script could bring down the whole system. People blame systemd for pid 1 instability, but at least systemd crashes get fixed eventually. With Shepherd, any package on the system can cause PID 1 to crash, and this just seems to crazy for me.

lfam 8y ago
Can you give any more detail about this? Or a reference to a bug report or mailing list thread?
- theamk 8y ago
  The manual [1] starts by saying: "When shepherd gets started, it reads and evaluates a configuration file." That config is full-featured, seemingly unrestricted, lisp file. While I saw some mentions of handling syntax errors on the mailing list, I saw no mentions of what happens if there is an infinite loop or if a script just reads a huge files and OOM's the process. Now, one could imagine that maybe configs are parsed in a separate process, and then serialized and shipped to the main process for safe execution. But it does not seem to be the case -- the manual explicitly says you can use any Guile code to start processes, not just a safe subset. Another option is that maybe there is a thread or a forked process for parsing config. But this does not seem to be the case either -- the manual does not mention that, and searching of the source code for "fork" or "thread" or "timeout" does not seem to indicate anything related. The only mention of timeness expectation seem to be a single sentence "Actions are expected to be non-blocking." in one of the mailing lists [2]. So together, it seems to me that any bad script will just cause the daemon, and by extension the whole system, to not start. [1] "https://www.gnu.org/software/shepherd/manual/shepherd.html#Services" https://www.gnu.org/software/shepherd/manual/shepherd.html#S... [2] http://lists.gnu.org/archive/html/guix-devel/2016-11/msg00945.html http://lists.gnu.org/archive/html/guix-devel/2016-11/msg0094...

_emacsomancer_ 8y ago

There's a quote from the GNU Shepherd manual I really appreciated: > I decided to not make it too sophisticated by trying to guess what the user might want just to theoretically fulfill the request we are processing. If something goes wrong, it is usually better to tell the user about the problem and let her fix it, taking care to make finding solutions or workarounds for problems (like a misconfigured service) easy. This way, the user is in control of what happens and we can keep the implementation simple. To make a long story short, we don't try to be too clever, which is usually a good idea in developing software.