5 ms·

I don't understand why people use Gentoo in production servers. Is it good practice? I can see so many downsides.

by jakobdabo 9y ago

sgtmas2006 9y ago
Like what?
- switch007 9y ago
  Well it's not an OS upon which they "run a comprehensive functional, regression, and stress test suite ... on a continuous basis" for starters (CentOS and Ubuntu, only) http://docs.ceph.com/docs/master/start/os-recommendations/ http://docs.ceph.com/docs/master/start/os-recommendations/
- kitotik 9y ago
  It’s almost impossible to achieve any sort of reproducibility when all packages are built from source. Portage is a great tool, but I always thought it was better suited to build packages and operating systems as opposed being an end user package management solution.
  - KirinDave 9y ago
    I've learned recently that even using Nix when building from source is something of a false hope of reproducibility. Many packages use the fetchgit function and fetchgit violates the contract.
    - Filligree 9y ago
      fetchgit does no such thing. To use it, you need to specify the git revision and the hash of the output; it's a constant-output derivation just like fetchurl. Reproducbility is still a difficult problem, but that isn't one of them, and Nix comes far closer than Gentoo.
      - KirinDave 9y ago
        https://twitter.com/shlevy/status/956313877800566784?ref_src=twcamp%5Ecopy%7Ctwsrc%5Eandroid%7Ctwgr%5Ecopy%7Ctwcon%5E7090%7Ctwterm%5E0 https://twitter.com/shlevy/status/956313877800566784?ref_src...
        mst 9y ago
        I do hope somebody either makes that start warning, then removes it, or at least adds it to a linter of some sort.
  - korethr 9y ago
    If I am correctly recalling a conversation I had with Gentoo founder Daniel Robbins on IRC a couple years back, Portage was originally intended as a tool for building binary packages, not necessarily the end-user package management. As for reproducibility when building from source, that depends on your practices. Is every box a special snowflake with it's own USE flags and package set? Well then yes, good luck with reproducibility. Or are you standardizing your targets and building one package to deploy to them all?
    - kitotik 9y ago
      Build a gentoo system each day for seven days on the same hardware with the same configuration(package versions, USE flags, etc) and see how well they match up.
      - korethr 9y ago
        Okay, I'm tempted now, partly out of a someone-might-be-wrong-on-the-Internet cussedness, and partly out of curiosity. By what metrics would you propose measuring how consistent the systems are or are not?
        TheDong 9y ago
        Binpkgs output from builds on the same machine are often bit-for-bit identical. It, of course, depends on the project; if the ./configure script drops the current date in source code, it'll be different, but that doesn't happen often.
      - laumars 9y ago
        I have done this. I used to manage a fleet of 50 servers with everything built from source aside the base Linux install and build utilities. They were all identical.
  - darkr 9y ago
    Hard, but not impossible: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds
- old-gregg 9y ago
  ...like saying "5 days of downtime" followed by having to "rebuild the world"?
- jakobdabo 9y ago
  Like you have to regularly compile and build big packages like Glibc (while under load) just to update the system. Also, as it's a rolling release distribution, as I understand you won't have stable enough software (an update might require configuration file change, this doesn't happen in CentOS or Debian Stable).
  - laumars 9y ago
    I wish people would quit peddling the "rolling release means less stable" trope because it's simply untrue. Many rolling release distros tend to be bleeding edge, _thats_ why they might be unstable (though honestly I've never had ArchLinux crash for any reason that wasn't my own fault). What's more, CentOS and Debian do sometimes have updates they require configuration changes. That's why Debian likes to separate out base config files from ones that are likely to change. So you can put your preferential overrides in a .local (iirc) file and the master version is free to be upgraded. Eg This is how fail2ban works on Debian.
    - peterwwillis 9y ago
      Rolling release is unstable. You have absolutely no idea if any given update will crash some part of your system, because every variable of the known use cases has not yet been re-tested and certified. Software and firmware updates from Linux distros have recently bricked modern hardware several times. This should never have happened, because a period of testing is necessary to evaluate whether a given update will cause failures. With rolling release, you update and pray.
      - laumars 9y ago
        Again, what you're describing is bleeding edge; not rolling release.
    - jeremyjh 9y ago
      I had three breaking issues in a year and a half using Arch. By breaking - I mean some piece of software I rely on quit working reliably after an update. These were all upstream bugs - not Arch's fault (KDE in two cases, Erlang/OTP in the other), but Ubuntu and RHEL never saw these versions of software hit their repositories because the problems were found (in at least one case by Arch and Gentoo users) and fixed in upstream long before they made it through the long release and testing cycles. The nice thing about Arch, its absolutely trivial to rollback a package, or even rollback all your packages to the ones published on any given date. So I was able to deal with all of these issues quite effectively which is a credit to Arch, but it simply is the case that you will see more breakage on the leading edge.
hathawsh 9y ago
Compared with Debian or Red Hat, Gentoo is really a meta-distribution. It's a nice basis for creating your own distribution. In theory, Gentoo lets you mix the latest version of some packages with well-tested, old versions of other packages and generate a release perfected for your organization. If the Gentoo release of some package, Ceph for example, isn't as up to date as you want, you can easily update it yourself and publish a new release to your servers. In practice, many people don't understand that Gentoo is a meta-distribution and try to run it like a simple distribution, leading to arbitrary ABI breakage and chaos. If you want to run Gentoo on servers, you need to test all upgrades on build machines before rolling them out. You need to publish your own binary packages. You need to do a lot of the testing that Debian, Ubuntu, Red Hat, etc. would be doing for you. I have some friends who set up their own hosting business based on Gentoo and they seem to be doing it right. Their servers are very stable. Testing updates is an ongoing expense, but I think it has worked out well for them.
- jakobdabo 9y ago
  Thank you for this information. So the bottom line is that, when comparing to more traditional distros, it's a lot of work to do it right.
- mastax 9y ago
  Thank you, that's a very interesting point. I suppose shared hosting providers already do their own builds of PHP, MySQL, Apache, etc. so a distro that's designed for that can work well.
- cies 9y ago
  I do not want to sound annoying, but this is how I read your very informative comment: > Compared with Debian or Red Hat, Gentoo is really a meta-distribution. It's a nice basis for creating your own distribution. I do not want that on my production server. We're in the $SOME_BACKEND_SERVICE business, not in the distro creating business. > In theory, Gentoo lets you mix the latest version of some packages with well-tested, old versions of other packages and generate a release perfected for your organization. I want a hyper stable base that is tested by the masses and when possible has some big org behind it who's supporting it (and thus very conservative when changing stuff). On top of that base I might roll my own packages or include a fringy package repository. Never I want the base to be different for everyone. > If the Gentoo release of some package, Ceph for example, isn't as up to date as you want, you can easily update it yourself and publish a new release to your servers. This is not too hard for any distro I've used. > In practice, many people don't understand that Gentoo is a meta-distribution and try to run it like a simple distribution, leading to arbitrary ABI breakage and chaos. If you want to run Gentoo on servers, you need to test all upgrades on build machines before rolling them out. You need to publish your own binary packages. You need to do a lot of the testing that Debian, Ubuntu, Red Hat, etc. would be doing for you. So may we conclude, as GP said, that running Gentoo on production servers is just not a good idea? > I have some friends who set up their own hosting business based on Gentoo and they seem to be doing it right. Their servers are very stable. This is a little too anecdotal for me :) I've done my share of Gentoo, and this system in intriguing. But when I need stability, as I need for backend services, then it will not even be on the long list. I found some similar-level intriguing features in NixOS. I'd probably still not run it for backend services, but I might be compelled to run it on the application layer in some cases.

I don't understand why people use Gentoo in production servers. Is it good practice? I can see so many downsides.

sgtmas2006 9y ago

Like what?

switch007 9y ago
Well it's not an OS upon which they "run a comprehensive functional, regression, and stress test suite ... on a continuous basis" for starters (CentOS and Ubuntu, only) http://docs.ceph.com/docs/master/start/os-recommendations/ http://docs.ceph.com/docs/master/start/os-recommendations/
kitotik 9y ago
It’s almost impossible to achieve any sort of reproducibility when all packages are built from source. Portage is a great tool, but I always thought it was better suited to build packages and operating systems as opposed being an end user package management solution.
- KirinDave 9y ago
  I've learned recently that even using Nix when building from source is something of a false hope of reproducibility. Many packages use the fetchgit function and fetchgit violates the contract.
  - Filligree 9y ago
    fetchgit does no such thing. To use it, you need to specify the git revision and the hash of the output; it's a constant-output derivation just like fetchurl. Reproducbility is still a difficult problem, but that isn't one of them, and Nix comes far closer than Gentoo.
    - KirinDave 9y ago
      https://twitter.com/shlevy/status/956313877800566784?ref_src=twcamp%5Ecopy%7Ctwsrc%5Eandroid%7Ctwgr%5Ecopy%7Ctwcon%5E7090%7Ctwterm%5E0 https://twitter.com/shlevy/status/956313877800566784?ref_src...
      - mst 9y ago
        I do hope somebody either makes that start warning, then removes it, or at least adds it to a linter of some sort.
- korethr 9y ago
  If I am correctly recalling a conversation I had with Gentoo founder Daniel Robbins on IRC a couple years back, Portage was originally intended as a tool for building binary packages, not necessarily the end-user package management. As for reproducibility when building from source, that depends on your practices. Is every box a special snowflake with it's own USE flags and package set? Well then yes, good luck with reproducibility. Or are you standardizing your targets and building one package to deploy to them all?
  - kitotik 9y ago
    Build a gentoo system each day for seven days on the same hardware with the same configuration(package versions, USE flags, etc) and see how well they match up.
    - korethr 9y ago
      Okay, I'm tempted now, partly out of a someone-might-be-wrong-on-the-Internet cussedness, and partly out of curiosity. By what metrics would you propose measuring how consistent the systems are or are not?
      - TheDong 9y ago
        Binpkgs output from builds on the same machine are often bit-for-bit identical. It, of course, depends on the project; if the ./configure script drops the current date in source code, it'll be different, but that doesn't happen often.
    - laumars 9y ago
      I have done this. I used to manage a fleet of 50 servers with everything built from source aside the base Linux install and build utilities. They were all identical.
- darkr 9y ago
  Hard, but not impossible: https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds
old-gregg 9y ago
...like saying "5 days of downtime" followed by having to "rebuild the world"?
jakobdabo 9y ago
Like you have to regularly compile and build big packages like Glibc (while under load) just to update the system. Also, as it's a rolling release distribution, as I understand you won't have stable enough software (an update might require configuration file change, this doesn't happen in CentOS or Debian Stable).
- laumars 9y ago
  I wish people would quit peddling the "rolling release means less stable" trope because it's simply untrue. Many rolling release distros tend to be bleeding edge, _thats_ why they might be unstable (though honestly I've never had ArchLinux crash for any reason that wasn't my own fault). What's more, CentOS and Debian do sometimes have updates they require configuration changes. That's why Debian likes to separate out base config files from ones that are likely to change. So you can put your preferential overrides in a .local (iirc) file and the master version is free to be upgraded. Eg This is how fail2ban works on Debian.
  - peterwwillis 9y ago
    Rolling release is unstable. You have absolutely no idea if any given update will crash some part of your system, because every variable of the known use cases has not yet been re-tested and certified. Software and firmware updates from Linux distros have recently bricked modern hardware several times. This should never have happened, because a period of testing is necessary to evaluate whether a given update will cause failures. With rolling release, you update and pray.
    - laumars 9y ago
      Again, what you're describing is bleeding edge; not rolling release.
  - jeremyjh 9y ago
    I had three breaking issues in a year and a half using Arch. By breaking - I mean some piece of software I rely on quit working reliably after an update. These were all upstream bugs - not Arch's fault (KDE in two cases, Erlang/OTP in the other), but Ubuntu and RHEL never saw these versions of software hit their repositories because the problems were found (in at least one case by Arch and Gentoo users) and fixed in upstream long before they made it through the long release and testing cycles. The nice thing about Arch, its absolutely trivial to rollback a package, or even rollback all your packages to the ones published on any given date. So I was able to deal with all of these issues quite effectively which is a credit to Arch, but it simply is the case that you will see more breakage on the leading edge.

hathawsh 9y ago

Compared with Debian or Red Hat, Gentoo is really a meta-distribution. It's a nice basis for creating your own distribution. In theory, Gentoo lets you mix the latest version of some packages with well-tested, old versions of other packages and generate a release perfected for your organization. If the Gentoo release of some package, Ceph for example, isn't as up to date as you want, you can easily update it yourself and publish a new release to your servers. In practice, many people don't understand that Gentoo is a meta-distribution and try to run it like a simple distribution, leading to arbitrary ABI breakage and chaos. If you want to run Gentoo on servers, you need to test all upgrades on build machines before rolling them out. You need to publish your own binary packages. You need to do a lot of the testing that Debian, Ubuntu, Red Hat, etc. would be doing for you. I have some friends who set up their own hosting business based on Gentoo and they seem to be doing it right. Their servers are very stable. Testing updates is an ongoing expense, but I think it has worked out well for them.

jakobdabo 9y ago
Thank you for this information. So the bottom line is that, when comparing to more traditional distros, it's a lot of work to do it right.
mastax 9y ago
Thank you, that's a very interesting point. I suppose shared hosting providers already do their own builds of PHP, MySQL, Apache, etc. so a distro that's designed for that can work well.
cies 9y ago
I do not want to sound annoying, but this is how I read your very informative comment: > Compared with Debian or Red Hat, Gentoo is really a meta-distribution. It's a nice basis for creating your own distribution. I do not want that on my production server. We're in the $SOME_BACKEND_SERVICE business, not in the distro creating business. > In theory, Gentoo lets you mix the latest version of some packages with well-tested, old versions of other packages and generate a release perfected for your organization. I want a hyper stable base that is tested by the masses and when possible has some big org behind it who's supporting it (and thus very conservative when changing stuff). On top of that base I might roll my own packages or include a fringy package repository. Never I want the base to be different for everyone. > If the Gentoo release of some package, Ceph for example, isn't as up to date as you want, you can easily update it yourself and publish a new release to your servers. This is not too hard for any distro I've used. > In practice, many people don't understand that Gentoo is a meta-distribution and try to run it like a simple distribution, leading to arbitrary ABI breakage and chaos. If you want to run Gentoo on servers, you need to test all upgrades on build machines before rolling them out. You need to publish your own binary packages. You need to do a lot of the testing that Debian, Ubuntu, Red Hat, etc. would be doing for you. So may we conclude, as GP said, that running Gentoo on production servers is just not a good idea? > I have some friends who set up their own hosting business based on Gentoo and they seem to be doing it right. Their servers are very stable. This is a little too anecdotal for me :) I've done my share of Gentoo, and this system in intriguing. But when I need stability, as I need for backend services, then it will not even be on the long list. I found some similar-level intriguing features in NixOS. I'd probably still not run it for backend services, but I might be compelled to run it on the application layer in some cases.