14 ms·

Fermilab/CERN recommendation for Linux distribution

225 points by jaboutboul 4y ago

bayesian_horse 4y ago
What I didn't like about AlmaLinux so far is that it takes more effort to google solutions to problems than for Ubuntu. That may sound stupid or lazy, but there it is.
- keewee7 4y ago
  AlmaLinux is highly compatible with Fedora and CentOS. You should be able to use the same solutions.
  - bayesian_horse 4y ago
    I am. The impression remains.
- flumpcakes 4y ago
  Ubuntu was/is by far the more popular distro to use outside of the corporate world. There's a lot more blogs, container base images, tutorials out there because it was popular for tinkerers. This is also the reason it has now become popular inside the corporate world. You may find it harder to find resources for AlmaLinux/CentOS/etc. if you are using the same methods for searching for resources as you would for a Ubuntu (or even Debian) based OS. However, there are much better and more fully comprehensive training material out there for 'Enterprise Linux' distributions (RHEL, CentOS which means also AlmaLinux and Rocky) because of the widely industry accepted RH certifications. So to wildly make broad statements: Ubuntu based distros are good for finding resources like blogs and tutorials that are aimed at people who are tinkering and just starting out, maybe from a dev background. EL distros are good for comprehensive training for people who probably have had "Sys admin" as a job title at some point.
  - KronisLV 4y ago
    > You may find it harder to find resources for AlmaLinux/CentOS/etc. if you are using the same methods for searching for resources as you would for a Ubuntu (or even Debian) based OS. Not only that, but as soon as you venture outside of the "happy path", you'll run into difficulties if others haven't also solved those problems ahead of time, which can be the case with many DEB distros but isn't always so with RPM ones. For example, in many RPM distros you'd typically reach for using Podman for running container workloads, maybe even use something like OpenShift (their MicroShift project still being in the works). In contrast, on DEB you'd at least sometimes go with Docker, because it's widely known, widely used and still has lots of market share - offering such lighweight orchestrators like Nomad or even Docker Swarm out of the box, which still is an okay choice for getting a dev environment running on a few boxes with 8 GB of RAM, when you don't have a separate node for managing the cluster. Now suppose that you decide to go with Docker on RPM. It's definitely feasible and has gotten better, but I recall just a few years back Docker having issues with SELinux and also the container networking utterly breaking because of firewall configuration. You had to use the masquerade option if I recall correctly, which definitely fixed the issue but also meant that the software package ships broken by default, in a sense. On an unrelated note, I recently ran into an issue with one of the RPM distros a week back, where if the node has swap enabled and it runs low on swap memory, it's going to have the kswapd process start eating a lot of CPU resources and basically bring everything else on the node on a standstill. Not to say that RPM is worse, the LTS EOL cycle alone is amazing, so I wish Rocky and Alma (As well as other distros) the best of luck! But I can also definitely recognize why many would stick with Debian or Ubuntu (or other DEB distros).
    - rascul 4y ago
      It seems like you might be conflating RPM distros with RHEL and it's clones, which seems to me a somewhat common thing to do. Suse distros are independent of Red Hat and have a different set of pros and cons, but I don't recall seeing or experiencing the ones you mention about docker or kswapd.
      - KronisLV 4y ago
- vbezhenar 4y ago
  Half of my searches are thoroughly explained in access.redhat by very competent RH support answers. Ubuntu answers usually are just bad.
- jeffbee 4y ago
  It’s totally impossible to google the solutions for Ubuntu because they’ve completely changed the way everything works, multiple times, so all the answers are wrong.
HexDecOctBin 4y ago
> reflecting recent experience What happened? Is this about CentOS's demise?
- cozzyd 4y ago
  Yes. It's too bad because they used to support Scientific Linux, but stopped because CentOS was so popular... Anyway, I'm glad I went with Alma over Rocky, albeit for the wrong reasons (my daughter's name is Alma...)
  - sgc 4y ago
    Sounds like a wholesome reason to me.
  - thiagocsf 4y ago
    You probably know this, but Alma means “soul” in Portuguese. Is this why you picked the name? I like it. I went with Ada myself (after Lovelace - wife only found-out why years later).
    - 5e92cb50239222b 4y ago
      FWIW, it also means "apple" in several Turkic languages (Kazakh, Kyrgyz, Turkmen, Azerbaijani, and maybe others). Uzbek and Turkish are also pretty close.
    - cozzyd 4y ago
      I was aware of the Spanish meaning which is the same, but the reason we picked it was more mundane. It was one of the few names my wife and I both liked a lot. ALMA is also a radio telescope in Chile (the Atacama Large Millimeter Array), which is somewhat closer afield to what I do.
    - NotYourLawyer 4y ago
      Ha, I also suggested Ada for the same reason. And also didn’t tell my wife why. But we ended up not using it.
djha-skin 4y ago
All this time I had thought that Rocky Linux was winning the fork war over Alma (in the fight to be the successor to CentOS), but this post might change that with a good chunk of the science community throwing their weight behind Alma. Do we have anyone else in the audience that has any insights over whether Alma is more prevalent over Rocky or is it the other way around? I know I can run Rocky Linux from DigitalOcean which I appreciate.
- sliken 4y ago
  We picked Rocky Linux because it was the first to get full support from Mellanox/Nvidia.
- darthrupert 4y ago
  > Alma Having just played through the F.E.A.R. games that name gives me weird feelings.
  - themodelplumber 4y ago
    Having been assigned the name in a creepy religious temple ceremony we are in the same boat
  - pelagicAustral 4y ago
    I love that game, but the first thing that came to mind was the satellite dish array somewhere in the desert.
  - actualwitch 4y ago
    Having just watched through undone series that name also gives me weird feelings.
  - spookie 4y ago
    Means soul in Spanish/Portuguese
- friendzis 4y ago
  > Do we have anyone else in the audience that has any insights over whether Alma is more prevalent over Rocky or is it the other way around? This is wrong reasoning. Distributions live on the shoulders of maintainers/developers. While popularity may be a factor attracting maintainers/developers it is not a deciding factor. If the less popular distribution attracts more maintenance/development effort (again, not necessarily head count) it will win back market share in medium term and once popular distribution will stagnate and fade.
notorandit 4y ago
Not really interesting. Without any technical reasoning other than "we want red hat with extra perks".
- djha-skin 4y ago
  It's interesting because there's a question in the air right now over what fork of CentOS will be its spiritual successor since it got bought out. Will it be Alma, or Rocky? This post holds in favor of Alma.
  - dotancohen 4y ago
    As much as it pains me to say it, FOSS devs are a finite resource and it seems that being divided is just as if having been conquered. Neither Rocky nor Alma are progressing at the rate that a CentOS fork with a hostile RH upstream would progress if they were working together and not duplicating efforts. What is the philosophical schism between the two? Why can't they split the workload on a single distro and reunite the communities?
    - liamnal 4y ago
      It's just like with any distribution, each have their own philosophies and how they want to tackle something. Oracle Linux is still around for example, but we don't talk about them too often. Rocky seems to have mostly volunteers. Alma has people who are likely paid because they come from or likely still work for CloudLinux. Either way, you're getting a RHEL clone if you go with either. And that is keeping in mind that devs are a finite resource and always will be, regardless of how you look at it. The thing to keep in mind is that more choices are better than just having one. Think about it, let's say Scientific Linux actually made an 8. CentOS users would've likely gone over to them instead at the EOL date. Since SL didn't keep going, there was only CentOS (and Oracle Linux, but again no one really wants to talk about that - and there are folks who will avoid oracle like the plague; I don't blame them). You take away the one distribution a large amount of folks used and where are they going to go? Stream? Fermilab/CERN were going to stream and... not anymore. Perhaps the bugs and instability were a bit too much. The long story short of it is, you don't want another situation like CentOS. More options, the better. All EL derivatives/clones should all operate and work the same.
1letterunixname 4y ago
I went through the Rocks Cluster cycles in 3, 4, and 5 with SLC and Scientific until those fell by the wayside. Rocky is the underdog you want to win. Alma is the leader except in terms of security update latency. The problem is that Cent 8/9 Stream has quicker critical CVE patches because it's essentially the source and is closer to mirroring RHEL. It's hard to convince corporate folks to use Alma when Cent is still the "safer" choice technologically and provides continuity even if its governance and lifecycle maybe worse.
- Proven 4y ago
- jacooper 4y ago
  Apperantly stream is the slowest. https://news.ycombinator.com/item?id=33905616 https://news.ycombinator.com/item?id=33905616
  - carlwgeorge 4y ago
    Not exactly. For most CVEs, CentOS Stream gets them months before RHEL and RHEL clones. But CVEs rated important/critical (or otherwise embargoed) are required to go out to RHEL customers first. Once the fix is live for RHEL customers, two things happen more or less simultaneously: - The RHEL package source with the fix is published, allowing clone distros to start their rebuild work. - RHEL maintainer starts working on the fix for CentOS Stream in public. This may or may not be the same patch as what was released in RHEL, depending on whatever other changes have already happened in CentOS Stream. These tasks are not the same and take different amounts of time. On top of that, the release pipelines to get the fix out to users are different between the various distros. Sometimes the fix is live for CentOS Stream before the rebuilds, sometimes after. There were some notably slower exceptions with CentOS Stream 8 in the early days, because CentOS Stream 8 is built "inside out". https://twitter.com/carlwgeorge/status/1439724296742576130 https://twitter.com/carlwgeorge/status/1439724296742576130 In CentOS Stream 9 going forward, RHEL maintainers own their CentOS builds and things are working much better. There is work in progress to migrate 8 workflows to match 9.
Mikhail_K 4y ago
Is AlmaLinux systemd-free?
- Dracophoenix 4y ago
  It's a RHEL clone, so no. However, you can install runit or OpenRC if you want.
  - Mikhail_K 4y ago
    That doesn't solve the problem, because systemd developers have insinuated dependencies on it into multiple unrelated system components.
    - usr1106 4y ago
      Linux developers have created dependencies on Linux. Portability is a lot of work and perfect portability does not exist. If you think you don't want systemd, why don't you use BSD?
      - dijit 4y ago
        > Linux developers have created dependencies on Linux. Sorry, ignoring the rest of the thread: this statement isn't true even marginally. There are linux distributions that can operate on other kernels than Linux. Debian was my favourite choice for this with their HURD (https://www.debian.org/ports/hurd/ https://www.debian.org/ports/hurd/) and kFreeBSD (https://www.debian.org/ports/kfreebsd-gnu/ https://www.debian.org/ports/kfreebsd-gnu/) implementation. The GNU userland itself does not depend on Linux either. You can run GNU utilities on MacOS, FreeBSD, OpenBSD.
        flumpcakes 4y ago
        > There are linux distributions that can operate on other kernels than Linux. Then by definition they're not specifically a Linux distribution?
keewee7 4y ago
How are these long-term support Linux distros able to provide LTS for the thousands of packages they include? If a security bug is discovered in an old version of RabbitMQ do the distro maintainers learn Erlang/OTP to patch the bug?
- IceWreck 4y ago
  Alma/Rocky are just rebuilds of RHEL. And afaik, Red Hat does employ people to do that.
  - jabl 4y ago
    And, the package selection in RHEL is much smaller than in, say, Debian or Ubuntu [1]. There are the popular EPEL repositories with lots of extra packages, but these are community-maintained on a best-effort basis. [1] And I think the Ubuntu 'guarantees' only apply to the 'main' repository, not the MUCH larger 'universe'.
qwerty456127 4y ago
Obviously stability is the most important factor for the people running nuclear experiments so they choose RHEL. Nevertheless, if I were them, I would consider trying Nix or Guix to potrntially turn the configs/builds into comparably strict math and making these reproducible.
- acatton 4y ago
  Config can also be made reproducible with VM images, containers, home-built packages, configuration-management tools (ansible, puppet, …) or a mix of all or some of them. Nix/Guix is no silverbullet.
  - amelius 4y ago
    How do you grow your system, or do you suggest that you build your system as a bunch of microservices running inside containers?
    - acatton 4y ago
      What do you mean growing your system? “How do you run more service?” Or “how do you handle more traffic?” Or “how do you handle more complexity of the application?” All of these can be handled with any or many of VM images, containers, home-built packages and configuration-management tools. If you're following the latest trend, you just run VM based on an image with pre-installed kubernetes. Want to run more services? Just deploy more services on your k8s cluster. Want to scale horizontally to handle more traffic? Just boot up more VMs with the same image, and increase the size of your k8s pods. Want to handle multiple more complexity, split your job into micro services, and give responsibility of different namespaces to different teams. If you're using a late 2000s, early 2010s model, you would just use VM, and boot them when necessary. You have a VM with HAProxy with dynamic backends. Want to run more services? Just add more to the base image and route between them with a nginx local to your base VM image. Want to handle more traffic, just spin up more VMs with the base image. Want to handle more complexity? Use multiple types of VMs with different base images, depending on the service, and give the responsability to each team for configuring the base image. I can go on and on for packages and/or configuration-management tools. But I think you get the idea.
xorcist 4y ago
Alma produces a rather snazzy script to migrate from CentOS which works really well. I'm told it also works with Rocky, but I haven't tried it.
- bclemens 4y ago
  Rocky Linux has migration scripts available at https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky https://github.com/rocky-linux/rocky-tools/tree/main/migrate... An easy shortcut though, in case you have to hand type it out somewhere, is https://rockylinux.org/migrate2rocky.sh https://rockylinux.org/migrate2rocky.sh
  - jonathanspw 4y ago
    Pretty sure he was talking about ELevate to go from EL7 to Alma 8.
fancyfredbot 4y ago
What happened to scientific Linux? It's a bit odd that it seems to be being discontinued rather than rebased on Alma? Do they no longer believe scientific Linux adds value?
- brnt 4y ago
  It had been discontinued a while ago, in favour of a flavour of CentOS (at the time CentOS 7 was released, instead of Scientific Linux 7).
jl6 4y ago
I’m out of the loop on RHEL forks. Can someone explain why it is important to be binary compatible with RHEL? Is there a lot of software whose binaries will only run on RHEL? Or is this more a case of wanting something free that can still make use of RHEL-specific skills and knowledge?
- sllabres 4y ago
  As soon as you use something like SAP or Oracle the number of offically _supported_ distributions quickly decreases. As example SAP HANA: RHEL and SLES or for example Oracle Database: RHEL, OEL (based on RHEL AFAIK) and again SLES. So it is not only a question if it would theoretically/technically fail on another platform.
  - jl6 4y ago
    I don’t imagine SAP certification would extend to a community-supported fork of RHEL even if it was a 1:1 clone.
    - Moose_the_Goose 4y ago
      Many businesses will use the Forks for their dev environment.
      - carlwgeorge 4y ago
        Which is silly because Red Hat will literally give you free RHEL for non-prod if you pay for RHEL in prod. https://developers.redhat.com/articles/2022/05/10/access-rhel-developer-teams-subscription https://developers.redhat.com/articles/2022/05/10/access-rhe...
- Symmetry 4y ago
  When I was working for the US government we had to obey rules against shipping "freeware," which RHEL wasn't because we payed for it. But it was cheaper and easier to do most of our development on CentOS systems and because the software was the same we weren't worried about compatibility issues.
hsbauauvhabzb 4y ago
I would prefer to use RedHat based distros but their lack of official non free repos annoy me - I don’t want to add a community repo, I want packages maintained by trusted core engineers. Sure the community repos probably have that, but I last I checked I couldn’t find any assurance on the security of the community repos. SBOMs don’t mean shit if you’re installing stuff from random no-assurance locations.
- ilyt 4y ago
  Then you want Debian. We've had FAR less problems with Debians than anything from Red Hat stable
  - MisterTea 4y ago
    I've used a bunch of Linux distros over the years and always find myself installing Debian when I just want a desktop.
    - hsbauauvhabzb 4y ago
      I’m currently on Ubuntu and have considered Debian. I like the concept of containerisation (snap, flatpack, etc) but the overly pushy snap integration has broken my workflows in multiple situations without much redemption so maybe I will give it a try.
      - throw_a_grenade 4y ago
        If you want "just desktop" like grandparent, you can try https://linuxmint.com/ https://linuxmint.com/, it's downstream from Ubuntu LTS, but with bad stuff (snap) removed: https://linuxmint-user-guide.readthedocs.io/en/latest/snap.html https://linuxmint-user-guide.readthedocs.io/en/latest/snap.h...
        hsbauauvhabzb 4y ago
        Yeah I’m all for those distros, but as I work in security industry i kind of feel like a ‘nobody ever got fired for buying an ibm approach’ is good. If I’m hacked it’s not a good look at all, if im hacked and use an esoteric distro like gentoo, it would certainly look much, much worse. My key pain point is trust within repos, Ubuntu audit their repos as best they can (sast/sanity check) so at least there’s some security there. Im otherwise very supportive of mint/arch/gentoo and similar systems.
jacooper 4y ago
And after all of that, dnf auto update still doesn't support auto restarts for kernel updates, amazing. Also no in-place upgrades since for ever.
- vbezhenar 4y ago
  I'm afraid to even update ordinary packages on a running system. People who want to update kernel on the fly are just reckless!
  - jacooper 4y ago
    Since everything is already running as a container, it really doesn't matter
    - vbezhenar 4y ago
      Containers are ordinary processes, just with some namespacing for separation. And not everything is running in a container.
warinukraine 4y ago
Not a linux expert. Skimming wikipedia it seems the main selling point of both Rocky and Alma is that they're binary-compatible with red hat, right? Could some experts clarify: if you want to be binary-compatible with red hat, why not just use red had?
- zamalek 4y ago
  RHEL costs money.
  - warinukraine 4y ago
    Yikes. Why do people use it exactly? (I use debian, that's all I know)
    - Moose_the_Goose 4y ago
      RHEL is used by companies that need the support Red Hat provides. RHEL and it's derivatives have less packages in the repo due to Red Hat willing to support every package they ship.
grysgt 4y ago
jeffbee 4y ago
I bet these guys have a lot of unusual legacy requirements, like wanting Motif and remote X11/Xdm to keep working, which takes Ubuntu out of the competition.
egberts1 4y ago
Absolutely love the integrated signing of containers (Sigstore technology). If more distros support this, then our life would be far less stressful. https://www.sigstore.dev/ https://www.sigstore.dev/

225 points by jaboutboul 4y ago

What I didn't like about AlmaLinux so far is that it takes more effort to google solutions to problems than for Ubuntu. That may sound stupid or lazy, but there it is.

keewee7 4y ago
AlmaLinux is highly compatible with Fedora and CentOS. You should be able to use the same solutions.
- bayesian_horse 4y ago
  I am. The impression remains.
flumpcakes 4y ago
Ubuntu was/is by far the more popular distro to use outside of the corporate world. There's a lot more blogs, container base images, tutorials out there because it was popular for tinkerers. This is also the reason it has now become popular inside the corporate world. You may find it harder to find resources for AlmaLinux/CentOS/etc. if you are using the same methods for searching for resources as you would for a Ubuntu (or even Debian) based OS. However, there are much better and more fully comprehensive training material out there for 'Enterprise Linux' distributions (RHEL, CentOS which means also AlmaLinux and Rocky) because of the widely industry accepted RH certifications. So to wildly make broad statements: Ubuntu based distros are good for finding resources like blogs and tutorials that are aimed at people who are tinkering and just starting out, maybe from a dev background. EL distros are good for comprehensive training for people who probably have had "Sys admin" as a job title at some point.
- KronisLV 4y ago
  > You may find it harder to find resources for AlmaLinux/CentOS/etc. if you are using the same methods for searching for resources as you would for a Ubuntu (or even Debian) based OS. Not only that, but as soon as you venture outside of the "happy path", you'll run into difficulties if others haven't also solved those problems ahead of time, which can be the case with many DEB distros but isn't always so with RPM ones. For example, in many RPM distros you'd typically reach for using Podman for running container workloads, maybe even use something like OpenShift (their MicroShift project still being in the works). In contrast, on DEB you'd at least sometimes go with Docker, because it's widely known, widely used and still has lots of market share - offering such lighweight orchestrators like Nomad or even Docker Swarm out of the box, which still is an okay choice for getting a dev environment running on a few boxes with 8 GB of RAM, when you don't have a separate node for managing the cluster. Now suppose that you decide to go with Docker on RPM. It's definitely feasible and has gotten better, but I recall just a few years back Docker having issues with SELinux and also the container networking utterly breaking because of firewall configuration. You had to use the masquerade option if I recall correctly, which definitely fixed the issue but also meant that the software package ships broken by default, in a sense. On an unrelated note, I recently ran into an issue with one of the RPM distros a week back, where if the node has swap enabled and it runs low on swap memory, it's going to have the kswapd process start eating a lot of CPU resources and basically bring everything else on the node on a standstill. Not to say that RPM is worse, the LTS EOL cycle alone is amazing, so I wish Rocky and Alma (As well as other distros) the best of luck! But I can also definitely recognize why many would stick with Debian or Ubuntu (or other DEB distros).
  - rascul 4y ago
    It seems like you might be conflating RPM distros with RHEL and it's clones, which seems to me a somewhat common thing to do. Suse distros are independent of Red Hat and have a different set of pros and cons, but I don't recall seeing or experiencing the ones you mention about docker or kswapd.
    - KronisLV 4y ago
vbezhenar 4y ago
Half of my searches are thoroughly explained in access.redhat by very competent RH support answers. Ubuntu answers usually are just bad.
jeffbee 4y ago
It’s totally impossible to google the solutions for Ubuntu because they’ve completely changed the way everything works, multiple times, so all the answers are wrong.

HexDecOctBin 4y ago

> reflecting recent experience What happened? Is this about CentOS's demise?

cozzyd 4y ago
Yes. It's too bad because they used to support Scientific Linux, but stopped because CentOS was so popular... Anyway, I'm glad I went with Alma over Rocky, albeit for the wrong reasons (my daughter's name is Alma...)
- sgc 4y ago
  Sounds like a wholesome reason to me.
- thiagocsf 4y ago
  You probably know this, but Alma means “soul” in Portuguese. Is this why you picked the name? I like it. I went with Ada myself (after Lovelace - wife only found-out why years later).
  - 5e92cb50239222b 4y ago
    FWIW, it also means "apple" in several Turkic languages (Kazakh, Kyrgyz, Turkmen, Azerbaijani, and maybe others). Uzbek and Turkish are also pretty close.
  - cozzyd 4y ago
    I was aware of the Spanish meaning which is the same, but the reason we picked it was more mundane. It was one of the few names my wife and I both liked a lot. ALMA is also a radio telescope in Chile (the Atacama Large Millimeter Array), which is somewhat closer afield to what I do.
  - NotYourLawyer 4y ago
    Ha, I also suggested Ada for the same reason. And also didn’t tell my wife why. But we ended up not using it.

djha-skin 4y ago

All this time I had thought that Rocky Linux was winning the fork war over Alma (in the fight to be the successor to CentOS), but this post might change that with a good chunk of the science community throwing their weight behind Alma. Do we have anyone else in the audience that has any insights over whether Alma is more prevalent over Rocky or is it the other way around? I know I can run Rocky Linux from DigitalOcean which I appreciate.

sliken 4y ago
We picked Rocky Linux because it was the first to get full support from Mellanox/Nvidia.
darthrupert 4y ago
> Alma Having just played through the F.E.A.R. games that name gives me weird feelings.
- themodelplumber 4y ago
  Having been assigned the name in a creepy religious temple ceremony we are in the same boat
- pelagicAustral 4y ago
  I love that game, but the first thing that came to mind was the satellite dish array somewhere in the desert.
- actualwitch 4y ago
  Having just watched through undone series that name also gives me weird feelings.
- spookie 4y ago
  Means soul in Spanish/Portuguese
friendzis 4y ago
> Do we have anyone else in the audience that has any insights over whether Alma is more prevalent over Rocky or is it the other way around? This is wrong reasoning. Distributions live on the shoulders of maintainers/developers. While popularity may be a factor attracting maintainers/developers it is not a deciding factor. If the less popular distribution attracts more maintenance/development effort (again, not necessarily head count) it will win back market share in medium term and once popular distribution will stagnate and fade.

notorandit 4y ago

Not really interesting. Without any technical reasoning other than "we want red hat with extra perks".

djha-skin 4y ago
It's interesting because there's a question in the air right now over what fork of CentOS will be its spiritual successor since it got bought out. Will it be Alma, or Rocky? This post holds in favor of Alma.
- dotancohen 4y ago
  As much as it pains me to say it, FOSS devs are a finite resource and it seems that being divided is just as if having been conquered. Neither Rocky nor Alma are progressing at the rate that a CentOS fork with a hostile RH upstream would progress if they were working together and not duplicating efforts. What is the philosophical schism between the two? Why can't they split the workload on a single distro and reunite the communities?
  - liamnal 4y ago
    It's just like with any distribution, each have their own philosophies and how they want to tackle something. Oracle Linux is still around for example, but we don't talk about them too often. Rocky seems to have mostly volunteers. Alma has people who are likely paid because they come from or likely still work for CloudLinux. Either way, you're getting a RHEL clone if you go with either. And that is keeping in mind that devs are a finite resource and always will be, regardless of how you look at it. The thing to keep in mind is that more choices are better than just having one. Think about it, let's say Scientific Linux actually made an 8. CentOS users would've likely gone over to them instead at the EOL date. Since SL didn't keep going, there was only CentOS (and Oracle Linux, but again no one really wants to talk about that - and there are folks who will avoid oracle like the plague; I don't blame them). You take away the one distribution a large amount of folks used and where are they going to go? Stream? Fermilab/CERN were going to stream and... not anymore. Perhaps the bugs and instability were a bit too much. The long story short of it is, you don't want another situation like CentOS. More options, the better. All EL derivatives/clones should all operate and work the same.

1letterunixname 4y ago

I went through the Rocks Cluster cycles in 3, 4, and 5 with SLC and Scientific until those fell by the wayside. Rocky is the underdog you want to win. Alma is the leader except in terms of security update latency. The problem is that Cent 8/9 Stream has quicker critical CVE patches because it's essentially the source and is closer to mirroring RHEL. It's hard to convince corporate folks to use Alma when Cent is still the "safer" choice technologically and provides continuity even if its governance and lifecycle maybe worse.

Proven 4y ago
jacooper 4y ago
Apperantly stream is the slowest. https://news.ycombinator.com/item?id=33905616 https://news.ycombinator.com/item?id=33905616
- carlwgeorge 4y ago
  Not exactly. For most CVEs, CentOS Stream gets them months before RHEL and RHEL clones. But CVEs rated important/critical (or otherwise embargoed) are required to go out to RHEL customers first. Once the fix is live for RHEL customers, two things happen more or less simultaneously: - The RHEL package source with the fix is published, allowing clone distros to start their rebuild work. - RHEL maintainer starts working on the fix for CentOS Stream in public. This may or may not be the same patch as what was released in RHEL, depending on whatever other changes have already happened in CentOS Stream. These tasks are not the same and take different amounts of time. On top of that, the release pipelines to get the fix out to users are different between the various distros. Sometimes the fix is live for CentOS Stream before the rebuilds, sometimes after. There were some notably slower exceptions with CentOS Stream 8 in the early days, because CentOS Stream 8 is built "inside out". https://twitter.com/carlwgeorge/status/1439724296742576130 https://twitter.com/carlwgeorge/status/1439724296742576130 In CentOS Stream 9 going forward, RHEL maintainers own their CentOS builds and things are working much better. There is work in progress to migrate 8 workflows to match 9.

Mikhail_K 4y ago

Is AlmaLinux systemd-free?

Dracophoenix 4y ago
It's a RHEL clone, so no. However, you can install runit or OpenRC if you want.
- Mikhail_K 4y ago
  That doesn't solve the problem, because systemd developers have insinuated dependencies on it into multiple unrelated system components.
  - usr1106 4y ago
    Linux developers have created dependencies on Linux. Portability is a lot of work and perfect portability does not exist. If you think you don't want systemd, why don't you use BSD?
    - dijit 4y ago
      > Linux developers have created dependencies on Linux. Sorry, ignoring the rest of the thread: this statement isn't true even marginally. There are linux distributions that can operate on other kernels than Linux. Debian was my favourite choice for this with their HURD (https://www.debian.org/ports/hurd/ https://www.debian.org/ports/hurd/) and kFreeBSD (https://www.debian.org/ports/kfreebsd-gnu/ https://www.debian.org/ports/kfreebsd-gnu/) implementation. The GNU userland itself does not depend on Linux either. You can run GNU utilities on MacOS, FreeBSD, OpenBSD.
      - flumpcakes 4y ago
        > There are linux distributions that can operate on other kernels than Linux. Then by definition they're not specifically a Linux distribution?

keewee7 4y ago

How are these long-term support Linux distros able to provide LTS for the thousands of packages they include? If a security bug is discovered in an old version of RabbitMQ do the distro maintainers learn Erlang/OTP to patch the bug?

IceWreck 4y ago
Alma/Rocky are just rebuilds of RHEL. And afaik, Red Hat does employ people to do that.
- jabl 4y ago
  And, the package selection in RHEL is much smaller than in, say, Debian or Ubuntu [1]. There are the popular EPEL repositories with lots of extra packages, but these are community-maintained on a best-effort basis. [1] And I think the Ubuntu 'guarantees' only apply to the 'main' repository, not the MUCH larger 'universe'.

qwerty456127 4y ago

Obviously stability is the most important factor for the people running nuclear experiments so they choose RHEL. Nevertheless, if I were them, I would consider trying Nix or Guix to potrntially turn the configs/builds into comparably strict math and making these reproducible.

acatton 4y ago
Config can also be made reproducible with VM images, containers, home-built packages, configuration-management tools (ansible, puppet, …) or a mix of all or some of them. Nix/Guix is no silverbullet.
- amelius 4y ago
  How do you grow your system, or do you suggest that you build your system as a bunch of microservices running inside containers?
  - acatton 4y ago
    What do you mean growing your system? “How do you run more service?” Or “how do you handle more traffic?” Or “how do you handle more complexity of the application?” All of these can be handled with any or many of VM images, containers, home-built packages and configuration-management tools. If you're following the latest trend, you just run VM based on an image with pre-installed kubernetes. Want to run more services? Just deploy more services on your k8s cluster. Want to scale horizontally to handle more traffic? Just boot up more VMs with the same image, and increase the size of your k8s pods. Want to handle multiple more complexity, split your job into micro services, and give responsibility of different namespaces to different teams. If you're using a late 2000s, early 2010s model, you would just use VM, and boot them when necessary. You have a VM with HAProxy with dynamic backends. Want to run more services? Just add more to the base image and route between them with a nginx local to your base VM image. Want to handle more traffic, just spin up more VMs with the base image. Want to handle more complexity? Use multiple types of VMs with different base images, depending on the service, and give the responsability to each team for configuring the base image. I can go on and on for packages and/or configuration-management tools. But I think you get the idea.

xorcist 4y ago

Alma produces a rather snazzy script to migrate from CentOS which works really well. I'm told it also works with Rocky, but I haven't tried it.

bclemens 4y ago
Rocky Linux has migration scripts available at https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky https://github.com/rocky-linux/rocky-tools/tree/main/migrate... An easy shortcut though, in case you have to hand type it out somewhere, is https://rockylinux.org/migrate2rocky.sh https://rockylinux.org/migrate2rocky.sh
- jonathanspw 4y ago
  Pretty sure he was talking about ELevate to go from EL7 to Alma 8.

fancyfredbot 4y ago

What happened to scientific Linux? It's a bit odd that it seems to be being discontinued rather than rebased on Alma? Do they no longer believe scientific Linux adds value?

brnt 4y ago
It had been discontinued a while ago, in favour of a flavour of CentOS (at the time CentOS 7 was released, instead of Scientific Linux 7).

jl6 4y ago

I’m out of the loop on RHEL forks. Can someone explain why it is important to be binary compatible with RHEL? Is there a lot of software whose binaries will only run on RHEL? Or is this more a case of wanting something free that can still make use of RHEL-specific skills and knowledge?

sllabres 4y ago
As soon as you use something like SAP or Oracle the number of offically _supported_ distributions quickly decreases. As example SAP HANA: RHEL and SLES or for example Oracle Database: RHEL, OEL (based on RHEL AFAIK) and again SLES. So it is not only a question if it would theoretically/technically fail on another platform.
- jl6 4y ago
  I don’t imagine SAP certification would extend to a community-supported fork of RHEL even if it was a 1:1 clone.
  - Moose_the_Goose 4y ago
    Many businesses will use the Forks for their dev environment.
    - carlwgeorge 4y ago
      Which is silly because Red Hat will literally give you free RHEL for non-prod if you pay for RHEL in prod. https://developers.redhat.com/articles/2022/05/10/access-rhel-developer-teams-subscription https://developers.redhat.com/articles/2022/05/10/access-rhe...
Symmetry 4y ago
When I was working for the US government we had to obey rules against shipping "freeware," which RHEL wasn't because we payed for it. But it was cheaper and easier to do most of our development on CentOS systems and because the software was the same we weren't worried about compatibility issues.

hsbauauvhabzb 4y ago

I would prefer to use RedHat based distros but their lack of official non free repos annoy me - I don’t want to add a community repo, I want packages maintained by trusted core engineers. Sure the community repos probably have that, but I last I checked I couldn’t find any assurance on the security of the community repos. SBOMs don’t mean shit if you’re installing stuff from random no-assurance locations.

ilyt 4y ago
Then you want Debian. We've had FAR less problems with Debians than anything from Red Hat stable
- MisterTea 4y ago
  I've used a bunch of Linux distros over the years and always find myself installing Debian when I just want a desktop.
  - hsbauauvhabzb 4y ago
    I’m currently on Ubuntu and have considered Debian. I like the concept of containerisation (snap, flatpack, etc) but the overly pushy snap integration has broken my workflows in multiple situations without much redemption so maybe I will give it a try.
    - throw_a_grenade 4y ago
      If you want "just desktop" like grandparent, you can try https://linuxmint.com/ https://linuxmint.com/, it's downstream from Ubuntu LTS, but with bad stuff (snap) removed: https://linuxmint-user-guide.readthedocs.io/en/latest/snap.html https://linuxmint-user-guide.readthedocs.io/en/latest/snap.h...
      - hsbauauvhabzb 4y ago
        Yeah I’m all for those distros, but as I work in security industry i kind of feel like a ‘nobody ever got fired for buying an ibm approach’ is good. If I’m hacked it’s not a good look at all, if im hacked and use an esoteric distro like gentoo, it would certainly look much, much worse. My key pain point is trust within repos, Ubuntu audit their repos as best they can (sast/sanity check) so at least there’s some security there. Im otherwise very supportive of mint/arch/gentoo and similar systems.

jacooper 4y ago

And after all of that, dnf auto update still doesn't support auto restarts for kernel updates, amazing. Also no in-place upgrades since for ever.

vbezhenar 4y ago
I'm afraid to even update ordinary packages on a running system. People who want to update kernel on the fly are just reckless!
- jacooper 4y ago
  Since everything is already running as a container, it really doesn't matter
  - vbezhenar 4y ago
    Containers are ordinary processes, just with some namespacing for separation. And not everything is running in a container.

warinukraine 4y ago

Not a linux expert. Skimming wikipedia it seems the main selling point of both Rocky and Alma is that they're binary-compatible with red hat, right? Could some experts clarify: if you want to be binary-compatible with red hat, why not just use red had?

zamalek 4y ago
RHEL costs money.
- warinukraine 4y ago
  Yikes. Why do people use it exactly? (I use debian, that's all I know)
  - Moose_the_Goose 4y ago
    RHEL is used by companies that need the support Red Hat provides. RHEL and it's derivatives have less packages in the repo due to Red Hat willing to support every package they ship.

grysgt 4y ago

jeffbee 4y ago

I bet these guys have a lot of unusual legacy requirements, like wanting Motif and remote X11/Xdm to keep working, which takes Ubuntu out of the competition.

egberts1 4y ago

Absolutely love the integrated signing of containers (Sigstore technology). If more distros support this, then our life would be far less stressful. https://www.sigstore.dev/ https://www.sigstore.dev/