11 ms·

>> Imagine if you were running a business, and deployed CentOS 8 based on the 10 year lifespan promise. You're totally screwed now, and Red Hat knows it. The

>> Imagine if you were running a business, and deployed CentOS 8 based on the 10 year lifespan promise. You're totally screwed now, and Red Hat knows it. The hypothetical you posed is the actual situation, I am now learning, I have apparently forced on my team. We've ramped up labor 3x revenue preparing product launch in 90 - 180 days. We created an image containing centos 8 , Java , postgres and tomcat a year ago and that what is deployed to beta clients and what we've been testing. What's ironic is that I sort of went out on a limb with my team by forcing us to go with Linux over Windows and the way I allayed concerns was to ask them to just "wait and see" in hopes that the performance differential would make it a moot point. edit: after a little thought it seems that moving to RHEL might cost us the least amount of money and downtime. Still sucks and not what we need to be working on right now.

vbezhenar 6y ago
Consider downgrading to CentOS 7. It should work for 4 more years.
- p49k 6y ago
  Running modern versions of software on CentOS 7 can be a giant pain due to the old version of glibc that’s baked in.
  - vbezhenar 6y ago
    Java and Postgres will work.
    - nix23 6y ago
      I go and drink some Corretto :)
  - AshamedCaptain 6y ago
    I was supporting CentOS 6 until about a couple weeks ago.
    - yjftsjthsd-h 6y ago
      Then surely you understand what a pain it is.
  - chaz6 6y ago
    An easy way around this problem is to run the software in a docker container with a newer glibc.
    - dijit 6y ago
      Docker "works" but has huge performance costs (disk IO) on CentOS7.
- soraminazuki 6y ago
  Can we take RedHat's word for it, though? CentOS 8 was supposed to be supported until 2029 before this bombshell announcement. https://web.archive.org/web/20201101131417/https://wiki.centos.org/About/Product https://web.archive.org/web/20201101131417/https://wiki.cent...
  - rbowen 6y ago
    The dates on that page were always dependent on Red Hat. In retrospect, obviously we should have made that clearer, and we will endeavor to do so in the future.
wronglebowski 6y ago
Out of curiosity why did you decide to deploy your application as a VM image? Is there a reason you didn't go with a Helm chart or other container native deployment?
- chefkoch 6y ago
  >a Helm chart As a sysadmin i have never heard of this and es tun 800 VMs.
  - hxtk 6y ago
    It's a bundle of jinja2 templates for YAML files defining Kubernetes API resources. There's an accompanying application, Helm, which applies values to the templates and handles the interaction with the Kubernetes API.
    - the_duke 6y ago
      Helm uses Golangs built-in template language which is definitely not jinja2 compatible.
      - joana035 6y ago
        Basically the same thing: templating.
- chippiewill 6y ago
  You're asking why someone who uses CentOS hasn't gone with Helm charts? That's like polar opposites of stability and bleeding edge.
  - wronglebowski 6y ago
    I was curious what drove the decision to deploy that way. I was under the impression most new applications being developed today would choose a more modern deployment method. There’s a lot to maintain in a VM image like that, containers just seem easier to me. Helm chart or otherwise
fweespeech 6y ago
> edit: after a little thought it seems that moving to RHEL might cost us the least amount of money and downtime. Still sucks and not what we need to be working on right now. And that is exactly what IBM is counting on. Vendor lock in. We are switching to Debian-based distros because, frankly, we don't trust IBM not to knife us in the back even on RHEL for more money. Of course we have the advantage of being able to take the time to convert.
- bonzini 6y ago
  This is not about IBM. It may not be a pleasant change for everyone, but it's a change that has strictly technical motives. CentOS was acqui-hired because Red Hat's upstream for layered products (at the time mostly RDO/OpenStack and oVirt/RHEV) could not use Fedora because it was too far from RHEL a year of two after RHEL was released, could not use RHEL because upstream contributors would have to pay, and could not use CentOS because its releases had too large delays. The solution was to make CentOS releases happen timely by paying people to make them. These days a RHEL downstream is not enough for the layered products. Some of them require the kind of bleeding edge feature that is backported every six months to the RHEL kernel, and corresponding userspace changes (BPF, virtualization, etc.) and cannot afford waiting for the CentOS release because development must be done in parallel with RHEL. So the solution was to move CentOS from happening after RHEL to before RHEL which is what CentOS Stream is. I can confidently say that the reasons are technical because other CentOS downstream have the same needs (e.g. Facebook's) and they also want to send patches to CentOS for bugfixes or features themselves, instead of waiting for Red Hat to find out about that bug, or decide they need the same feature. Plus there's no reason for rebuilds to disappear. The SRPMs will still be released by Red Hat.
  - kbenson 6y ago
    That in no way explains why they don't continue to have both. They indeed will have both for another year. There was no requirement the Stream product even use the CentOS name. CentOS was a community project whose leadership and control was taken over (acqui-hired as you say) by Red Hat and then it's core use case for the majority of people actually using it was discontinued. That is a statement of facts that happened as I understand them, not some spin on my part. If Red Hat had not stepped in, perhaps some of CentOS problems (trouble getting releases out on time) would have been worse, or perhaps some other companies would have stepped in. We don't know, but we do know that CentOS has not been changed to be something different than it was before. It used to be a free re-spin of RHEL. Going forward it's something entirely different. Red Hat always had the option to stop funding/providing resources to CentOS and name their new thing something else, but they didn't, and now they've effectively co-opted CentOS to be something different than it was originally intended to be.
deleted 6y ago
[deleted]
norswap 6y ago
I'm wondering, what ties you so strongly to a single OS? Nay, to a single Linux distribution? I have a hard time imagining, and the only scenarios that come to mind are those where things have gone awfully haywire. I'm sure I'm missing something. Enlighten me?
- nine_k 6y ago
  An existing support contract? If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong.
  - webmaven 6y ago
    > If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong. There is a difference between asking "how do I ensure there is one throat to choke when things go wrong?", and "how do I minimize the potential for things to go wrong?" RHEL is a decent solution if you are trying to answer the former question (or both), but many budget conscious orgs focused on the latter and chose CentOS. Red Hat has now pulled the rug out from under them by trimming eight years off the EOL previously committed to with little notice. EOL in this case means "no security updates" so even if your org was prepared, technically, to deal with a zero-day for example by rolling out an update in a timely manner without relying on paying a vendor for hand-holding, that option has now been eliminated. Essentially, you now only get the stability and problem-minimization if you also pay the vendor for support. Otherwise you're stuck with a (relatively) unstable rolling release that will keep your internal teams very busy with a constant stream of minor issues, or potentially trying to roll-your-own updates or backports after the EOL for anything serious. It's difficult to see this as anything other than a naked, money-grabbing betrayal of users.
- soraminazuki 6y ago
  Software doesn’t run on its own. They rely on bunch of other software that is typically provided by distributions. So unless there’s some magic piece of technology that can take CentOS RPMs and make it work flawlessly on any Linux distro, the entire software industry is suddenly going to have to spend significant amounts of time on repackaging work.
deleted 6y ago
[deleted]
kseifried 6y ago
What I don't get is this: You build software, you ship it on an OS so it works, cool, this makes sense (I assume you need hardware and VM support, not just VM). Why would you accept additional risk on the OS if you can easily reduce the risk, and ultimate cost, by going with an OS that has vendor support written into the actual contract? RHEL is 11-13 years total, Windows is... I'm guessing my grandkids will be using some form of Windows 10. CentOS is, and always was a community "best effort", with some serious delays occasionally (not often, but it happened). A RHEL server license starts at $349, I have to assume that's at least an order of magnitude (or two or three) less than the cost of your software based on the technologies involved (sounds enterprise-solutiony). In other words a rounding error overall.
- deleted 6y ago
  [deleted]
- hjb100 6y ago
  As a sysadmin in academia, this is not so straightforward. Since number of servers/VMs are in ballpark of over one hundred, RedHat license costs will be over 30,000$/year. This is not insignificant amount of money and not easy to get the money suddenly.
  - bierjunge 6y ago
    $30k is a significant amount, especially in academic environment, I don't deny it. But it's not much for > 100 servers. In my previous job, I had licenses which were about 15k€ for one server (thanks Oracle...) and the Windows Server licenses were also pretty expensive. $300-500 per server/year is not much more than a Photoshop license for one designer. And if we talk big business (1000's of servers and even more), you will get very special pricing anyway.
    - russfink 6y ago
      Not just price, but ability to function without constant access to the satellite portal. It's darn near impossible to upgrade a RHEL box from a work at home VPN environment.
deleted 6y ago
[deleted]
montjoy 6y ago
I’d recommend using Oracle Linux 8 if you still need 1 to 1 compatibility. At my company I’m recommending Amazon Linux 2.
- montjoy 6y ago
  Not sure why I got downvoted. I’m trying to offer the poster a viable alternative to CentOS 8 with a minimum of changes needed. I get that a lot of people don’t like Oracle (and for good reasons) but really they are the best alternative (and free) EL8 disto now.

vbezhenar 6y ago

Consider downgrading to CentOS 7. It should work for 4 more years.

p49k 6y ago
Running modern versions of software on CentOS 7 can be a giant pain due to the old version of glibc that’s baked in.
- vbezhenar 6y ago
  Java and Postgres will work.
  - nix23 6y ago
    I go and drink some Corretto :)
- AshamedCaptain 6y ago
  I was supporting CentOS 6 until about a couple weeks ago.
  - yjftsjthsd-h 6y ago
    Then surely you understand what a pain it is.
- chaz6 6y ago
  An easy way around this problem is to run the software in a docker container with a newer glibc.
  - dijit 6y ago
    Docker "works" but has huge performance costs (disk IO) on CentOS7.
soraminazuki 6y ago
Can we take RedHat's word for it, though? CentOS 8 was supposed to be supported until 2029 before this bombshell announcement. https://web.archive.org/web/20201101131417/https://wiki.centos.org/About/Product https://web.archive.org/web/20201101131417/https://wiki.cent...
- rbowen 6y ago
  The dates on that page were always dependent on Red Hat. In retrospect, obviously we should have made that clearer, and we will endeavor to do so in the future.

wronglebowski 6y ago

Out of curiosity why did you decide to deploy your application as a VM image? Is there a reason you didn't go with a Helm chart or other container native deployment?

chefkoch 6y ago
>a Helm chart As a sysadmin i have never heard of this and es tun 800 VMs.
- hxtk 6y ago
  It's a bundle of jinja2 templates for YAML files defining Kubernetes API resources. There's an accompanying application, Helm, which applies values to the templates and handles the interaction with the Kubernetes API.
  - the_duke 6y ago
    Helm uses Golangs built-in template language which is definitely not jinja2 compatible.
    - joana035 6y ago
      Basically the same thing: templating.
chippiewill 6y ago
You're asking why someone who uses CentOS hasn't gone with Helm charts? That's like polar opposites of stability and bleeding edge.
- wronglebowski 6y ago
  I was curious what drove the decision to deploy that way. I was under the impression most new applications being developed today would choose a more modern deployment method. There’s a lot to maintain in a VM image like that, containers just seem easier to me. Helm chart or otherwise

fweespeech 6y ago

> edit: after a little thought it seems that moving to RHEL might cost us the least amount of money and downtime. Still sucks and not what we need to be working on right now. And that is exactly what IBM is counting on. Vendor lock in. We are switching to Debian-based distros because, frankly, we don't trust IBM not to knife us in the back even on RHEL for more money. Of course we have the advantage of being able to take the time to convert.

bonzini 6y ago
This is not about IBM. It may not be a pleasant change for everyone, but it's a change that has strictly technical motives. CentOS was acqui-hired because Red Hat's upstream for layered products (at the time mostly RDO/OpenStack and oVirt/RHEV) could not use Fedora because it was too far from RHEL a year of two after RHEL was released, could not use RHEL because upstream contributors would have to pay, and could not use CentOS because its releases had too large delays. The solution was to make CentOS releases happen timely by paying people to make them. These days a RHEL downstream is not enough for the layered products. Some of them require the kind of bleeding edge feature that is backported every six months to the RHEL kernel, and corresponding userspace changes (BPF, virtualization, etc.) and cannot afford waiting for the CentOS release because development must be done in parallel with RHEL. So the solution was to move CentOS from happening after RHEL to before RHEL which is what CentOS Stream is. I can confidently say that the reasons are technical because other CentOS downstream have the same needs (e.g. Facebook's) and they also want to send patches to CentOS for bugfixes or features themselves, instead of waiting for Red Hat to find out about that bug, or decide they need the same feature. Plus there's no reason for rebuilds to disappear. The SRPMs will still be released by Red Hat.
- kbenson 6y ago
  That in no way explains why they don't continue to have both. They indeed will have both for another year. There was no requirement the Stream product even use the CentOS name. CentOS was a community project whose leadership and control was taken over (acqui-hired as you say) by Red Hat and then it's core use case for the majority of people actually using it was discontinued. That is a statement of facts that happened as I understand them, not some spin on my part. If Red Hat had not stepped in, perhaps some of CentOS problems (trouble getting releases out on time) would have been worse, or perhaps some other companies would have stepped in. We don't know, but we do know that CentOS has not been changed to be something different than it was before. It used to be a free re-spin of RHEL. Going forward it's something entirely different. Red Hat always had the option to stop funding/providing resources to CentOS and name their new thing something else, but they didn't, and now they've effectively co-opted CentOS to be something different than it was originally intended to be.

deleted 6y ago

[deleted]

norswap 6y ago

I'm wondering, what ties you so strongly to a single OS? Nay, to a single Linux distribution? I have a hard time imagining, and the only scenarios that come to mind are those where things have gone awfully haywire. I'm sure I'm missing something. Enlighten me?

nine_k 6y ago
An existing support contract? If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong.
- webmaven 6y ago
  > If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong. There is a difference between asking "how do I ensure there is one throat to choke when things go wrong?", and "how do I minimize the potential for things to go wrong?" RHEL is a decent solution if you are trying to answer the former question (or both), but many budget conscious orgs focused on the latter and chose CentOS. Red Hat has now pulled the rug out from under them by trimming eight years off the EOL previously committed to with little notice. EOL in this case means "no security updates" so even if your org was prepared, technically, to deal with a zero-day for example by rolling out an update in a timely manner without relying on paying a vendor for hand-holding, that option has now been eliminated. Essentially, you now only get the stability and problem-minimization if you also pay the vendor for support. Otherwise you're stuck with a (relatively) unstable rolling release that will keep your internal teams very busy with a constant stream of minor issues, or potentially trying to roll-your-own updates or backports after the EOL for anything serious. It's difficult to see this as anything other than a naked, money-grabbing betrayal of users.
soraminazuki 6y ago
Software doesn’t run on its own. They rely on bunch of other software that is typically provided by distributions. So unless there’s some magic piece of technology that can take CentOS RPMs and make it work flawlessly on any Linux distro, the entire software industry is suddenly going to have to spend significant amounts of time on repackaging work.

deleted 6y ago

[deleted]

kseifried 6y ago

What I don't get is this: You build software, you ship it on an OS so it works, cool, this makes sense (I assume you need hardware and VM support, not just VM). Why would you accept additional risk on the OS if you can easily reduce the risk, and ultimate cost, by going with an OS that has vendor support written into the actual contract? RHEL is 11-13 years total, Windows is... I'm guessing my grandkids will be using some form of Windows 10. CentOS is, and always was a community "best effort", with some serious delays occasionally (not often, but it happened). A RHEL server license starts at $349, I have to assume that's at least an order of magnitude (or two or three) less than the cost of your software based on the technologies involved (sounds enterprise-solutiony). In other words a rounding error overall.

deleted 6y ago
[deleted]
hjb100 6y ago
As a sysadmin in academia, this is not so straightforward. Since number of servers/VMs are in ballpark of over one hundred, RedHat license costs will be over 30,000$/year. This is not insignificant amount of money and not easy to get the money suddenly.
- bierjunge 6y ago
  $30k is a significant amount, especially in academic environment, I don't deny it. But it's not much for > 100 servers. In my previous job, I had licenses which were about 15k€ for one server (thanks Oracle...) and the Windows Server licenses were also pretty expensive. $300-500 per server/year is not much more than a Photoshop license for one designer. And if we talk big business (1000's of servers and even more), you will get very special pricing anyway.
  - russfink 6y ago
    Not just price, but ability to function without constant access to the satellite portal. It's darn near impossible to upgrade a RHEL box from a work at home VPN environment.

deleted 6y ago

[deleted]

montjoy 6y ago

I’d recommend using Oracle Linux 8 if you still need 1 to 1 compatibility. At my company I’m recommending Amazon Linux 2.

montjoy 6y ago
Not sure why I got downvoted. I’m trying to offer the poster a viable alternative to CentOS 8 with a minimum of changes needed. I get that a lot of people don’t like Oracle (and for good reasons) but really they are the best alternative (and free) EL8 disto now.