9 ms·

Whats missing is an analysis of why CentOS failed. I think Rocky Linux needs to put out a plan how they will make themselves financially viable as we've had 3

Whats missing is an analysis of why CentOS failed. I think Rocky Linux needs to put out a plan how they will make themselves financially viable as we've had 3 high profile RHEL respins go down in the last 10 years. CentOS failed twice, it ran out of money in 2014 and was rescued back then by Redhat sponsership. Again in 2020. Another widely used RHEL respin was Scientific Linux which mothballed when RHEL 7 was released. There seems to be lots of potential users but not lots of potential money for a RHEL respin.

fierarul 6y ago
It's not really surprising that a project that companies use since they don't want to pay RedHat licenses will not get any money at all...
pinewurst 6y ago
It’s hard to continually fund any project defined by its users’ abhorrence of compensation.
- cpncrunch 6y ago
  I think Ubuntu has a decent business plan, charging people $225/yr to extend support from 5 to 10 years. I would happily pay that in order to avoid having to migrate as often. CentOS just gave everything away for free and then is wondering why they're not making any money.
  - loeg 6y ago
    RHEL has a business model too.
    - cpncrunch 6y ago
      Yes, indeed, but I was referring to Centos. There isn't enough of a benefit for most of us to upgrade if we get security updates free for 10 years and don't need support.
      - johannes1234321 6y ago
        And that shows how pointless the CentOS kill is. Those users won't go to RHEL but Debian/Ubuntu or if it has to be RHEL-compatible Oracle Linux and then lobby their other vendors to support Debian/Ubuntu or some other distribution as well.
  - someperson 6y ago
    CentOS's 10 year support window was unreasonably long given that there's no benefit to upgrading to RHEL (and it's paid employees at RedHat during the security backporting work). The free RedHat tier should have a shorter support window that's still long enough to attract users who want a stable platform to build long-lived appliances.
  - reader_mode 6y ago
    Paid CentOS support is called RHEL - the whole point of CentOS is repackage GPL RHEL stuff without the licensing fee. The only way I see this being viable is them getting bankrolled by a huge cloud provider - but cloud providers already decided that having their own distros was a better option.
    - cpncrunch 6y ago
      Indeed, but most of us don't need support. We just need security updates, ideally for 10 years.
      - reader_mode 6y ago
        AFAIK there's a licensing tier for RHEL that gets you that - don't know how it compares to Ubuntu.
        mroche 6y ago
        For RHEL Server (MSRP) Self-Support: $350/yr Standard: $800/yr Premium: $1300/yr Those are all starting points without including potential addons.
        Nullabillity 6y ago
        Apparently the self-support edition isn't allowed to be ran on VMs. WTF?
    - toyg 6y ago
      > GPL RHEL stuff without the licensing fee Without the absurdly high licensing fee. A more reasonable amount (be it a saas-like low monthly charge or one-off 3-digit fee) would probably go down fine with enough institutional users to generate a decent amount of money.
      - corty 6y ago
        Well, absurdely high isn't really true, it is quite similar to e.g. https://www.microsoft.com/en-us/windows-server/pricing https://www.microsoft.com/en-us/windows-server/pricing There is a some up/down depending on various rebates, volume licenses, support included/excluded, etc ("nobody pays sticker price"). But in general, RedHat is in the same order of magnitude as Windows but a little cheaper due to no per-core-pricing, no necessary CAL shenannigans or weird limitations on number of users/size of company/VM/PM and stuff. But of course it's true that in domains like HPC or cloud computing, the huge number of licenses and machines involved make a few hundred bucks per year just too expensive in sum.
  - ilikeerp 6y ago
    I charge more per hour for my consulting, I had a bottle of wine that cost about that last week for dinner. What revenue do they actually get out of that model?
    - dralley 6y ago
      They don't, Canonical's finances are terrible. https://news.ycombinator.com/item?id=25450835 https://news.ycombinator.com/item?id=25450835
      - toyg 6y ago
        To be fair, they do much more work than CentOS ever did (or Rocky would have to do). In fact, Ubuntu does too much development for its own good. People jumped on it because it was “a usable Debian updated more often”, and they got all sorts of crazy UX experiments.
  - dralley 6y ago
    Every year between 2009 and 2017, Canonical lost money. In 2018 after several rounds of layoffs they finally made a profit of ~10 million, but their revenue actually dropped from the year before. Whether or not it is a good plan in theory, it clearly doesn't seem to be working that well for them in practice.
  - iso947 6y ago
    Do AWS ubuntu images come with a software charge of say 2 or 3 cents an hour?
konjin 6y ago
It failed because it let redhat people in. I know mentioning incentives and that people lie is not popular here but when the same thing happens to two different projects you have to be willfully blind to not see a pattern. As for making it viable long term: a clear goal, community norms and quick removal of anyone not toeing the line.
qz2 6y ago
Why it failed is because they sold out. I really think everyone should just say “fuck this”, abandon it and throw the time at LTS Debian. Red hat control way too much of the ecosystem at the same time so there needs to be a strong, stable, non-commercial alternative.
dotancohen 6y ago
> Whats missing is an analysis of why CentOS failed. CentOS did not fail. CentOS was a wild success. Like the vast majority of tech success storeis, CentOS was picked up by a major player and turned against its roots to protect the incumbent moneymaker. But the terrific thing about the GPL, for all its flaws and despite its crazy founder, is that it ensures the health and longevity of a project despite well-funded attacks against it.
- someperson 6y ago
  I don't think anybody is against CentOS being an onboarding ramp to paid RHEL deployments. CentOS is actually Red Hat's greatest advertisement If IBM Red Hat wanted to push for RHEL upgrades, they should have changed the CentOS support window from 10 years to 3-5 years. If they had to wind back the EOL for CentOS date from 2029, they should have at least move it back to eg, 2024 not 2021.
- mbreese 6y ago
  It failed... it is not able to sustain itself. Whatever replaces it either needs a better business model (to pay for maintenance, RHEL, Ubuntu) or more community involvement (work for free, Debian). But when you’re effectively repackaging another distro, it’s probably hard to get other people excited enough to help.
em500 6y ago
Maybe it's a bit too obvious to spell it out, but 1. Red Hat Inc. does not want people to build and/or distribute gratis RHEL8 or clones. It would be trivial to just put the actual RHEL8 iso as an unsupported download on their ftp/www server and sell the support separately, like Oracle or Canonical do. Instead, they kept this ridiculous make work project called CentOS around, that involved non-trivial manual labor meticulously rebranding RHEL into RedHat owned CentOS-the-laggard-RHEL-clone, whose users apparently mainly value it for being as close to RHEL as possible without paying. To a distant observer, the whole setup looks quite absurd. I think it's actually good that they finally put an end to it. But they should have either not started CentOS 8 at all or rode it out till the end. Pulling the plug at 20% of it's lifetime is plainly a shitty move. 2. It appears that building a legal RHEL8+ clone is quite a task, and I'm guessing the amount of work involved is largely controlled by Red Hat Inc. I.e., they can make running/maintaining a project like Rocky or others more and more expensive if they choose to. I believe they going to test the limits of how difficult they can make building from their sources within the boundaries of the GPL. If you think I'm wrong, just put up a mirror of the RHEL8 (not CentOS8) SRPMs and see how long it stays up. Clearly they're not acting in the spirit of the GPL, even if they are in the letter. 3. Given the previous points, I believe any project like Rocky is a losing proposition. If the "community" really values enterprise stability so much, better put the effort into an extra-stable-extra-LTS fork of Debian or even Ubuntu, and preparing for transition away from RHEL. Or just pay up, if you really believes the RHEL stability is so valuable. Clone projects are by necessity extremely dependent on actions of Red Hat Inc which has largely opposite interests. I don't know why people would volunteer for that.
- evol262 6y ago
  > 1. Red Hat Inc. does not want people to build and/or distribute gratis RHEL8 or clones. It would be trivial to just put the actual RHEL8 iso as an unsupported download on their ftp/www server and sell the support separately, like Oracle or Canonical do. Instead, they kept this ridiculous make work project called CentOS around, that involved non-trivial manual labor meticulously rebranding RHEL into RedHat owned CentOS, whose users apparently mainly value it for being as close to RHEL as possible without paying. To a distant observer, the whole setup looks quite absurd. I think it's actually good that they finally put an end to it. But they should have either not started CentOS 8 at all or rode it out till the end. Pulling the plug at 20% of it's lifetime is plainly a shitty move. But [they do](https://developers.redhat.com/products/rhel/download https://developers.redhat.com/products/rhel/download). A subscription buys you updates and support. RHEL itself is free. In addition, considering that almost all of Red Hat's products are "upstream first", branding is generally a single additional RPM which changes some colors. It is _not_ hard to rebrand RHEL > 2. It appears that building a legal RHEL8+ clone is quite a task, and I'm guessing the amount of work involved is largely controlled by Red Hat Inc. I.e., they can make running/maintaining a project like Rocky or others more and more expensive if they choose to. I believe they going to test the limits of how difficult they can make building from their sources within the boundaries of the GPL. If you think I'm wrong, just put up a mirror of the RHEL8 (not CentOS8) SRPMs and see how long it stays up. Clearly they're not acting in the spirit of the GPL, if they are in the letter. The only case in which this has happened is changing the packaging of the kernel-sources SRPM so Oracle could not pick and choose particular patches, and instead had to deal with a single massive diff to make stuff like kpatch work. The "expensive" part of building an EL8 clone is standing up a bunch of Koji builders. That isn't necessary either, strictly. It just makes "turn this SRPM into an RPM and combine it with a bunch of others into a temporary repo we can dogfood/release" easier. As always, it comes down to cost. > 3. Given the previous point, I believe a project like Rocky is a losing proposition. If the "community" really values enterprise stability so much, better put the effort into an extra-LTS fork of Debian or even Ubuntu, and preparing for transition away from RHEL. Or just pay up, if you really believes the RHEL stability is so valuable. Clone projects are by necessity extremely dependent on actions of Red Hat Inc which has largely opposite interests. I don't know why people would volunteer for that. The problem with this, broadly, is that the "community" is comprised of a lot of Red Hat-employed engineers. This idea that Linux is a bunch of "community" people that RH/IBM/whomever siphon off is completely fallacious. Sure, they exist, but the vast majority of Linux development is commercial. Some bug is reported in a downstream product that a customer pays for, or a feature is requested by a major customer (or the technical debt involved in maintaining something gets too high, or whatever), and professional engineers who are being paid to work on it write the patches upstream. Once they're merged, they get picked downstream into some kind of productized version. Given that most of that happens in Fedora (or OKD, or oVirt, or whatever upstream is for a given product), CentOS being run by RH was pretty much the same embrace+extend+extinguish philosophy. The vast majority of RH engineers run Fedora, or an EL clone, and that isn't gonna change. What's different is that RHEL is no longer a growing revenue stream, and CentOS users didn't convert to RHEL at a large rate (or they reported CentOS bugs against RHEL, which is strictly against Red Hat's support policy). What customers values is indemnity and the ability to point their finger at someone external, plus normal stuff like a security response team and responsible+timely disclosure during major CVEs. What the community value(d|s) is the ability to run commercial software that the vendor supports on RHEL without actually _paying_ for RHEL. It's not that they value 'enterprise stability'. It's that they value being able to run SAS or whatever without hacking the hell out of the installer. That isn't offered by an extra-LTS fork of Debian or Ubuntu, and no amount of complaining on Hackernews is gonna change that.
chillfox 6y ago
Well yeah, The people who are willing to spend money are on RHEL instead of a clone.
znpy 6y ago
> Whats missing is an analysis of why CentOS failed. nah, it's not missing. Red Hat bought the project and more importantly the trademark, domains etc. And it was mostly fine. Then IBM bought Red Hat and things went south. CentOS did not fail, CentOS was killed. It's different.
- wfuser 6y ago
  > Then IBM bought Red Hat and things went south. I think IBM had nothing to do with this one. People think Red Hat won't do anything like this. However, there exists a part of Red Hat which is capable of doing this. That part of Red Hat usually stays behind the scenes and comes to the fore to announce that a decision has been made and the developers (hired within Red Hat as well as the general community) who are involved in the day to day running of the projects will have no say. Plus they will throw in some confusion (like the limited use license that is in the works but not yet ready for CentOS use) around the future of the project being killed just to let the community expect something good to come out of this exercise. This is not new. They did the exact same thing to the JBoss community application server[1]. [1] https://news.ycombinator.com/item?id=25358847 https://news.ycombinator.com/item?id=25358847
- playcache 6y ago
  IBM had nothing to do with it.
  - johannes1234321 6y ago
    While I think that the decision was done before IBM took over, that statement is too much imo. IBM could have stopped the plan, if they wanted to.
goatinaboat 6y ago
Whats missing is an analysis of why CentOS failed. It did not fail. It succeeded too well! It was impacting RH’s bottom line, so they killed it.

fierarul 6y ago

It's not really surprising that a project that companies use since they don't want to pay RedHat licenses will not get any money at all...

pinewurst 6y ago

It’s hard to continually fund any project defined by its users’ abhorrence of compensation.

cpncrunch 6y ago
I think Ubuntu has a decent business plan, charging people $225/yr to extend support from 5 to 10 years. I would happily pay that in order to avoid having to migrate as often. CentOS just gave everything away for free and then is wondering why they're not making any money.
- loeg 6y ago
  RHEL has a business model too.
  - cpncrunch 6y ago
    Yes, indeed, but I was referring to Centos. There isn't enough of a benefit for most of us to upgrade if we get security updates free for 10 years and don't need support.
    - johannes1234321 6y ago
      And that shows how pointless the CentOS kill is. Those users won't go to RHEL but Debian/Ubuntu or if it has to be RHEL-compatible Oracle Linux and then lobby their other vendors to support Debian/Ubuntu or some other distribution as well.
- someperson 6y ago
  CentOS's 10 year support window was unreasonably long given that there's no benefit to upgrading to RHEL (and it's paid employees at RedHat during the security backporting work). The free RedHat tier should have a shorter support window that's still long enough to attract users who want a stable platform to build long-lived appliances.
- reader_mode 6y ago
  Paid CentOS support is called RHEL - the whole point of CentOS is repackage GPL RHEL stuff without the licensing fee. The only way I see this being viable is them getting bankrolled by a huge cloud provider - but cloud providers already decided that having their own distros was a better option.
  - cpncrunch 6y ago
    Indeed, but most of us don't need support. We just need security updates, ideally for 10 years.
    - reader_mode 6y ago
      AFAIK there's a licensing tier for RHEL that gets you that - don't know how it compares to Ubuntu.
      - mroche 6y ago
        For RHEL Server (MSRP) Self-Support: $350/yr Standard: $800/yr Premium: $1300/yr Those are all starting points without including potential addons.
        Nullabillity 6y ago
        Apparently the self-support edition isn't allowed to be ran on VMs. WTF?
  - toyg 6y ago
    > GPL RHEL stuff without the licensing fee Without the absurdly high licensing fee. A more reasonable amount (be it a saas-like low monthly charge or one-off 3-digit fee) would probably go down fine with enough institutional users to generate a decent amount of money.
    - corty 6y ago
      Well, absurdely high isn't really true, it is quite similar to e.g. https://www.microsoft.com/en-us/windows-server/pricing https://www.microsoft.com/en-us/windows-server/pricing There is a some up/down depending on various rebates, volume licenses, support included/excluded, etc ("nobody pays sticker price"). But in general, RedHat is in the same order of magnitude as Windows but a little cheaper due to no per-core-pricing, no necessary CAL shenannigans or weird limitations on number of users/size of company/VM/PM and stuff. But of course it's true that in domains like HPC or cloud computing, the huge number of licenses and machines involved make a few hundred bucks per year just too expensive in sum.
- ilikeerp 6y ago
  I charge more per hour for my consulting, I had a bottle of wine that cost about that last week for dinner. What revenue do they actually get out of that model?
  - dralley 6y ago
    They don't, Canonical's finances are terrible. https://news.ycombinator.com/item?id=25450835 https://news.ycombinator.com/item?id=25450835
    - toyg 6y ago
      To be fair, they do much more work than CentOS ever did (or Rocky would have to do). In fact, Ubuntu does too much development for its own good. People jumped on it because it was “a usable Debian updated more often”, and they got all sorts of crazy UX experiments.
- dralley 6y ago
  Every year between 2009 and 2017, Canonical lost money. In 2018 after several rounds of layoffs they finally made a profit of ~10 million, but their revenue actually dropped from the year before. Whether or not it is a good plan in theory, it clearly doesn't seem to be working that well for them in practice.
- iso947 6y ago
  Do AWS ubuntu images come with a software charge of say 2 or 3 cents an hour?

konjin 6y ago

It failed because it let redhat people in. I know mentioning incentives and that people lie is not popular here but when the same thing happens to two different projects you have to be willfully blind to not see a pattern. As for making it viable long term: a clear goal, community norms and quick removal of anyone not toeing the line.

qz2 6y ago

Why it failed is because they sold out. I really think everyone should just say “fuck this”, abandon it and throw the time at LTS Debian. Red hat control way too much of the ecosystem at the same time so there needs to be a strong, stable, non-commercial alternative.

dotancohen 6y ago

> Whats missing is an analysis of why CentOS failed. CentOS did not fail. CentOS was a wild success. Like the vast majority of tech success storeis, CentOS was picked up by a major player and turned against its roots to protect the incumbent moneymaker. But the terrific thing about the GPL, for all its flaws and despite its crazy founder, is that it ensures the health and longevity of a project despite well-funded attacks against it.

someperson 6y ago
I don't think anybody is against CentOS being an onboarding ramp to paid RHEL deployments. CentOS is actually Red Hat's greatest advertisement If IBM Red Hat wanted to push for RHEL upgrades, they should have changed the CentOS support window from 10 years to 3-5 years. If they had to wind back the EOL for CentOS date from 2029, they should have at least move it back to eg, 2024 not 2021.
mbreese 6y ago
It failed... it is not able to sustain itself. Whatever replaces it either needs a better business model (to pay for maintenance, RHEL, Ubuntu) or more community involvement (work for free, Debian). But when you’re effectively repackaging another distro, it’s probably hard to get other people excited enough to help.

em500 6y ago

Maybe it's a bit too obvious to spell it out, but 1. Red Hat Inc. does not want people to build and/or distribute gratis RHEL8 or clones. It would be trivial to just put the actual RHEL8 iso as an unsupported download on their ftp/www server and sell the support separately, like Oracle or Canonical do. Instead, they kept this ridiculous make work project called CentOS around, that involved non-trivial manual labor meticulously rebranding RHEL into RedHat owned CentOS-the-laggard-RHEL-clone, whose users apparently mainly value it for being as close to RHEL as possible without paying. To a distant observer, the whole setup looks quite absurd. I think it's actually good that they finally put an end to it. But they should have either not started CentOS 8 at all or rode it out till the end. Pulling the plug at 20% of it's lifetime is plainly a shitty move. 2. It appears that building a legal RHEL8+ clone is quite a task, and I'm guessing the amount of work involved is largely controlled by Red Hat Inc. I.e., they can make running/maintaining a project like Rocky or others more and more expensive if they choose to. I believe they going to test the limits of how difficult they can make building from their sources within the boundaries of the GPL. If you think I'm wrong, just put up a mirror of the RHEL8 (not CentOS8) SRPMs and see how long it stays up. Clearly they're not acting in the spirit of the GPL, even if they are in the letter. 3. Given the previous points, I believe any project like Rocky is a losing proposition. If the "community" really values enterprise stability so much, better put the effort into an extra-stable-extra-LTS fork of Debian or even Ubuntu, and preparing for transition away from RHEL. Or just pay up, if you really believes the RHEL stability is so valuable. Clone projects are by necessity extremely dependent on actions of Red Hat Inc which has largely opposite interests. I don't know why people would volunteer for that.

evol262 6y ago
> 1. Red Hat Inc. does not want people to build and/or distribute gratis RHEL8 or clones. It would be trivial to just put the actual RHEL8 iso as an unsupported download on their ftp/www server and sell the support separately, like Oracle or Canonical do. Instead, they kept this ridiculous make work project called CentOS around, that involved non-trivial manual labor meticulously rebranding RHEL into RedHat owned CentOS, whose users apparently mainly value it for being as close to RHEL as possible without paying. To a distant observer, the whole setup looks quite absurd. I think it's actually good that they finally put an end to it. But they should have either not started CentOS 8 at all or rode it out till the end. Pulling the plug at 20% of it's lifetime is plainly a shitty move. But [they do](https://developers.redhat.com/products/rhel/download https://developers.redhat.com/products/rhel/download). A subscription buys you updates and support. RHEL itself is free. In addition, considering that almost all of Red Hat's products are "upstream first", branding is generally a single additional RPM which changes some colors. It is _not_ hard to rebrand RHEL > 2. It appears that building a legal RHEL8+ clone is quite a task, and I'm guessing the amount of work involved is largely controlled by Red Hat Inc. I.e., they can make running/maintaining a project like Rocky or others more and more expensive if they choose to. I believe they going to test the limits of how difficult they can make building from their sources within the boundaries of the GPL. If you think I'm wrong, just put up a mirror of the RHEL8 (not CentOS8) SRPMs and see how long it stays up. Clearly they're not acting in the spirit of the GPL, if they are in the letter. The only case in which this has happened is changing the packaging of the kernel-sources SRPM so Oracle could not pick and choose particular patches, and instead had to deal with a single massive diff to make stuff like kpatch work. The "expensive" part of building an EL8 clone is standing up a bunch of Koji builders. That isn't necessary either, strictly. It just makes "turn this SRPM into an RPM and combine it with a bunch of others into a temporary repo we can dogfood/release" easier. As always, it comes down to cost. > 3. Given the previous point, I believe a project like Rocky is a losing proposition. If the "community" really values enterprise stability so much, better put the effort into an extra-LTS fork of Debian or even Ubuntu, and preparing for transition away from RHEL. Or just pay up, if you really believes the RHEL stability is so valuable. Clone projects are by necessity extremely dependent on actions of Red Hat Inc which has largely opposite interests. I don't know why people would volunteer for that. The problem with this, broadly, is that the "community" is comprised of a lot of Red Hat-employed engineers. This idea that Linux is a bunch of "community" people that RH/IBM/whomever siphon off is completely fallacious. Sure, they exist, but the vast majority of Linux development is commercial. Some bug is reported in a downstream product that a customer pays for, or a feature is requested by a major customer (or the technical debt involved in maintaining something gets too high, or whatever), and professional engineers who are being paid to work on it write the patches upstream. Once they're merged, they get picked downstream into some kind of productized version. Given that most of that happens in Fedora (or OKD, or oVirt, or whatever upstream is for a given product), CentOS being run by RH was pretty much the same embrace+extend+extinguish philosophy. The vast majority of RH engineers run Fedora, or an EL clone, and that isn't gonna change. What's different is that RHEL is no longer a growing revenue stream, and CentOS users didn't convert to RHEL at a large rate (or they reported CentOS bugs against RHEL, which is strictly against Red Hat's support policy). What customers values is indemnity and the ability to point their finger at someone external, plus normal stuff like a security response team and responsible+timely disclosure during major CVEs. What the community value(d|s) is the ability to run commercial software that the vendor supports on RHEL without actually _paying_ for RHEL. It's not that they value 'enterprise stability'. It's that they value being able to run SAS or whatever without hacking the hell out of the installer. That isn't offered by an extra-LTS fork of Debian or Ubuntu, and no amount of complaining on Hackernews is gonna change that.

chillfox 6y ago

Well yeah, The people who are willing to spend money are on RHEL instead of a clone.

znpy 6y ago

> Whats missing is an analysis of why CentOS failed. nah, it's not missing. Red Hat bought the project and more importantly the trademark, domains etc. And it was mostly fine. Then IBM bought Red Hat and things went south. CentOS did not fail, CentOS was killed. It's different.

wfuser 6y ago
> Then IBM bought Red Hat and things went south. I think IBM had nothing to do with this one. People think Red Hat won't do anything like this. However, there exists a part of Red Hat which is capable of doing this. That part of Red Hat usually stays behind the scenes and comes to the fore to announce that a decision has been made and the developers (hired within Red Hat as well as the general community) who are involved in the day to day running of the projects will have no say. Plus they will throw in some confusion (like the limited use license that is in the works but not yet ready for CentOS use) around the future of the project being killed just to let the community expect something good to come out of this exercise. This is not new. They did the exact same thing to the JBoss community application server[1]. [1] https://news.ycombinator.com/item?id=25358847 https://news.ycombinator.com/item?id=25358847
playcache 6y ago
IBM had nothing to do with it.
- johannes1234321 6y ago
  While I think that the decision was done before IBM took over, that statement is too much imo. IBM could have stopped the plan, if they wanted to.

goatinaboat 6y ago

Whats missing is an analysis of why CentOS failed. It did not fail. It succeeded too well! It was impacting RH’s bottom line, so they killed it.