10 ms·

I'm continually baffled that so many companies follow RHEL compatibility to this day. I've been using Linux for nearly 30 years. Admining as a profession for a

I'm continually baffled that so many companies follow RHEL compatibility to this day. I've been using Linux for nearly 30 years. Admining as a profession for at least a quarter of that. 20 years ago, it made a ton of sense. Today, less so. The 'stable version but we backport patches' mantra doesn't make any sense today. I can't even describe how many things that have broken that you can't even find an answer for because it was some RH specific patch. Between Debian, Nix, Arch, and others, I can't figure out why RHEL compat is so desirable. I'll go as far as to say that my Arch boxes have been far more predictable than my RHEL boxes.

pcw888 3y ago
It's just been the corporate default for so long that people learn it for their jobs and get comfortable with it
- vetinari 3y ago
  It is a standardized, known quantity. Just like McDonald's: it isn't the best meal you can get, whether in quality, nutrition or taste. But you will know what you are getting, even when you appear on the other side of the planet. You would be better off with a meal in a some local restaurant, but that one is unknown to you. So corporate will get something they know for their people. Most of them have zero interest in exploring better options, just want their job done today and go home.
pipo234 3y ago
"Nobody ever got fired for buying CentOS". Till recently. In terms of actual, material benefits I believe it's simply because transitive "certifications" (hardware, support, continuity, ...) that are implied by Red Hat's corporate presence.
- pipo234 3y ago
  To add to this from personal experience; we sell proprietary software that runs on multiple platforms. Our market is broadcasters, CDNs, telcos, media companies. We target Ubuntu and Alpine, but also ship CentOS/Rocky/RHEL/Oracle/Alma and Windows. Even though customers rarely ask for the latter targets, we need to offer those simply because we wouldn't be taken seriously if this major market segment were absent on our website.
  - generalizations 3y ago
    I’m surprised (and happy) to hear that you target alpine. Is it really in use in those market segments?
    - ilyt 3y ago
      Nope, but it's used as basis of Docker containers a lot
znpy 3y ago
[flagged]
- sgarland 3y ago
  Hell, even in my homelab I’ve moved past wanting new shiny. I just want NFS to play nicely and stop silently failing with nary a log message. Playing around with distros and bleeding edge is fun until you just want the OS to get out of your way so you can do stuff. Then you regret your decisions. I don’t run Arch, BTW. Debian.
- ZeroSolstice 3y ago
  So if stability is a strict requirement why are you dependent on an OS that you have no contractual obligation with? I mean if the community OS model was working for the CentOS team why go under the RedHat umbrella? My guess is going to be people demanding enterprise support without enterprise licensing costs[1]. "In 2014, the CentOS development team still had a distribution with far more marketshare than resources." [1]https://arstechnica.com/gadgets/2020/12/centos-shifts-from-red-hat-unbranded-to-red-hat-beta/ https://arstechnica.com/gadgets/2020/12/centos-shifts-from-r...
b112 3y ago
The 'stable version but we backport patches' mantra Just because Redhat handles this poorly (like pretty much everything), does not make the model bad. For example, almost every single security update on Debian, is backported when not handled by upstream. And there is a lot of that. You lose so much stability, if you track all-new. In fact, you literally spend more time chasing underlying bugs, instead your bugs, and dealing with api changes, if your stack is bleeding edge.
- usrusr 3y ago
  Makes me wonder who they are, the nameless heroes who run into all those bugs so we don't have to.
  - b112 3y ago
    Anyone running Debian testing, and filing bug reports, is your hero.
imiric 3y ago
Red Hat offers corporate support contracts, while other distros have community or consultant support at best. This is attractive to executive managers in corporations, as they can be seen as investing into a mature technology, and the systems are contractually guaranteed to keep running by a 3rd party, so shifting blame is easy.
- zvmaz 3y ago
  > so shifting blame is easy. But are the contracts actually useful? I wager that to have support from Red Hat is really useful beyond what you may view as "corporate politics".
  - getoffmyyawn 3y ago
    Yes, very. I previously worked for RH and there really wasn't a top end of support options. I knew of companies with a contract that had RH maintaining custom kernel patched for them. Another paid for a highly skilled SRE to be in their office ready to help anytime. A small number of companies are willing to pay a lot of money for these kinds of options.
  - neurostimulant 3y ago
    Maybe not, but when your certification requires using supported OS and patching know exploits as soon as possible, then you don't have much distro options to choose from if you want easier time to pass an audit.
  - kmac_ 3y ago
    Long-term support is critical. For example, Microsoft excels in that area and has patched its really old products to meet new PCI recommendations such as TLS 1.3, which can actually save a lot of money.
  - SSLy 3y ago
    We've submitted bugs related to IO, and had them promptly fixed.
- em-bee 3y ago
mst 3y ago
The time (some years back now, it was a 5.10 patch and a 5.8 rpm) they mis-backported a perl patch to work around a bug in a deprecated CPAN module for one of their enterprise customers and in the process caused a 2x-30x slowdown of lots of other newer code (including the library that had replaced it in the majority of production environments by that point) was 'fun'. Took me a couple years to get together a coalition of commercial support customers to apply sufficient pressure that they'd listen when I got the original author of the patch to break out the small words and crayon drawings and explain what they'd done wrong and how to backport it properly. Happily, they now generally snapshot Fedora's perl builds pretty directly and the current Fedora team have been fantastic to work with as a downstream but it was a spectacular mess at the time and there are plenty of people out there who still haven't forgiven them (I think I try to act -as if- I've forgiven them but I'm not sure I actually have).
- bonzini 3y ago
  To be honest it's a pretty good track record if you can only remember one instance of botching a backport, it's many years old, and it didn't have any security impact unlike Debian's ssh key generation.
  - pm215 3y ago
    True. The one I remember is when they mis-backported some version of libncurses so it was the v4 ABI but had the v3 soname, but that was like two decades ago :-)
  - mst 3y ago
    Given I work for https://shadow.cat/ https://shadow.cat/ and we do primarily perl/CPAN contracting, consultancy and commercial support (and don't run RHEL ourselves) I remember this one because I had to help multiple customers and probably by the end three digits' worth of companies via community support to work around the debacle. I suspect more man-hours were lost to the (now reversed for quite a while thanks to the Fedora team) decision to have their 'perl' package only be half a perl install (and to mass report the resulting problems that decision created to cpan authors without ever sending a single patch) but the original subject was things they botched by accident rather than things they broke deliberately. Though the uninformed arrogance behind the poor decisions and dismissive attitude to the resulting problems, even when they were impacting RHEL support customers, was very similar in both cases. Had they responded to realising they'd completely broken a bunch of paying customers' primary revenue generating applications by actually trying to do something about it I would have been happy to file the original mistake under "shit happens." As it is, I don't think 'pretty good track record' applies, I'm afraid.
PlutoIsAPlanet 3y ago
> I'm continually baffled that so many companies follow RHEL compatibility to this day. Because RHEL, and subsequently it's clones, are a really good server OS. You get updates for 10 years, Red Hat or it's employees maintain a large chunk of the generic-Linux stack, large software support, Cockpit, more modern features when compared to the Debian version at the same time (e.g. dracut, firewalld, systemd, NetworkManager, Podman etc). Then when you want commercial support, converting to RHEL (or vice versa) it's painless and quick.
tannhaeuser 3y ago
> I can't figure out why RHEL compat is so desirable RHEL compat per se is irrelevant, it's the 10 years guaranteed maintenance enterprise customers are after, plus a path forward for their third-party software investments only certified on RHEL (because of said guarantees) and internal IT deployment/admin line processes. Despite making the rounds on HN, enterprise and other commercial users give a flying fuck to io_uring, systemd (up until RHEL 7 when they were forced to), namespaces, and other Linux "innovations" which in fact are just annoyances to justify contracts for 30+ years of ongoing maintenance of an age-old POSIX core operating system once developed in a couple of months and praised for its minimalism.
- richrichardsson 3y ago
  So the enterprise and commercial users do care? Or perhaps they "don't give a flying fuck"? This ommission of the negative really grinds my gears, and I couldn't care less about American Idiom when it literally changes the meaning of the words.
  - tannhaeuser 3y ago
    > I couldn't care less about American Idiom Sorry for my choice of words, you may be right after all, English isn't my native language. But of all idioms, yours isn't without irony either ;)
  - messe 3y ago
    You know what grinds my gears? Needless pedantry and linguistic prescriptivism. > This ommission of the negative really grinds my gears, and I couldn't care less about American Idiom when it literally changes the meaning of the words. Tough luck? Language is like that, idioms especially. Just get used to the fact that not everybody speaks the same dialect of English as you do, and none of them are objectively the correct one in any sense. You clearly knew what they meant, so there's no communication barrier here. You're being pedantic for the sake of it.
    - richrichardsson 3y ago
bandrami 3y ago
Our contract with the Air Force required that we document guarantees from every parts vendor for the servers we built for them that they would keep making those parts for at least a decade. They also demanded RHEL exclusively. It's a great example of how extreme stability is more important than any other question in a lot of business decisions.
- tourmalinetaco 3y ago
  The way my engineering professor put it was so: when even 5 minutes of off-time costs $5,000 of product you want to be sure your systems are stable.
  - kijin 3y ago
    $5,000 is pocket change. When it comes to the Air Force, 5 minutes of downtime at the wrong time could cost you a country, or perhaps even a planet.
  - BozeWolf 3y ago
    We are at 500k/hour for at least one system i know of at my current client (energy). Not sure what causes these expenses. It can be missing out on trading, or fines. There may be more of these systems :) Lots of redhat there.
    - gdprrrr 3y ago
      I've worked for an automotive company that calculates with 1 Million euros per hour. But that's for a just-in-time assembly line.
  - andrewaylett 3y ago
    That's the right sort of ballpark for the systems my team maintains. We don't run RHEL. "Stability" in the RHEL sense is akin to ossification: you don't change anything, and then you can't change anything, and then you've got problems. The costs aren't necessarily as obvious as pursuing stability via dynamism, and in any case can often either be avoided by limiting the lifespan of the system as a whole or at least punted on to a successor. For military applications, the trade off is even more intense: you build things hoping that they'll never be used, but knowing that they need to work when called upon. Stuff that'll be on the front line tomorrow can have a very different set of lifecycle guarantees from stuff that's got a planned life of 25 years but which (from experience) you know will probably still be around in 50.
B1FF_PSUVM 3y ago
Some software vendors - e.g. major engineering applications - only support their products on RHEL.
- terlisimo 3y ago
  Yep. A ton of proprietary software (like anything from IBM) supports only RHEL or SLES. If you open a support ticket and say "Hi I'm using Debian/Arch/Nix" they will just laugh and close the ticket.
  - mekster 3y ago
    How about Ubuntu?
    - natebc 3y ago
      For big enterprise software I'm around no Ubuntu is supported. SAP: https://community.sap.com/topics/linux/supported-platforms https://community.sap.com/topics/linux/supported-platforms PeopleSoft/Oracle RDBMS: https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/operating-system-requirements-for-x86-64-linux-platforms.html#GUID-3F647617-7FF5-4EE4-BBA8-9A5E3EEC888F https://docs.oracle.com/en/database/oracle/oracle-database/1...
- ZeroSolstice 3y ago
  Right, but I think the OP's question is why are you using RHEL compat instead of just getting licensed RHEL. In your example a company is already fine with paying full price for major engineering applications why wouldn't you do the same for your underlying OS.
  - B1FF_PSUVM 3y ago
    I've seen it done, although small-scale, many years ago. But yielding that point would mean that RH/IBM would have you over a barrel on pricing. Probably why Suse is forcing the issue. P.S. and Oracle is piling on ...
bayindirh 3y ago
> I can't figure out why RHEL compat is so desirable. When you're a part of an international research infrastructure which works on the same stack for a decade and half, being supported by RH for free (because of CERN) and creating tons of software at every layer of this research stack over that time period is one of the bigger reasons, but it's not the only one. I personally use Debian for my personal computers and small servers elsewhere for 15+ years, but when it comes to infrastructure homogenization amongst high thousands of nodes, it's a different story.
coldtea 3y ago
>The 'stable version but we backport patches' mantra doesn't make any sense today. Why, did dependability and enteprise long term stability needs go out of fashion?
- ZeroSolstice 3y ago
  How exactly is the dependability and stability determined? By back-porting you are already changing the original version of software. What then becomes the difference at that point verse tracking the actual project version? Assuming both go through regular testing. It seems more straight forward to be able to look at a change log from the application, lets say httpd, then to go search for cherry picked back-porting changes. https://downloads.apache.org/httpd/CHANGES_2.4 https://downloads.apache.org/httpd/CHANGES_2.4 verse https://centos.pkgs.org/7/centos-updates-x86_64/httpd-2.4.6-99.el7.centos.1.x86_64.rpm.html https://centos.pkgs.org/7/centos-updates-x86_64/httpd-2.4.6-... I just picked httpd since its a common application and as far as I can tell all the updates are minor release numbers. If you have another example that would be useful to see for consideration as I'm genuinely curious which applications change so much that people are dependent upon back-porting for stability.
itomato 3y ago
Grandfathered in during a period of perceived risk, mostly.
SanjayMehta 3y ago
Can't speak for other commercial tools but EDA tool vendors usually support very specific versions of Linux usually RHEL and SUSE. Until recently even Xilinx's tools would balk at installation on Centos. When the corp has spent millions on EDA tools, RHEL's yearly subscription looks like peanuts.
freeone3000 3y ago
> I can't figure out why RHEL compat is so desirable. Hey, here’s a comprehensive guide complete with commands, expected outcomes, and side effects of each command for setting up a Windows Domain using an RHEL domain controller: https://access.redhat.com/documentation/en-us/red_hat_single_sign-on/7.5/html/server_administration_guide/identity_broker https://access.redhat.com/documentation/en-us/red_hat_single... Oh, that’s for 7.5? One that’s 15 years old and no longer under standard support? It still works! Wait, but you’re a new customer. Sorry, here’s the one for 9: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/planning_identity_management/index https://access.redhat.com/documentation/en-us/red_hat_enterp... It is like this for everything. The documentation is clear, concise, comprehensive, complete, and versioned. This is what being RHEL compatible gets you.
- AndyMcConachie 3y ago
  I wish I could upvote this 10,000 times. Documentation isn't sexy or something. I don't know why Linux documentation sucks so much, but Redhat makes it suck must less. It takes lots of effort, money and writer hours to create the kind of documentation that Redhat maintains and businesses love documentation. A HOWTO or a FAQ are not documentation. I actually prefer man pages and I dropped Linux for my personal project years ago in favor of OpenBSD (sometimes FreeBSD) precisely because their man pages are well maintained and really well written. I still have to interact with Linux sometimes and it's almost always Debian or Redhat. And I gotta say Redhat beats Debian on documentation.
  - mjw1007 3y ago
    Well, part of the reason Linux documentation often isn't very good is that Red Hat doesn't treat documentation work as something that should be submitted upstream. Their incentives favour creating Red-Hat-specific documentation, and this then reduces their users' incentive to improve upstream documentation.
    - bonzini 3y ago
      There is a difference between package documentation and system documentation. Red Hat contributes package documentation upstream, but system documentation is by its nature distro-dependent. Red Hat has contributed work that made distros more homogeneous (cough systemd cough) and excellent package documentation for that work; but also got flak for that...
rwmj 3y ago
I wrote this a few months ago explaining why Red Hat makes money: https://news.ycombinator.com/item?id=35588297 https://news.ycombinator.com/item?id=35588297 (Still amazed that no one understands these basic points)
- ZeroSolstice 3y ago
  I'm not totally surprised as most employee's don't know how they fit into their current organization as a whole or how their organization fits into their current market, never mind other companies and vendors. Its along the same lines as people evaluating company licensing and equipment costs as if its coming out of their pocket. Its almost as bad as people believing that Walmart, Target, etc are closing their stores because of "so much theft" as if shrink budgets weren't a thing. Having a loss prevention department is a misnomer, its a paperwork and accounting department so that the losses can be written off to the IRS. These places have KPI's for everything and probably know the locations where they have a store but a majority of people are ordering online. I'm sure their lease has some notion of financial hardship which "theft" falls into allowing them to exit the contract early.
  - _ea1k 3y ago
    > Its almost as bad as people believing that Walmart, Target, etc are closing their stores because of "so much theft" as if shrink budgets weren't a thing. I may not be understanding your point here, but having a shrink budget and closing due to theft aren't mutually exclusive. It is possible for theft to exceed the shrink budget and make a store too costly to operate.
    - ZeroSolstice 3y ago
      You are most certainly correct and I suppose without the actual numbers its a bit of speculation. My main point was that there is no additional cost, even if you exceed the shrink budget. The losses are all written off because with a loss prevention department you have shown, to the insurance company /or IRS, that you are putting in a reasonable effort to deter theft. Stores don't want anyone to actually stop people for a number of reasons the main one being they are writing the loss off or being reimbursed. An employee getting harmed, the thief getting harmed, employee's getting summons to court, another customer getting harmed are all more costly than just documenting what was stolen. Even in the cases where they build up evidence so that the amount gets close to grand larceny is just paper work. They already have people and technology deployed to do the work its not a new investment. I did a bit of digging and while "theft" is listed as one of the items "reduced foot-traffic"[2] seems to be the main culprit. They throw out some fancy percentages but the quoted stats seem to be from 2015 and theft is only 1% of their annual revenue[7]. It would be interesting to see the raw data. [1] https://www.jacksonville.com/story/news/2015/01/15/after-taking-too-much-too-fast-target-closing-stores-canada/15647123007/ https://www.jacksonville.com/story/news/2015/01/15/after-tak... [2] https://www.al.com/news/2023/04/target-store-closings-2023-see-the-full-list.html https://www.al.com/news/2023/04/target-store-closings-2023-s... [3] https://www.cleveland.com/news/2023/04/target-to-close-stores-in-four-midwest-east-coast-states-but-none-in-ohio.html https://www.cleveland.com/news/2023/04/target-to-close-store... [4] https://www.irs.gov/pub/irs-pdf/p584b.pdf https://www.irs.gov/pub/irs-pdf/p584b.pdf [5] https://www.reuters.com/article/us-wal-mart-stores-theft/wal-marts-u-s-chief-takes-aim-at-urgent-item-theft-idUKKBN0OK2NQ20150607 https://www.reuters.com/article/us-wal-mart-stores-theft/wal... (from 2015) [6] https://www.wsaz.com/2022/12/09/walmart-may-close-stores-increase-prices-due-theft-ceo-says/ https://www.wsaz.com/2022/12/09/walmart-may-close-stores-inc... [7] https://blog.gitnux.com/walmart-theft-statistics/ https://blog.gitnux.com/walmart-theft-statistics/
pelasaco 3y ago
> I can't figure out why RHEL compat is so desirable. If you cannot figure out, then you are not working with Linux, despite 30y+ of experience. Big Companies need certifications, maintenance, support and compatibility is necessary. Specially companies running linux in mainframe, or for instance, delivering embedded devices like storages or medical devices
- ZeroSolstice 3y ago
  Why would you be dependent upon CentOS(RHEL compat) if you are in those industries? If you need certification and LTS then pay for it by purchasing RHEL licenses. It seems very silly for people to be getting upset now that a free OS hasn't taken their business considerations to heart. Unless of course you were going for the model of charging everyone enterprise rates for your devices/software/work/etc and pocketing what you would have to normally pay in enterprise licensing because its a "community" OS, which everyone is free to do, but these are the risks that come with that.
  - pelasaco 3y ago
    One scenario that I can imagine, is for development environment. People had CentOS as dev environment and deployed RHEL. Or ran CentOS guests on RHEL hosts (VM or Docker)
    - ZeroSolstice 3y ago
      Thats a good point. Looks like RedHat considered this and offer free company/professional developer RHEL images for non-production use[1]. In 2021 they allowed zero-cost licensing for individuals[2]. [1] https://developers.redhat.com/articles/2022/05/10/access-rhel-developer-teams-subscription https://developers.redhat.com/articles/2022/05/10/access-rhe... [2] https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux https://developers.redhat.com/articles/faqs-no-cost-red-hat-...
ZeroSolstice 3y ago
You are correct on this observation. The answer is most people/organizations don't actually know if they need to be RHEL compatible. Outside of any places that have a compliance requirements, where saying you are RHEL compatible by using CentOS is the easiest path to checking a box. However if you are operating in an arena where compliance certification is needed you would just pay for RHEL and charge the cost to your customer. We paid for support for RHEV/RHV for a number of years and it was useless. You ended up troubleshooting most of the work yourself and in our case we could have just run Ovirt and skipped the whole inventorying and licensing of cores. The "having someone to blame" or a support contract baffles me for open source. Why advocate for using open source if executive suite is concerned with downtime? They aren't passing the savings back to you and what did having a contract save you if you have to get director or c-level people on the phone to escalate? Presumably they are getting paid more per hour than a number of other people. You have now paid support, plus Dev/Ops troubleshooting, then getting an executive involved. Not to mention how big does your organization need to be for a vendor to consider "your" issue to be a real issue to them. The back porting and LTS that most people talk about seems to be out of place for a time in which security is now falling under insurance and you need to be regularly updating anyway. Every security audit has one of these "banner shows this version, oh but wait did you check OVAL, or its a back-ported version" conversations. Wasn't this the point of all the fancy DevOps tools, workflows, containerization, etc. Shouldn't you be able to roll out new packages and OS versions quickly since you have all your automated tests implemented?
deleted 3y ago
[deleted]
bityard 3y ago
When enterprise-sized IT departments wanted to switch away from cumbersome, expensive commercial Unix systems, RHEL was there to step in. As a result, it became the default enterprise Linux distribution and vendor. The reason why it is still special is because even in 2023, if any commercial enterprise applications and hardware support Linux at all, it is generally only RHEL.

pcw888 3y ago

It's just been the corporate default for so long that people learn it for their jobs and get comfortable with it

vetinari 3y ago
It is a standardized, known quantity. Just like McDonald's: it isn't the best meal you can get, whether in quality, nutrition or taste. But you will know what you are getting, even when you appear on the other side of the planet. You would be better off with a meal in a some local restaurant, but that one is unknown to you. So corporate will get something they know for their people. Most of them have zero interest in exploring better options, just want their job done today and go home.

pipo234 3y ago

"Nobody ever got fired for buying CentOS". Till recently. In terms of actual, material benefits I believe it's simply because transitive "certifications" (hardware, support, continuity, ...) that are implied by Red Hat's corporate presence.

pipo234 3y ago
To add to this from personal experience; we sell proprietary software that runs on multiple platforms. Our market is broadcasters, CDNs, telcos, media companies. We target Ubuntu and Alpine, but also ship CentOS/Rocky/RHEL/Oracle/Alma and Windows. Even though customers rarely ask for the latter targets, we need to offer those simply because we wouldn't be taken seriously if this major market segment were absent on our website.
- generalizations 3y ago
  I’m surprised (and happy) to hear that you target alpine. Is it really in use in those market segments?
  - ilyt 3y ago
    Nope, but it's used as basis of Docker containers a lot

znpy 3y ago

[flagged]

sgarland 3y ago
Hell, even in my homelab I’ve moved past wanting new shiny. I just want NFS to play nicely and stop silently failing with nary a log message. Playing around with distros and bleeding edge is fun until you just want the OS to get out of your way so you can do stuff. Then you regret your decisions. I don’t run Arch, BTW. Debian.
ZeroSolstice 3y ago
So if stability is a strict requirement why are you dependent on an OS that you have no contractual obligation with? I mean if the community OS model was working for the CentOS team why go under the RedHat umbrella? My guess is going to be people demanding enterprise support without enterprise licensing costs[1]. "In 2014, the CentOS development team still had a distribution with far more marketshare than resources." [1]https://arstechnica.com/gadgets/2020/12/centos-shifts-from-red-hat-unbranded-to-red-hat-beta/ https://arstechnica.com/gadgets/2020/12/centos-shifts-from-r...

b112 3y ago

The 'stable version but we backport patches' mantra Just because Redhat handles this poorly (like pretty much everything), does not make the model bad. For example, almost every single security update on Debian, is backported when not handled by upstream. And there is a lot of that. You lose so much stability, if you track all-new. In fact, you literally spend more time chasing underlying bugs, instead your bugs, and dealing with api changes, if your stack is bleeding edge.

usrusr 3y ago
Makes me wonder who they are, the nameless heroes who run into all those bugs so we don't have to.
- b112 3y ago
  Anyone running Debian testing, and filing bug reports, is your hero.

imiric 3y ago

Red Hat offers corporate support contracts, while other distros have community or consultant support at best. This is attractive to executive managers in corporations, as they can be seen as investing into a mature technology, and the systems are contractually guaranteed to keep running by a 3rd party, so shifting blame is easy.

zvmaz 3y ago
> so shifting blame is easy. But are the contracts actually useful? I wager that to have support from Red Hat is really useful beyond what you may view as "corporate politics".
- getoffmyyawn 3y ago
  Yes, very. I previously worked for RH and there really wasn't a top end of support options. I knew of companies with a contract that had RH maintaining custom kernel patched for them. Another paid for a highly skilled SRE to be in their office ready to help anytime. A small number of companies are willing to pay a lot of money for these kinds of options.
- neurostimulant 3y ago
  Maybe not, but when your certification requires using supported OS and patching know exploits as soon as possible, then you don't have much distro options to choose from if you want easier time to pass an audit.
- kmac_ 3y ago
  Long-term support is critical. For example, Microsoft excels in that area and has patched its really old products to meet new PCI recommendations such as TLS 1.3, which can actually save a lot of money.
- SSLy 3y ago
  We've submitted bugs related to IO, and had them promptly fixed.
em-bee 3y ago

mst 3y ago

The time (some years back now, it was a 5.10 patch and a 5.8 rpm) they mis-backported a perl patch to work around a bug in a deprecated CPAN module for one of their enterprise customers and in the process caused a 2x-30x slowdown of lots of other newer code (including the library that had replaced it in the majority of production environments by that point) was 'fun'. Took me a couple years to get together a coalition of commercial support customers to apply sufficient pressure that they'd listen when I got the original author of the patch to break out the small words and crayon drawings and explain what they'd done wrong and how to backport it properly. Happily, they now generally snapshot Fedora's perl builds pretty directly and the current Fedora team have been fantastic to work with as a downstream but it was a spectacular mess at the time and there are plenty of people out there who still haven't forgiven them (I think I try to act -as if- I've forgiven them but I'm not sure I actually have).

bonzini 3y ago
To be honest it's a pretty good track record if you can only remember one instance of botching a backport, it's many years old, and it didn't have any security impact unlike Debian's ssh key generation.
- pm215 3y ago
  True. The one I remember is when they mis-backported some version of libncurses so it was the v4 ABI but had the v3 soname, but that was like two decades ago :-)
- mst 3y ago
  Given I work for https://shadow.cat/ https://shadow.cat/ and we do primarily perl/CPAN contracting, consultancy and commercial support (and don't run RHEL ourselves) I remember this one because I had to help multiple customers and probably by the end three digits' worth of companies via community support to work around the debacle. I suspect more man-hours were lost to the (now reversed for quite a while thanks to the Fedora team) decision to have their 'perl' package only be half a perl install (and to mass report the resulting problems that decision created to cpan authors without ever sending a single patch) but the original subject was things they botched by accident rather than things they broke deliberately. Though the uninformed arrogance behind the poor decisions and dismissive attitude to the resulting problems, even when they were impacting RHEL support customers, was very similar in both cases. Had they responded to realising they'd completely broken a bunch of paying customers' primary revenue generating applications by actually trying to do something about it I would have been happy to file the original mistake under "shit happens." As it is, I don't think 'pretty good track record' applies, I'm afraid.

PlutoIsAPlanet 3y ago

> I'm continually baffled that so many companies follow RHEL compatibility to this day. Because RHEL, and subsequently it's clones, are a really good server OS. You get updates for 10 years, Red Hat or it's employees maintain a large chunk of the generic-Linux stack, large software support, Cockpit, more modern features when compared to the Debian version at the same time (e.g. dracut, firewalld, systemd, NetworkManager, Podman etc). Then when you want commercial support, converting to RHEL (or vice versa) it's painless and quick.

tannhaeuser 3y ago

> I can't figure out why RHEL compat is so desirable RHEL compat per se is irrelevant, it's the 10 years guaranteed maintenance enterprise customers are after, plus a path forward for their third-party software investments only certified on RHEL (because of said guarantees) and internal IT deployment/admin line processes. Despite making the rounds on HN, enterprise and other commercial users give a flying fuck to io_uring, systemd (up until RHEL 7 when they were forced to), namespaces, and other Linux "innovations" which in fact are just annoyances to justify contracts for 30+ years of ongoing maintenance of an age-old POSIX core operating system once developed in a couple of months and praised for its minimalism.

richrichardsson 3y ago
So the enterprise and commercial users do care? Or perhaps they "don't give a flying fuck"? This ommission of the negative really grinds my gears, and I couldn't care less about American Idiom when it literally changes the meaning of the words.
- tannhaeuser 3y ago
  > I couldn't care less about American Idiom Sorry for my choice of words, you may be right after all, English isn't my native language. But of all idioms, yours isn't without irony either ;)
- messe 3y ago
  You know what grinds my gears? Needless pedantry and linguistic prescriptivism. > This ommission of the negative really grinds my gears, and I couldn't care less about American Idiom when it literally changes the meaning of the words. Tough luck? Language is like that, idioms especially. Just get used to the fact that not everybody speaks the same dialect of English as you do, and none of them are objectively the correct one in any sense. You clearly knew what they meant, so there's no communication barrier here. You're being pedantic for the sake of it.
  - richrichardsson 3y ago

bandrami 3y ago

Our contract with the Air Force required that we document guarantees from every parts vendor for the servers we built for them that they would keep making those parts for at least a decade. They also demanded RHEL exclusively. It's a great example of how extreme stability is more important than any other question in a lot of business decisions.

tourmalinetaco 3y ago
The way my engineering professor put it was so: when even 5 minutes of off-time costs $5,000 of product you want to be sure your systems are stable.
- kijin 3y ago
  $5,000 is pocket change. When it comes to the Air Force, 5 minutes of downtime at the wrong time could cost you a country, or perhaps even a planet.
- BozeWolf 3y ago
  We are at 500k/hour for at least one system i know of at my current client (energy). Not sure what causes these expenses. It can be missing out on trading, or fines. There may be more of these systems :) Lots of redhat there.
  - gdprrrr 3y ago
    I've worked for an automotive company that calculates with 1 Million euros per hour. But that's for a just-in-time assembly line.
- andrewaylett 3y ago
  That's the right sort of ballpark for the systems my team maintains. We don't run RHEL. "Stability" in the RHEL sense is akin to ossification: you don't change anything, and then you can't change anything, and then you've got problems. The costs aren't necessarily as obvious as pursuing stability via dynamism, and in any case can often either be avoided by limiting the lifespan of the system as a whole or at least punted on to a successor. For military applications, the trade off is even more intense: you build things hoping that they'll never be used, but knowing that they need to work when called upon. Stuff that'll be on the front line tomorrow can have a very different set of lifecycle guarantees from stuff that's got a planned life of 25 years but which (from experience) you know will probably still be around in 50.

B1FF_PSUVM 3y ago

Some software vendors - e.g. major engineering applications - only support their products on RHEL.

terlisimo 3y ago
Yep. A ton of proprietary software (like anything from IBM) supports only RHEL or SLES. If you open a support ticket and say "Hi I'm using Debian/Arch/Nix" they will just laugh and close the ticket.
- mekster 3y ago
  How about Ubuntu?
  - natebc 3y ago
    For big enterprise software I'm around no Ubuntu is supported. SAP: https://community.sap.com/topics/linux/supported-platforms https://community.sap.com/topics/linux/supported-platforms PeopleSoft/Oracle RDBMS: https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/operating-system-requirements-for-x86-64-linux-platforms.html#GUID-3F647617-7FF5-4EE4-BBA8-9A5E3EEC888F https://docs.oracle.com/en/database/oracle/oracle-database/1...
ZeroSolstice 3y ago
Right, but I think the OP's question is why are you using RHEL compat instead of just getting licensed RHEL. In your example a company is already fine with paying full price for major engineering applications why wouldn't you do the same for your underlying OS.
- B1FF_PSUVM 3y ago
  I've seen it done, although small-scale, many years ago. But yielding that point would mean that RH/IBM would have you over a barrel on pricing. Probably why Suse is forcing the issue. P.S. and Oracle is piling on ...

bayindirh 3y ago

> I can't figure out why RHEL compat is so desirable. When you're a part of an international research infrastructure which works on the same stack for a decade and half, being supported by RH for free (because of CERN) and creating tons of software at every layer of this research stack over that time period is one of the bigger reasons, but it's not the only one. I personally use Debian for my personal computers and small servers elsewhere for 15+ years, but when it comes to infrastructure homogenization amongst high thousands of nodes, it's a different story.

coldtea 3y ago

>The 'stable version but we backport patches' mantra doesn't make any sense today. Why, did dependability and enteprise long term stability needs go out of fashion?

ZeroSolstice 3y ago
How exactly is the dependability and stability determined? By back-porting you are already changing the original version of software. What then becomes the difference at that point verse tracking the actual project version? Assuming both go through regular testing. It seems more straight forward to be able to look at a change log from the application, lets say httpd, then to go search for cherry picked back-porting changes. https://downloads.apache.org/httpd/CHANGES_2.4 https://downloads.apache.org/httpd/CHANGES_2.4 verse https://centos.pkgs.org/7/centos-updates-x86_64/httpd-2.4.6-99.el7.centos.1.x86_64.rpm.html https://centos.pkgs.org/7/centos-updates-x86_64/httpd-2.4.6-... I just picked httpd since its a common application and as far as I can tell all the updates are minor release numbers. If you have another example that would be useful to see for consideration as I'm genuinely curious which applications change so much that people are dependent upon back-porting for stability.

itomato 3y ago

Grandfathered in during a period of perceived risk, mostly.

SanjayMehta 3y ago

Can't speak for other commercial tools but EDA tool vendors usually support very specific versions of Linux usually RHEL and SUSE. Until recently even Xilinx's tools would balk at installation on Centos. When the corp has spent millions on EDA tools, RHEL's yearly subscription looks like peanuts.

freeone3000 3y ago

> I can't figure out why RHEL compat is so desirable. Hey, here’s a comprehensive guide complete with commands, expected outcomes, and side effects of each command for setting up a Windows Domain using an RHEL domain controller: https://access.redhat.com/documentation/en-us/red_hat_single_sign-on/7.5/html/server_administration_guide/identity_broker https://access.redhat.com/documentation/en-us/red_hat_single... Oh, that’s for 7.5? One that’s 15 years old and no longer under standard support? It still works! Wait, but you’re a new customer. Sorry, here’s the one for 9: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/planning_identity_management/index https://access.redhat.com/documentation/en-us/red_hat_enterp... It is like this for everything. The documentation is clear, concise, comprehensive, complete, and versioned. This is what being RHEL compatible gets you.

AndyMcConachie 3y ago
I wish I could upvote this 10,000 times. Documentation isn't sexy or something. I don't know why Linux documentation sucks so much, but Redhat makes it suck must less. It takes lots of effort, money and writer hours to create the kind of documentation that Redhat maintains and businesses love documentation. A HOWTO or a FAQ are not documentation. I actually prefer man pages and I dropped Linux for my personal project years ago in favor of OpenBSD (sometimes FreeBSD) precisely because their man pages are well maintained and really well written. I still have to interact with Linux sometimes and it's almost always Debian or Redhat. And I gotta say Redhat beats Debian on documentation.
- mjw1007 3y ago
  Well, part of the reason Linux documentation often isn't very good is that Red Hat doesn't treat documentation work as something that should be submitted upstream. Their incentives favour creating Red-Hat-specific documentation, and this then reduces their users' incentive to improve upstream documentation.
  - bonzini 3y ago
    There is a difference between package documentation and system documentation. Red Hat contributes package documentation upstream, but system documentation is by its nature distro-dependent. Red Hat has contributed work that made distros more homogeneous (cough systemd cough) and excellent package documentation for that work; but also got flak for that...

rwmj 3y ago

I wrote this a few months ago explaining why Red Hat makes money: https://news.ycombinator.com/item?id=35588297 https://news.ycombinator.com/item?id=35588297 (Still amazed that no one understands these basic points)

ZeroSolstice 3y ago
I'm not totally surprised as most employee's don't know how they fit into their current organization as a whole or how their organization fits into their current market, never mind other companies and vendors. Its along the same lines as people evaluating company licensing and equipment costs as if its coming out of their pocket. Its almost as bad as people believing that Walmart, Target, etc are closing their stores because of "so much theft" as if shrink budgets weren't a thing. Having a loss prevention department is a misnomer, its a paperwork and accounting department so that the losses can be written off to the IRS. These places have KPI's for everything and probably know the locations where they have a store but a majority of people are ordering online. I'm sure their lease has some notion of financial hardship which "theft" falls into allowing them to exit the contract early.
- _ea1k 3y ago
  > Its almost as bad as people believing that Walmart, Target, etc are closing their stores because of "so much theft" as if shrink budgets weren't a thing. I may not be understanding your point here, but having a shrink budget and closing due to theft aren't mutually exclusive. It is possible for theft to exceed the shrink budget and make a store too costly to operate.
  - ZeroSolstice 3y ago
    You are most certainly correct and I suppose without the actual numbers its a bit of speculation. My main point was that there is no additional cost, even if you exceed the shrink budget. The losses are all written off because with a loss prevention department you have shown, to the insurance company /or IRS, that you are putting in a reasonable effort to deter theft. Stores don't want anyone to actually stop people for a number of reasons the main one being they are writing the loss off or being reimbursed. An employee getting harmed, the thief getting harmed, employee's getting summons to court, another customer getting harmed are all more costly than just documenting what was stolen. Even in the cases where they build up evidence so that the amount gets close to grand larceny is just paper work. They already have people and technology deployed to do the work its not a new investment. I did a bit of digging and while "theft" is listed as one of the items "reduced foot-traffic"[2] seems to be the main culprit. They throw out some fancy percentages but the quoted stats seem to be from 2015 and theft is only 1% of their annual revenue[7]. It would be interesting to see the raw data. [1] https://www.jacksonville.com/story/news/2015/01/15/after-taking-too-much-too-fast-target-closing-stores-canada/15647123007/ https://www.jacksonville.com/story/news/2015/01/15/after-tak... [2] https://www.al.com/news/2023/04/target-store-closings-2023-see-the-full-list.html https://www.al.com/news/2023/04/target-store-closings-2023-s... [3] https://www.cleveland.com/news/2023/04/target-to-close-stores-in-four-midwest-east-coast-states-but-none-in-ohio.html https://www.cleveland.com/news/2023/04/target-to-close-store... [4] https://www.irs.gov/pub/irs-pdf/p584b.pdf https://www.irs.gov/pub/irs-pdf/p584b.pdf [5] https://www.reuters.com/article/us-wal-mart-stores-theft/wal-marts-u-s-chief-takes-aim-at-urgent-item-theft-idUKKBN0OK2NQ20150607 https://www.reuters.com/article/us-wal-mart-stores-theft/wal... (from 2015) [6] https://www.wsaz.com/2022/12/09/walmart-may-close-stores-increase-prices-due-theft-ceo-says/ https://www.wsaz.com/2022/12/09/walmart-may-close-stores-inc... [7] https://blog.gitnux.com/walmart-theft-statistics/ https://blog.gitnux.com/walmart-theft-statistics/

pelasaco 3y ago

> I can't figure out why RHEL compat is so desirable. If you cannot figure out, then you are not working with Linux, despite 30y+ of experience. Big Companies need certifications, maintenance, support and compatibility is necessary. Specially companies running linux in mainframe, or for instance, delivering embedded devices like storages or medical devices

ZeroSolstice 3y ago
Why would you be dependent upon CentOS(RHEL compat) if you are in those industries? If you need certification and LTS then pay for it by purchasing RHEL licenses. It seems very silly for people to be getting upset now that a free OS hasn't taken their business considerations to heart. Unless of course you were going for the model of charging everyone enterprise rates for your devices/software/work/etc and pocketing what you would have to normally pay in enterprise licensing because its a "community" OS, which everyone is free to do, but these are the risks that come with that.
- pelasaco 3y ago
  One scenario that I can imagine, is for development environment. People had CentOS as dev environment and deployed RHEL. Or ran CentOS guests on RHEL hosts (VM or Docker)
  - ZeroSolstice 3y ago
    Thats a good point. Looks like RedHat considered this and offer free company/professional developer RHEL images for non-production use[1]. In 2021 they allowed zero-cost licensing for individuals[2]. [1] https://developers.redhat.com/articles/2022/05/10/access-rhel-developer-teams-subscription https://developers.redhat.com/articles/2022/05/10/access-rhe... [2] https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux https://developers.redhat.com/articles/faqs-no-cost-red-hat-...

ZeroSolstice 3y ago

You are correct on this observation. The answer is most people/organizations don't actually know if they need to be RHEL compatible. Outside of any places that have a compliance requirements, where saying you are RHEL compatible by using CentOS is the easiest path to checking a box. However if you are operating in an arena where compliance certification is needed you would just pay for RHEL and charge the cost to your customer. We paid for support for RHEV/RHV for a number of years and it was useless. You ended up troubleshooting most of the work yourself and in our case we could have just run Ovirt and skipped the whole inventorying and licensing of cores. The "having someone to blame" or a support contract baffles me for open source. Why advocate for using open source if executive suite is concerned with downtime? They aren't passing the savings back to you and what did having a contract save you if you have to get director or c-level people on the phone to escalate? Presumably they are getting paid more per hour than a number of other people. You have now paid support, plus Dev/Ops troubleshooting, then getting an executive involved. Not to mention how big does your organization need to be for a vendor to consider "your" issue to be a real issue to them. The back porting and LTS that most people talk about seems to be out of place for a time in which security is now falling under insurance and you need to be regularly updating anyway. Every security audit has one of these "banner shows this version, oh but wait did you check OVAL, or its a back-ported version" conversations. Wasn't this the point of all the fancy DevOps tools, workflows, containerization, etc. Shouldn't you be able to roll out new packages and OS versions quickly since you have all your automated tests implemented?

deleted 3y ago

[deleted]

bityard 3y ago

When enterprise-sized IT departments wanted to switch away from cumbersome, expensive commercial Unix systems, RHEL was there to step in. As a result, it became the default enterprise Linux distribution and vendor. The reason why it is still special is because even in 2023, if any commercial enterprise applications and hardware support Linux at all, it is generally only RHEL.