15 ms·

What's wrong with enterprise Linux

204 points by cylo 3y ago

mbreese 3y ago
> The Case for Oracle Linux I did not see that twist coming. Strangely, the author makes a good point. Oracle or not, sticking closer to the upstream kernel is not a bad way to manage an “enterprise” kernel. Maybe we’ve all been too willing to sit back and accept how RHEL does kernels as “the way”.
- gjsman-1000 3y ago
  I actually did really like the point; but I would be very interested in hearing a rebuttal or counterargument from Red Hat. Linus' "security problems are just bugs" approach has been worrisome.
  - smarx007 3y ago
    I would expect that tracking the Linux kernel master just 4 weeks behind is going to cause more cases where, combined with enterprise software of typical quality, you'd need vendor support than with the RH way of changing as little as possible for as long as possible. And support is exactly what Oracle sells. To be clear, I only used Oracle Linux briefly while playing with the free tier of Oracle Cloud. Edit: sorry, but I don't get it, please break it down for me. Latest UEK 7 is based off 5.15 LTS ( https://docs.oracle.com/en/operating-systems/uek/ https://docs.oracle.com/en/operating-systems/uek/ ), that puts them at branching off a kernel released on 2021-10-31 (according to https://en.wikipedia.org/wiki/Linux_kernel_version_history#Releases_5.x.y https://en.wikipedia.org/wiki/Linux_kernel_version_history#R... ). How is that even remotely 4 weeks behind master?! Edit 2: my bad, I didn't read the "behind the tip of the LTS tree" part carefully. So, the difference is that Oracle tries to ship all LTS kernel updates, while RH tries to cherry-pick critical security and bug fixes only?
    - bonzini 3y ago
      Red Hat doesn't change as little as possible, at least 30% of the changes to Linux make it to RHEL. Most of the RHEL kernel is a few months behind upstream despite the old version. The next minor release of RHEL, to be released in November, probably will have features up to 6.3 for many subsystems and bugfixes up to 6.5 for example.
      - smarx007 3y ago
        Interesting, thank you for the details. This means that in some way, RHEL is ahead of Oracle Linux in terms of kernels? I don't see an UEK version tracking Linux 6.1 at all, AFAIK.
        cylo 3y ago
        It's true that Red Hat does pull in changes from upstream for several subsystems of Linux. It's genuinely a frankenkernel mixed with code from 6.3, 6.1, etc. But you're still beholden to what the Red Hat maintainers are pulling in and focusing on. It's still not a general follow upstream wholesale. You can see and track what UEK is doing by looking at their Github: https://github.com/oracle/linux-uek/commits/uek7/u1 https://github.com/oracle/linux-uek/commits/uek7/u1
        deleted 3y ago
        [deleted]
        bonzini 3y ago
        See here for an article that describes how it's done --> https://news.ycombinator.com/item?id=36763935 https://news.ycombinator.com/item?id=36763935
  - chabad360 3y ago
    In his defence, most bugs in the kernel can be exploited, so it doesn't necessarily make sense to treat a "bug" with a PoC better than one without.
  - picozeta 3y ago
    > Linus' "security problems are just bugs" approach has been worrisome. Aren't they?
- indymike 3y ago
  > Oracle or not, sticking closer to the upstream kernel is not a bad way to manage an “enterprise” kernel. This is probably a lot better than expecting it to be possible to maintain a secure release with a level of compatibility guarantee for 10+ years.
  - deleted 3y ago
    [deleted]
  - dralley 3y ago
    Maybe, but it's not obvious. There are plenty of novel vulnerabilities in interfaces like io_uring and so forth. It's not that those features are bad, I'm just saying that there are tradeoffs to always getting the new stuff. Maybe the compromise solution is to use newer kernels but keep certain features turned off until they "bake" properly.
- weare138 3y ago
  But what stops Oracle from just pulling a Red Hat when people switch over to Oracle Linux? Oracle hasn't exactly proven itself to be trustworthy over the years and that's putting it lightly. I think we need a fully open source alternative to RHEL not bound to any company. Something akin to the Debian project that can serve as an upstream reference distro and repository.
  - MatthiasPortzel 3y ago
    Oracle Linux, Red Hat, and Debian are all “fully open source”. What you mean is “not maintained by a corporation.”
    - marcosdumay 3y ago
      Well, RHEL is now only "fully open source" with quotes, and won't allow code redistribution anymore.
  - tosihakkeri 3y ago
    > I think we need a fully open source alternative to RHEL not bound to any company I believe the problem is not that there wouldn’t be open source alternatives but that that’s not what enterprise wants. Enterprise wants a company behind the distribution.
    - noizejoy 3y ago
      But arguably, there's also a pretty large world that wants enterprise type of solutions (for some use cases) without being actual enterprises. Or am I the only one?
    - weare138 3y ago
      Of course, think about the relationship between Ubuntu and Debian. The idea is the upstream would be a fully open-source project that vendors build off of downstream. This way we can prevent vendors from subjugating a distribution like with Red Hat and CentOS and if a vendor pulls out of the market or goes out of business it doesn't take down the ecosystem with it.
- zokier 3y ago
  At the same time it is not that convincing point because in lot of cases the value from freezing is pretty small for kernel compared to other software, because kernels strong "do not break userspace" attitude. Version freezing is far more valuable for various other projects that do not take backwards compatibility that seriously.
  - mbreese 3y ago
    I thought the Linux kernel had a strong “don’t break userspace” attitude, but it was a free for all in kernel space. If you’re developing kernel modules or have custom hardware with drivers, I could see having a backported kernel as being a major problem for support and development.
    - pxc 3y ago
      My (spectator's) understanding is that ABI compatibility is not a priority for kernelspace in Linux, but there is care taken for backwards compatibility in other ways (like at an API level). You're just expected to recompile your out-of-tree kernel modules when there's a new kernel version.
      - comex 3y ago
        Nope, API compatibility is also explicitly disavowed: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/stable-api-nonsense.rst https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
        pxc 3y ago
        Thank you for sharing this! Nothing helps you remember a fact like having been wrong about it in public. :)
- vbezhenar 3y ago
  If you're using proprietary hardware drivers which target RHEL, you need stable kernel. Also kernel updates breaking drivers is a thing. So there's definitely value with frozen kernel version. And its almost the same value as frozen software. It just works and it won't break after update.
- deleted 3y ago
  [deleted]
CrampusDestrus 3y ago
Holy.... that Oracle linux page is so friendly it almost makes me puke. I can't believe anyone would fall for them
- gjsman-1000 3y ago
  The same company that just two weeks ago was sending out unsolicited emails asking companies about Java licensing - a friendly shakedown for using the official JRE... https://news.ycombinator.com/item?id=36599118 https://news.ycombinator.com/item?id=36599118 Frankly, I don't get it Oracle. I would say the priorities are backwards - JRE free, Oracle Linux paid would make more objective sense (especially because the JRE used to be free... and there's a lot of free JREs out there.) Also, don't be stupid enough to download the VirtualBox Extensions package at work; or your work will probably get an email from a licensing agent.
  - azlev 3y ago
    It's an huge company with mixed incentives. Not everyone is align whatever their alignment is.
    - CrampusDestrus 3y ago
      >Yes, we know that this is Oracle, but... There might be different alignments, but you simply cannot write stuff like this and get away with it if any of your superiors disagree. That page literally admits Oracle is an asshole company
  - Tainnor 3y ago
    > Also, don't be stupid enough to download the VirtualBox Extensions package at work; or your work will probably get an email from a licensing agent. Place I used to work blocked the virtualbox home page because exactly this had happened.
  - count 3y ago
    I don't think there is any cross P&L strategy at play....
darkhelmet 3y ago
There is another problem that wasn't covered in the article. The 10+ years of stability leads to behaviors and outcomes that remind me of the long-lived SSL certificate problem. Updating is done so infrequently that the "how?" is forgotten. As the 10 year support limit approaches, most of the old team members who did it last time are gone, tech debt is through the roof, few people know where everything is or how to build it, and so on. Enterprise Linux "stability" enables all sorts of bad behavior if your company is inclined that way. LetsEncrypt did us a huge favor by forcing automation vs having the guy who knows how to update the SSL certs every 4.9 years and left 6 months ago. I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it. (I know, most HN folk are well disciplined but there are a lot of corporate cultures that are not.)
- derekp7 3y ago
  Don't you end up with the same problem with the automation that has been running fine for 5 years, then suddenly breaks? And the person that set it up is either gone, or has no clue how they did it 5 years ago.
  - bonzini 3y ago
    The idea is that you deploy from scratch all the infrastructure every 6 months, first to testing and then to production.
    - retbull 3y ago
      Every 6 months? That seems like a pretty long window for tribal knowledge to get lost. Is 6 months arbitrary or is there some reasoning behind that cadence?
      - noizejoy 3y ago
        Arguably, tribal knowledge and the dependence on it need to be managed as much as anything else.
azlev 3y ago
What's not clear to me is that newer software is safer or just less tested. Although it's possible that newer versions are more secure, I couldn't find evidence of it yet. (Links are welcome).
- dboreham 3y ago
  Probably more of an ass-covering situation: bugs are known in older software and not doing something about a known risk is obviously bad. otoh not mitigating an unknown and unidentified risk (bugs in new software) is ok.
- wredue 3y ago
  See. I used to think similarly. Newer software must be more secure, because we definitely know about all these problems and how to deal with them. What that doesn’t consider is that the vast majority of programmers simply don’t care to keep up with security concerns. And that’s why two of the most dangerous, easy to exploit, easy to stop attacks out there (SQL injection, XSS) are still among the most prevalent. As long as features are growing, the claims that new is more secure is difficult to believe.
- dylan604 3y ago
  that's why the whole rewrite in $languageX is so annoyingly frustrating when touted as fixing so many issues just because it uses $languageX. How many bugs not present in original version get introduce is always swept under the rug of the $languageX evangelists
- Pet_Ant 3y ago
  Isn't an unknown vulnerability preferrable to a known vulnerabilty?
  - jossclimb 3y ago
    There is unknown and unreported, a large amount of unreported vulns are sold on the blackmarket or hoarded for exploit by foreign adversaries.
  - 3y ago
im_down_w_otp 3y ago
I could imagine an updated take on “Enterprise Distro”, in order to take advantage of the constantly shifting sands, being by and large defined & differentiated by some fantastical gauntlet of automated V&V pipelines: big piles of test suites, fuzzers, prop tests, trace-based testing, etc. etc. More or less setting up a rigorous de facto compliance regime and making the arms race for enterprise distros be about who can establish the best regime and best gauntlet automation.
TheIronMark 3y ago
If you have to run an "Enterprise Linux", you are probably required by regulations or compliance to have a support agreement in place. Whether it's RedHat or Oracle or something else, security is going take third place to compliance and stability.
- secabeen 3y ago
  Not necessarily. RHEL-clones are ubiquitous in high-performance computing. Clusters and super-computers are hard to setup, consist of hundreds or thousands of nodes, and are used by scientists who are not programmers by trade. The RHEL-clone model works well for them, as they can't expect their users to keep code up to date with changes in the OS, they are used by known, mostly-trusted users, and the systems they run on usually stay fixed their 5-10 year lifespan. Having a single supported OS for the entire lifetime of the system saves a lot of work.
  - itomato 3y ago
    Scientific Linux? Package availability and familiarity get you a long way.
    - secabeen 3y ago
      Scientific Linux was an RHEL-clone, and they couldn't keep it going even with RH sources. > In April 2019, it was announced that feature development for Scientific Linux would be discontinued
- dralley 3y ago
  "regulations" or "compliance" aren't the only reasons to value stability. If you have a $2 billion facility or a $200 million piece of industrial equipment, you're going to place value on stability regardless of what the regulations say.
its-summertime 3y ago
What's missing is an example of a case where there is a potentially relevant exploit: Many times, even Debian has avoided being hit by a vulnerability, just due to a good wack of issues being in fresh code changes. And many times, even when the vulnerability has been a long existing one: RHEL is pretty focused on defense-in-depth, through compiler flag choices, SELinux setups, etc.
villgax 3y ago
The problem is that there is Enterprise Linux. You don't see a consumer windows, right?
- olddustytrail 3y ago
  What do you mean? Windows 11 is consumer windows. Windows Server 2022 is the enterprise version.
- happymellon 3y ago
  Err, you absolutely do see Windows Pro, Enterprise, Data centre. And there are bigger differences between the versions of Windows than Linux too.
- dylan604 3y ago
  You're right, we don't see consumer windows. We see Windows Home, Windows Pro, Windows Server, Windows^N-1
smarx007 3y ago
What is Oracle going to do with the userland/RPM compat with RHEL? It's wonderful that they try to ship every fix from the LTS kernel tree and that UEK users already don't have an expectation of a 100% identical kernel. But you still need userland compatibility, which will be much harder to ensure given the sheer number of RPM packages out there.
- freedomben 3y ago
  The Linux kernel has a pretty strict "don't break userland" policy, and if it does it's a bug, so I wouldn't expect using a newer kernel to be a problem.
  - smarx007 3y ago
    No, I meant to ask how Oracle is going to ensure that they deliver 10s of thousands RPM packages to be sufficiently compatible with those that RHEL ships (following the rhelgate or whatever we should call it).
  - AshamedCaptain 3y ago
    Ha. The contents of /sys for example change about every other release, and yes, there's obviously software which depends on it.
    - dezgeg 3y ago
      Could you point to some examples of such breakage?
      - freedomben 3y ago
        Here's an example where user space broke, and Linus gets pissed. "SHUT THE FUCK UP" to the maintainer who was defending the kernel change[1][2]. [1]: https://lkml.org/lkml/2012/12/23/75 https://lkml.org/lkml/2012/12/23/75 [2]: https://news.ycombinator.com/item?id=36805308 https://news.ycombinator.com/item?id=36805308
zokier 3y ago
> Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing. That is different kind of stability. No rolling release distro offers API (nevermind ABI) stability the way rhel etc do, which is the big selling point. You can install updates on enterprise distros without needing to worry too much about your application breaking because someone decided to have some fun with the API.
filereaper 3y ago
>This collection of software will remain locked at its specific version throughout the lifespan of that Enterprise Linux distribution release – which is often 10 years or more. Is this 10 year assumption true? Are people running 10 year old versions of operating systems today in non-safety or non-high security aspects? 10 years ago today is when the following were roughly released: - Linux 3.8. - RHEL 6.4 with Linux 2.6.32-358 - RHEL 7 was Beta with Linux 3.10.0 Are these still alive out there?
- retzkek 3y ago
  Absolutely. RHEL 7 (and downstreams) has maintenance support for another year, so there are many systems that administrators and now looking at upgrading. Speculatively, Red Hat timed the source availability change strategically to coincide with this epoch and force these administrators into a tight spot and hopefully get more subscriptions. Should these systems have been upgraded sooner? One year has usually allowed plenty of time for testing and rollout of the next major release.
  - tmottabr 3y ago
    And after that there is still Extended Support. RHEL 6 is still under extended support until next year.
- axus 3y ago
  They want to install security patches, and not change anything else. 8 years ago was Red Hat 6.7 and 7.0. Windows Server 2012 R2 and 8.1 were 10 years ago.
- dogline 3y ago
  Yes, these are very much alive deep in corporate data centers, and we've got to work with them.
- tmottabr 3y ago
  Yes it is.. I have customers that are still on RHEL 6 today and are only migrating to RHEL 7 because they were forced to..
MR4D 3y ago
I think there is something much more fundamental going on... 1. Maintenance is expensive. 2. Maintenance of living systems is even more so. 3. People hate spending money on maintenance. 4. You reap what you sow. We can't even get bridge or road maintenance done correctly in the US [0]. Why should we think that IT systems would be any different when we just don't have that culture? This is not a Linux problem - it's a culture problem (and probably goes well beyond the US). [0] - https://infrastructurereportcard.org/cat-item/bridges-infrastructure/ https://infrastructurereportcard.org/cat-item/bridges-infras...
- Aerbil313 3y ago
  Yeah. Us programmers' minds keep looking for a 'solution', but this is one of those solutionless paradoxes modern technology brings. If technology is developing and improving, it will need maintainance. If it's mature and stable, it ideally won't need maintaince or there will be enough knowledgeable workers who know how to do it, and their knowledge won't get obsolete in a short time. Unfortunately, and fortunately, technology is improving.
itomato 3y ago
Talk to the CFOs and CTOs. They fund the hardware refresh cycles and have bonuses attached to regular appeasement of shareholders, the market as well as auditors and controllers. The initiative begets the spend. The spend and roadmap stretches 10-12 years, chassis fans and PSUs have MTBF that conveniently coincide with Moore’s law and the dark arts of finance, where ownership is good but only to that boundary.
arc9693 3y ago
How is Kernel live patching as an approach for reaching the middle ground for enterprise applications? Amazon linux 2023 makes use of it. Note how “Ksplice was the first project for live patching the Linux kernel; however, ksplice was sold to Oracle and eventually changed to a closed-source tool.” https://www.redhat.com/en/topics/linux/what-is-linux-kernel-live-patching?sc_cid=7013a000003Sd9tAAC&gclid=Cj0KCQjwzdOlBhCNARIsAPMwjbxqW0JdshQr8K649JVE3Zkrc7LDJ9vbs4NRIpTwdBqIeHMYLhUzWUYaAgKOEALw_wcB https://www.redhat.com/en/topics/linux/what-is-linux-kernel-...
gjvc 3y ago
[flagged]
- cylo 3y ago
  Hi. Author here. I’m not a sockpuppet and genuinely have no affiliation with Oracle. It’s purely my own technical opinion. I only recently registered the site and set up the blog because I had nowhere else to post it and did not want to use a hosted platform like Medium. I was watching the recent Red Hat debacle from the sidelines and had some thoughts to put together about the general Enterprise Linux model.
  - ineedasername 3y ago
    Okay, that's fair, but do you kind of see how a shiny new domain name & blog with exactly one post and an anonymous author saying you can trust them because they're not affiliated with Oracle... Can you see how that appeal for trust hasn't been built on firm ground?
    - cylo 3y ago
      Sure, I suppose. I've never been called a sockpuppet before so I'm simultaneously insulted but also...honored in an odd way? Anyway, I'm not sure what else you'd have me do here. We all have to start somewhere. I put my thoughts out there under something I directly control and I'd rather people focus on the central thesis of the thoughts being proposed over resorting to personal attacks. I'll continue posting small essays on relevant topics under the domain.
      - ineedasername 3y ago
        I didn't mean to say you were wrong in what you chose to write, but also that it wasn't wrong for someone to be highly skeptical of its authenticity. Write what you want and people will take it as they will, I just wanted to point out that how the GP comment took it wasn't without some basis. As to the merits of your thesis, I'll outline my understanding of it & my thoughts in response: 1) You're proposing OEL as an alternative for RHEL (and other such offerings) 2) People might object to it based on Oracle's reputation (well deserved) 3) People should put that repulsion aside though because... 4) Oracle really is letting people use it for free so users don't have to enter into a $ relationship with this predatory company However: People choose RHEL (or any paid enterprise distro) primarily because of the support contracts that guarantee service. This undermines your appeal in #3 & #4 above because it requires the exact thing you're saying users can avoid to overcome their repulsion. Also, if enough people start using this distro then I have no doubt that Oracle will at some point switch things up and try to monetize that userbase under threat of crippling legal bills. I say all of this as someone who has actually been deposed by lawyers in a lawsuit with Oracle, where events leading up to the lawsuit were very much a bait-and-switch. I won't bog down this comment with the details but if you're interested in the exact nature of that bait & switch let me know and I'll comment in reply-- it may change your mind (or not) about trusting Oracle not to do something like that with OEL
lasermike026 3y ago
This whole model is in decline. Most companies have moved to the cloud and are using kubernetes. If you're running servers a datacenter then you might use enterprise linux. You are probably using some kind of virtualization like ESX or Nutanix. You might be using k8s there too. The age of running your own linux servers is ending in a way.
- 5e92cb50239222b 3y ago
  Most unicorn companies with an average half-life measured in months, maybe. Most companies worldwide across all industries, probably not. The pendulum will swing the other way sooner or later anyway as it always does.
- smarx007 3y ago
  Sorry, and where do you install k8s exactly?
- ineedasername 3y ago
  >You are probably using some kind of virtualization like ESX or Nutanix. Yes, and we spin up RHEL in them for some of our application servers.
- bandrami 3y ago
  I think this is HN bias. Companies whose purpose is something other than writing software are not doing this.
Kaytaro 3y ago
> Oracle has also publicly continued to commit to keeping “…the binaries and source code for that distribution publicly and freely available”. At this point their distribution has been around for 17 years, and they’ve never tried to restrict access to their source code in that time. I’ll be impressed when they open source Oracle DB and let Microsoft fork it and sell support for it. It’s pretty easy to commit to “open source” a copy of RHEL. Especially when linux isn’t core to their business but a bait and switch to get their customers on Oracle’s platform.
jacooper 3y ago
TBH the gist I get from this article is that the way the Linux project deals with security issues is a bit of a joke.
samstave 3y ago
>*What's wrong with enterprise Linux* For me was when I was trying to incorporate Redhat, Intel and Mirantis as a secure platform for on-prem-cloud as a service offering - and we were setting up the environment with RH, Intel and Mirantis (I was the Mirantis tech PM) - and it was a nightmare of IBM-esque beurocracy. it reminded me of the nightmares of installing equipment in an IBM/Intel DC in the 90s... Too fn many non-technical stakes in an undefined landscape and RH attempting to "feel enterprise" with Intel - and it was a disaster. (two FN months to get IP allocations????) Yeah - I signed off on redhat way before this - but this is what made me hate RHEL. They got too smarmy, just as LinuxCare did... But to answer the Q -- They attempted to get 'too enterprisey with it' and emulate those who they were previously trying to take down. (want some linuxcare stories)
opk 3y ago
We use Oracle Linux where I work because we used to use Solaris. At one point it looked suspiciously like Redhat had explicitly removed support for some Oracle hardware from their kernel and the UEK saved us. Besides the kernel they also offer a few other extras and optional newer versions like dtrace, support for some extra filesystems and newer KVM. I get the impression that the fact that they're underdogs in the Linux market helps to keep them honest. Much of their key staff are clearly open source advocates even if Larry only cares about money. And while the Oracle support is not up to the standards of the old Sun support it really isn't a bad choice. And it'd be easy to switch away if we ever need to.
- ineedasername 3y ago
  >Redhat had explicitly removed support for some Oracle hardware Sure, if you already had Oracle as a hardware vendor then the cat is out of the bag, barn doors open w/ horse gone. After that, using their distro isn't going to increase your attack surface much, the killer is already inside the house.
  - noizejoy 3y ago
    Off-topic, but I have to admit enjoying the liberally mixed metaphors of things leaving and entering barns, bags and buildings. And the music maker side of me thinks your post has the beginnings of a promising blues lyric. :-)
    - ineedasername 3y ago
      Thanks! I never saw a problem with mixing metaphors so long as it still conveyed your thoughts accurately. Language is fluid, let it flow.
ineedasername 3y ago
>Oracle Linux... set aside your automatic (and justified) repulsion I can't for two reasons: 1) The only way I avoid the repulsive side of Oracle in this is if I don't care about having a reliable guaranteed enterprise support contract. No one is going to have more expertise in how this distro is built & functions than Oracle, and that's the no-go zone. 2) It's not just about the current status of the project but also the future and what happens if this author is successful and lots of people switch over to Oracle Linux. Nothing stops them from switching things up at short notice (unless I pay them for a service guarantee!) and pulling the rug out from underneath the whole thing. And the more that people adopt it as their distro of choice, the more likely Oracle is to look around for some way to monetize it at gun-point (well, crippling legal bills at least)
- monkeywork 3y ago
  Lets not forget as well that as pointed out in this article OEL is essentially just a play at undercutting RHEL and is the main reason (imho) that RH/IBM took the steps they have over the last few months is because of OEL undercutting them (I think it had VERY VERY little to do with Rocky/Alma)
dangus 3y ago
Businesses only care about security fixes as far as compliance and a reasonable level risk. It’s more important to them type be able to tell auditors that they are following a mitigation process. In my case that means all I’m doing is pulling Amazon Linux images and running automatic security updates. Using Amazon Linux is the path of least resistance because it comes with AWS tooling and it’s already optimized for AWS. Everything else is basically “who cares?”
tkuraku 3y ago
I think that one of the biggest problems for my use case is license management for things outside the data center. With windows I can buy a pro license and have it attached to the hardware. I never have to worry about tracking or managing the license. With RHEL I have to make sure that the computer is registered and logged in to the license correctly. There is a lot of extra work for things like developer workstations or instrument controllers etc. If I could just buy a license of a major version of RHEL for $200-$500 and have it locked to the hardware like windows that would relieve a major administrative burden.
waweic 3y ago
As a former hobbyist that is now working a lot with RHEL as a sysadmin, I was really surprised to learn how little advantage there actually is with buying Red Hat Support. You are always limited to their opinionated decisions on what your deployment should look like* (or risk losing support), but at the same time, the support you actually get is next to none. If I can't fix it myself, it ain't gonna be fixed. At this point, I don't know what we are paying for anyways. *As an example, more recent versions of RHEL only allow the use of NetworkManager for permanent network configuration. In a production hypervisor system, NM is completely unsuitable in my opinion. It's full of footguns and that will bite us at some point
- frankreyes 3y ago
  > At this point, I don't know what we are paying for anyways. You get to shift your blame on someone else. It's a commercial "covering your behind as a service", that you just blame $vendor when things go bad. It's the game big corps play, sometimes called "compliance".
rwmj 3y ago
Erm, enterprise Linux systems backport major new features all the time. The version number of some software means very little. This article's whole premise is flawed.
l0b0 3y ago
Reminds me of CERN Linux, which was downstream of RHEL when I was there. It took several years to certify all the CERN software on a new release, which surely must've cost the org many more years of cumulative developer productivity.
cbmuser 3y ago
»Defenders will counter that this is a necessary tradeoff – you can’t have both stability and being fully up-to-date on security fixes. I’m not convinced that’s true as there are other Linux distributions that have shown you can have both. Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing. Additionally, distributions like Fedora Linux, while not technically a rolling release, do tend to hew close to upstream versions at a more regular cadence.« I'm sorry, but this paragraph proves that the author has not fully understood the purpose of enterprise distributions. The primary selling point is not the stability of the software, but the stability of the API and ABI. There is closed source software such as SAP that is compiled against the ABI of RHEL and SLES. And lots of expensive enterprise hardware ships Linux kernel drivers in binary form only, so that a stable, i.e. never-changing kernel ABI is required. Examples for such hardware is the SGI UV series whose drivers come in binary form only and therefore the hardware is supported on enterprise distributions only. Both RedHat and SUSE put a lot of effort into keeping their API and ABI stable, so that enterprise customers are guaranteed that no distribution update is going to break the software that they're running on the hardware they're using. Also, when you deploy Linux on hundreds or thousands of clients, a rolling release distribution would be a pure nightmare. Given the large amount of clients and applications, there will always be a combination of applications and use-cases that will break after a distribution package was updated to a completely new upstream version. Companies don't want software that changes all the time since they want to control the time for an update rollout themselves.
- StillBored 3y ago
  Right, and part two, is that in the case of RHEL they backport features required to run on newer hardware, or enable things required for newer software stacks (crypto protocols for example). It just depends on whether its possible to maintain API/ABI stability and support the newer feature. Vs, the other LTS distros that just track the upstream LTS kernel. That kernel both breaks things, as well as is "old" because its only generally taking security and bug fixes.
- dralley 3y ago
  Also when it comes to things like medical imaging, changes to the graphics stack (which includes userspace drivers) can and have altered how (e.g.) an MRI renders. So you can't go around making major updates to that stack every other week. It has to be certified and then re-certified after major updates and the vendor might need to be involved in that process. And you can't just skip it and "hope" that the MRI results are rendering properly. Expand that out to all the other similar kinds of applications.
rlpb 3y ago
The article claims that since security fixes land in latest releases first but must be individually backported to older releases, therefore staying on an older release is less secure. However I think this is in itself a false premise. The obvious counterargument is that it is newer releases in which new security vulnerabilities are introduced, so after a while, an older release with backported security fixes is more secure. Which argument is true depends on the relative rates of fixes vs. vulnerability introductions. This will differ per project, and therefore the balance may tip both ways too, depending. But the article fails to make any assessment of this at all.
frankreyes 3y ago
Enterprise stuff is decided by directors over fancy dinners, charity events, and business class plane flights by bureaucrats. Enterprise Linux needs to learn how to play the lobbying game This is sometimes called "compliance". At a previous job we had a bunch of servers that were reaching End of Life and the tech staff already was planning on upgrading, but someone in the bureaucracy with veto and signature powers decided to extend their life to an extended warranty. We had to keep running those slow and power hungry things for a few more years because we run out of budget.
hulitu 3y ago
> By locking packages to specific versions and only backporting select fixes, these distributions lag significantly behind upstream versions when it comes to security bugfixes, albeit unintentionally. And the "upstream versions" are how better from a security POV when they are mostly an incompatible redesign ? Or the OP means that GTK4 has fewer CVEs than GTK3 because it is newer and better ? A new version of a library (or a program) does not mean a more secure version. It is mostly "new features" and the bugs are left there to rot.
cryptos 3y ago
Doesn't Debian Stable have the very same problem? You get dated software with bugs that are solved in newer versions. The fundamental model of freezing and (endlessly) testing some state of the packages that makes up a distribution simply doesn't make sense in a world with so many and so big changes. This model is the "waterfall" model for operating systems.
LeoNatan25 3y ago
“The engineers, maintainers, and testers […] put in the enormous amount of thankless work” No, they actually get paid pretty well to do it. And pretty sure all the paying enterprise customers are “thankful” for the distribution’s stability and security, or they’d take their business elsewhere.
sharts 3y ago
Just move to BSD.

204 points by cylo 3y ago

> The Case for Oracle Linux I did not see that twist coming. Strangely, the author makes a good point. Oracle or not, sticking closer to the upstream kernel is not a bad way to manage an “enterprise” kernel. Maybe we’ve all been too willing to sit back and accept how RHEL does kernels as “the way”.

gjsman-1000 3y ago
I actually did really like the point; but I would be very interested in hearing a rebuttal or counterargument from Red Hat. Linus' "security problems are just bugs" approach has been worrisome.
- smarx007 3y ago
  I would expect that tracking the Linux kernel master just 4 weeks behind is going to cause more cases where, combined with enterprise software of typical quality, you'd need vendor support than with the RH way of changing as little as possible for as long as possible. And support is exactly what Oracle sells. To be clear, I only used Oracle Linux briefly while playing with the free tier of Oracle Cloud. Edit: sorry, but I don't get it, please break it down for me. Latest UEK 7 is based off 5.15 LTS ( https://docs.oracle.com/en/operating-systems/uek/ https://docs.oracle.com/en/operating-systems/uek/ ), that puts them at branching off a kernel released on 2021-10-31 (according to https://en.wikipedia.org/wiki/Linux_kernel_version_history#Releases_5.x.y https://en.wikipedia.org/wiki/Linux_kernel_version_history#R... ). How is that even remotely 4 weeks behind master?! Edit 2: my bad, I didn't read the "behind the tip of the LTS tree" part carefully. So, the difference is that Oracle tries to ship all LTS kernel updates, while RH tries to cherry-pick critical security and bug fixes only?
  - bonzini 3y ago
    Red Hat doesn't change as little as possible, at least 30% of the changes to Linux make it to RHEL. Most of the RHEL kernel is a few months behind upstream despite the old version. The next minor release of RHEL, to be released in November, probably will have features up to 6.3 for many subsystems and bugfixes up to 6.5 for example.
    - smarx007 3y ago
      Interesting, thank you for the details. This means that in some way, RHEL is ahead of Oracle Linux in terms of kernels? I don't see an UEK version tracking Linux 6.1 at all, AFAIK.
      - cylo 3y ago
        It's true that Red Hat does pull in changes from upstream for several subsystems of Linux. It's genuinely a frankenkernel mixed with code from 6.3, 6.1, etc. But you're still beholden to what the Red Hat maintainers are pulling in and focusing on. It's still not a general follow upstream wholesale. You can see and track what UEK is doing by looking at their Github: https://github.com/oracle/linux-uek/commits/uek7/u1 https://github.com/oracle/linux-uek/commits/uek7/u1
        deleted 3y ago
        [deleted]
      - bonzini 3y ago
        See here for an article that describes how it's done --> https://news.ycombinator.com/item?id=36763935 https://news.ycombinator.com/item?id=36763935
- chabad360 3y ago
  In his defence, most bugs in the kernel can be exploited, so it doesn't necessarily make sense to treat a "bug" with a PoC better than one without.
- picozeta 3y ago
  > Linus' "security problems are just bugs" approach has been worrisome. Aren't they?
indymike 3y ago
> Oracle or not, sticking closer to the upstream kernel is not a bad way to manage an “enterprise” kernel. This is probably a lot better than expecting it to be possible to maintain a secure release with a level of compatibility guarantee for 10+ years.
- deleted 3y ago
  [deleted]
- dralley 3y ago
  Maybe, but it's not obvious. There are plenty of novel vulnerabilities in interfaces like io_uring and so forth. It's not that those features are bad, I'm just saying that there are tradeoffs to always getting the new stuff. Maybe the compromise solution is to use newer kernels but keep certain features turned off until they "bake" properly.
weare138 3y ago
But what stops Oracle from just pulling a Red Hat when people switch over to Oracle Linux? Oracle hasn't exactly proven itself to be trustworthy over the years and that's putting it lightly. I think we need a fully open source alternative to RHEL not bound to any company. Something akin to the Debian project that can serve as an upstream reference distro and repository.
- MatthiasPortzel 3y ago
  Oracle Linux, Red Hat, and Debian are all “fully open source”. What you mean is “not maintained by a corporation.”
  - marcosdumay 3y ago
    Well, RHEL is now only "fully open source" with quotes, and won't allow code redistribution anymore.
- tosihakkeri 3y ago
  > I think we need a fully open source alternative to RHEL not bound to any company I believe the problem is not that there wouldn’t be open source alternatives but that that’s not what enterprise wants. Enterprise wants a company behind the distribution.
  - noizejoy 3y ago
    But arguably, there's also a pretty large world that wants enterprise type of solutions (for some use cases) without being actual enterprises. Or am I the only one?
  - weare138 3y ago
    Of course, think about the relationship between Ubuntu and Debian. The idea is the upstream would be a fully open-source project that vendors build off of downstream. This way we can prevent vendors from subjugating a distribution like with Red Hat and CentOS and if a vendor pulls out of the market or goes out of business it doesn't take down the ecosystem with it.
zokier 3y ago
At the same time it is not that convincing point because in lot of cases the value from freezing is pretty small for kernel compared to other software, because kernels strong "do not break userspace" attitude. Version freezing is far more valuable for various other projects that do not take backwards compatibility that seriously.
- mbreese 3y ago
  I thought the Linux kernel had a strong “don’t break userspace” attitude, but it was a free for all in kernel space. If you’re developing kernel modules or have custom hardware with drivers, I could see having a backported kernel as being a major problem for support and development.
  - pxc 3y ago
    My (spectator's) understanding is that ABI compatibility is not a priority for kernelspace in Linux, but there is care taken for backwards compatibility in other ways (like at an API level). You're just expected to recompile your out-of-tree kernel modules when there's a new kernel version.
    - comex 3y ago
      Nope, API compatibility is also explicitly disavowed: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/stable-api-nonsense.rst https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
      - pxc 3y ago
        Thank you for sharing this! Nothing helps you remember a fact like having been wrong about it in public. :)
vbezhenar 3y ago
If you're using proprietary hardware drivers which target RHEL, you need stable kernel. Also kernel updates breaking drivers is a thing. So there's definitely value with frozen kernel version. And its almost the same value as frozen software. It just works and it won't break after update.
deleted 3y ago
[deleted]

CrampusDestrus 3y ago

Holy.... that Oracle linux page is so friendly it almost makes me puke. I can't believe anyone would fall for them

gjsman-1000 3y ago
The same company that just two weeks ago was sending out unsolicited emails asking companies about Java licensing - a friendly shakedown for using the official JRE... https://news.ycombinator.com/item?id=36599118 https://news.ycombinator.com/item?id=36599118 Frankly, I don't get it Oracle. I would say the priorities are backwards - JRE free, Oracle Linux paid would make more objective sense (especially because the JRE used to be free... and there's a lot of free JREs out there.) Also, don't be stupid enough to download the VirtualBox Extensions package at work; or your work will probably get an email from a licensing agent.
- azlev 3y ago
  It's an huge company with mixed incentives. Not everyone is align whatever their alignment is.
  - CrampusDestrus 3y ago
    >Yes, we know that this is Oracle, but... There might be different alignments, but you simply cannot write stuff like this and get away with it if any of your superiors disagree. That page literally admits Oracle is an asshole company
- Tainnor 3y ago
  > Also, don't be stupid enough to download the VirtualBox Extensions package at work; or your work will probably get an email from a licensing agent. Place I used to work blocked the virtualbox home page because exactly this had happened.
- count 3y ago
  I don't think there is any cross P&L strategy at play....

darkhelmet 3y ago

There is another problem that wasn't covered in the article. The 10+ years of stability leads to behaviors and outcomes that remind me of the long-lived SSL certificate problem. Updating is done so infrequently that the "how?" is forgotten. As the 10 year support limit approaches, most of the old team members who did it last time are gone, tech debt is through the roof, few people know where everything is or how to build it, and so on. Enterprise Linux "stability" enables all sorts of bad behavior if your company is inclined that way. LetsEncrypt did us a huge favor by forcing automation vs having the guy who knows how to update the SSL certs every 4.9 years and left 6 months ago. I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it. (I know, most HN folk are well disciplined but there are a lot of corporate cultures that are not.)

derekp7 3y ago
Don't you end up with the same problem with the automation that has been running fine for 5 years, then suddenly breaks? And the person that set it up is either gone, or has no clue how they did it 5 years ago.
- bonzini 3y ago
  The idea is that you deploy from scratch all the infrastructure every 6 months, first to testing and then to production.
  - retbull 3y ago
    Every 6 months? That seems like a pretty long window for tribal knowledge to get lost. Is 6 months arbitrary or is there some reasoning behind that cadence?
    - noizejoy 3y ago
      Arguably, tribal knowledge and the dependence on it need to be managed as much as anything else.

azlev 3y ago

What's not clear to me is that newer software is safer or just less tested. Although it's possible that newer versions are more secure, I couldn't find evidence of it yet. (Links are welcome).

dboreham 3y ago
Probably more of an ass-covering situation: bugs are known in older software and not doing something about a known risk is obviously bad. otoh not mitigating an unknown and unidentified risk (bugs in new software) is ok.
wredue 3y ago
See. I used to think similarly. Newer software must be more secure, because we definitely know about all these problems and how to deal with them. What that doesn’t consider is that the vast majority of programmers simply don’t care to keep up with security concerns. And that’s why two of the most dangerous, easy to exploit, easy to stop attacks out there (SQL injection, XSS) are still among the most prevalent. As long as features are growing, the claims that new is more secure is difficult to believe.
dylan604 3y ago
that's why the whole rewrite in $languageX is so annoyingly frustrating when touted as fixing so many issues just because it uses $languageX. How many bugs not present in original version get introduce is always swept under the rug of the $languageX evangelists
Pet_Ant 3y ago
Isn't an unknown vulnerability preferrable to a known vulnerabilty?
- jossclimb 3y ago
  There is unknown and unreported, a large amount of unreported vulns are sold on the blackmarket or hoarded for exploit by foreign adversaries.
- 3y ago

im_down_w_otp 3y ago

I could imagine an updated take on “Enterprise Distro”, in order to take advantage of the constantly shifting sands, being by and large defined & differentiated by some fantastical gauntlet of automated V&V pipelines: big piles of test suites, fuzzers, prop tests, trace-based testing, etc. etc. More or less setting up a rigorous de facto compliance regime and making the arms race for enterprise distros be about who can establish the best regime and best gauntlet automation.

TheIronMark 3y ago

If you have to run an "Enterprise Linux", you are probably required by regulations or compliance to have a support agreement in place. Whether it's RedHat or Oracle or something else, security is going take third place to compliance and stability.

secabeen 3y ago
Not necessarily. RHEL-clones are ubiquitous in high-performance computing. Clusters and super-computers are hard to setup, consist of hundreds or thousands of nodes, and are used by scientists who are not programmers by trade. The RHEL-clone model works well for them, as they can't expect their users to keep code up to date with changes in the OS, they are used by known, mostly-trusted users, and the systems they run on usually stay fixed their 5-10 year lifespan. Having a single supported OS for the entire lifetime of the system saves a lot of work.
- itomato 3y ago
  Scientific Linux? Package availability and familiarity get you a long way.
  - secabeen 3y ago
    Scientific Linux was an RHEL-clone, and they couldn't keep it going even with RH sources. > In April 2019, it was announced that feature development for Scientific Linux would be discontinued
dralley 3y ago
"regulations" or "compliance" aren't the only reasons to value stability. If you have a $2 billion facility or a $200 million piece of industrial equipment, you're going to place value on stability regardless of what the regulations say.

its-summertime 3y ago

What's missing is an example of a case where there is a potentially relevant exploit: Many times, even Debian has avoided being hit by a vulnerability, just due to a good wack of issues being in fresh code changes. And many times, even when the vulnerability has been a long existing one: RHEL is pretty focused on defense-in-depth, through compiler flag choices, SELinux setups, etc.

villgax 3y ago

The problem is that there is Enterprise Linux. You don't see a consumer windows, right?

olddustytrail 3y ago
What do you mean? Windows 11 is consumer windows. Windows Server 2022 is the enterprise version.
happymellon 3y ago
Err, you absolutely do see Windows Pro, Enterprise, Data centre. And there are bigger differences between the versions of Windows than Linux too.
dylan604 3y ago
You're right, we don't see consumer windows. We see Windows Home, Windows Pro, Windows Server, Windows^N-1

smarx007 3y ago

What is Oracle going to do with the userland/RPM compat with RHEL? It's wonderful that they try to ship every fix from the LTS kernel tree and that UEK users already don't have an expectation of a 100% identical kernel. But you still need userland compatibility, which will be much harder to ensure given the sheer number of RPM packages out there.

freedomben 3y ago
The Linux kernel has a pretty strict "don't break userland" policy, and if it does it's a bug, so I wouldn't expect using a newer kernel to be a problem.
- smarx007 3y ago
  No, I meant to ask how Oracle is going to ensure that they deliver 10s of thousands RPM packages to be sufficiently compatible with those that RHEL ships (following the rhelgate or whatever we should call it).
- AshamedCaptain 3y ago
  Ha. The contents of /sys for example change about every other release, and yes, there's obviously software which depends on it.
  - dezgeg 3y ago
    Could you point to some examples of such breakage?
    - freedomben 3y ago
      Here's an example where user space broke, and Linus gets pissed. "SHUT THE FUCK UP" to the maintainer who was defending the kernel change[1][2]. [1]: https://lkml.org/lkml/2012/12/23/75 https://lkml.org/lkml/2012/12/23/75 [2]: https://news.ycombinator.com/item?id=36805308 https://news.ycombinator.com/item?id=36805308

zokier 3y ago

> Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing. That is different kind of stability. No rolling release distro offers API (nevermind ABI) stability the way rhel etc do, which is the big selling point. You can install updates on enterprise distros without needing to worry too much about your application breaking because someone decided to have some fun with the API.

filereaper 3y ago

>This collection of software will remain locked at its specific version throughout the lifespan of that Enterprise Linux distribution release – which is often 10 years or more. Is this 10 year assumption true? Are people running 10 year old versions of operating systems today in non-safety or non-high security aspects? 10 years ago today is when the following were roughly released: - Linux 3.8. - RHEL 6.4 with Linux 2.6.32-358 - RHEL 7 was Beta with Linux 3.10.0 Are these still alive out there?

retzkek 3y ago
Absolutely. RHEL 7 (and downstreams) has maintenance support for another year, so there are many systems that administrators and now looking at upgrading. Speculatively, Red Hat timed the source availability change strategically to coincide with this epoch and force these administrators into a tight spot and hopefully get more subscriptions. Should these systems have been upgraded sooner? One year has usually allowed plenty of time for testing and rollout of the next major release.
- tmottabr 3y ago
  And after that there is still Extended Support. RHEL 6 is still under extended support until next year.
axus 3y ago
They want to install security patches, and not change anything else. 8 years ago was Red Hat 6.7 and 7.0. Windows Server 2012 R2 and 8.1 were 10 years ago.
dogline 3y ago
Yes, these are very much alive deep in corporate data centers, and we've got to work with them.
tmottabr 3y ago
Yes it is.. I have customers that are still on RHEL 6 today and are only migrating to RHEL 7 because they were forced to..

MR4D 3y ago

I think there is something much more fundamental going on... 1. Maintenance is expensive. 2. Maintenance of living systems is even more so. 3. People hate spending money on maintenance. 4. You reap what you sow. We can't even get bridge or road maintenance done correctly in the US [0]. Why should we think that IT systems would be any different when we just don't have that culture? This is not a Linux problem - it's a culture problem (and probably goes well beyond the US). [0] - https://infrastructurereportcard.org/cat-item/bridges-infrastructure/ https://infrastructurereportcard.org/cat-item/bridges-infras...

Aerbil313 3y ago
Yeah. Us programmers' minds keep looking for a 'solution', but this is one of those solutionless paradoxes modern technology brings. If technology is developing and improving, it will need maintainance. If it's mature and stable, it ideally won't need maintaince or there will be enough knowledgeable workers who know how to do it, and their knowledge won't get obsolete in a short time. Unfortunately, and fortunately, technology is improving.

itomato 3y ago

Talk to the CFOs and CTOs. They fund the hardware refresh cycles and have bonuses attached to regular appeasement of shareholders, the market as well as auditors and controllers. The initiative begets the spend. The spend and roadmap stretches 10-12 years, chassis fans and PSUs have MTBF that conveniently coincide with Moore’s law and the dark arts of finance, where ownership is good but only to that boundary.

arc9693 3y ago

How is Kernel live patching as an approach for reaching the middle ground for enterprise applications? Amazon linux 2023 makes use of it. Note how “Ksplice was the first project for live patching the Linux kernel; however, ksplice was sold to Oracle and eventually changed to a closed-source tool.” https://www.redhat.com/en/topics/linux/what-is-linux-kernel-live-patching?sc_cid=7013a000003Sd9tAAC&gclid=Cj0KCQjwzdOlBhCNARIsAPMwjbxqW0JdshQr8K649JVE3Zkrc7LDJ9vbs4NRIpTwdBqIeHMYLhUzWUYaAgKOEALw_wcB https://www.redhat.com/en/topics/linux/what-is-linux-kernel-...

gjvc 3y ago

[flagged]

cylo 3y ago
Hi. Author here. I’m not a sockpuppet and genuinely have no affiliation with Oracle. It’s purely my own technical opinion. I only recently registered the site and set up the blog because I had nowhere else to post it and did not want to use a hosted platform like Medium. I was watching the recent Red Hat debacle from the sidelines and had some thoughts to put together about the general Enterprise Linux model.
- ineedasername 3y ago
  Okay, that's fair, but do you kind of see how a shiny new domain name & blog with exactly one post and an anonymous author saying you can trust them because they're not affiliated with Oracle... Can you see how that appeal for trust hasn't been built on firm ground?
  - cylo 3y ago
    Sure, I suppose. I've never been called a sockpuppet before so I'm simultaneously insulted but also...honored in an odd way? Anyway, I'm not sure what else you'd have me do here. We all have to start somewhere. I put my thoughts out there under something I directly control and I'd rather people focus on the central thesis of the thoughts being proposed over resorting to personal attacks. I'll continue posting small essays on relevant topics under the domain.
    - ineedasername 3y ago
      I didn't mean to say you were wrong in what you chose to write, but also that it wasn't wrong for someone to be highly skeptical of its authenticity. Write what you want and people will take it as they will, I just wanted to point out that how the GP comment took it wasn't without some basis. As to the merits of your thesis, I'll outline my understanding of it & my thoughts in response: 1) You're proposing OEL as an alternative for RHEL (and other such offerings) 2) People might object to it based on Oracle's reputation (well deserved) 3) People should put that repulsion aside though because... 4) Oracle really is letting people use it for free so users don't have to enter into a $ relationship with this predatory company However: People choose RHEL (or any paid enterprise distro) primarily because of the support contracts that guarantee service. This undermines your appeal in #3 & #4 above because it requires the exact thing you're saying users can avoid to overcome their repulsion. Also, if enough people start using this distro then I have no doubt that Oracle will at some point switch things up and try to monetize that userbase under threat of crippling legal bills. I say all of this as someone who has actually been deposed by lawyers in a lawsuit with Oracle, where events leading up to the lawsuit were very much a bait-and-switch. I won't bog down this comment with the details but if you're interested in the exact nature of that bait & switch let me know and I'll comment in reply-- it may change your mind (or not) about trusting Oracle not to do something like that with OEL

lasermike026 3y ago

This whole model is in decline. Most companies have moved to the cloud and are using kubernetes. If you're running servers a datacenter then you might use enterprise linux. You are probably using some kind of virtualization like ESX or Nutanix. You might be using k8s there too. The age of running your own linux servers is ending in a way.

5e92cb50239222b 3y ago
Most unicorn companies with an average half-life measured in months, maybe. Most companies worldwide across all industries, probably not. The pendulum will swing the other way sooner or later anyway as it always does.
smarx007 3y ago
Sorry, and where do you install k8s exactly?
ineedasername 3y ago
>You are probably using some kind of virtualization like ESX or Nutanix. Yes, and we spin up RHEL in them for some of our application servers.
bandrami 3y ago
I think this is HN bias. Companies whose purpose is something other than writing software are not doing this.

Kaytaro 3y ago

> Oracle has also publicly continued to commit to keeping “…the binaries and source code for that distribution publicly and freely available”. At this point their distribution has been around for 17 years, and they’ve never tried to restrict access to their source code in that time. I’ll be impressed when they open source Oracle DB and let Microsoft fork it and sell support for it. It’s pretty easy to commit to “open source” a copy of RHEL. Especially when linux isn’t core to their business but a bait and switch to get their customers on Oracle’s platform.

jacooper 3y ago

TBH the gist I get from this article is that the way the Linux project deals with security issues is a bit of a joke.

samstave 3y ago

>*What's wrong with enterprise Linux* For me was when I was trying to incorporate Redhat, Intel and Mirantis as a secure platform for on-prem-cloud as a service offering - and we were setting up the environment with RH, Intel and Mirantis (I was the Mirantis tech PM) - and it was a nightmare of IBM-esque beurocracy. it reminded me of the nightmares of installing equipment in an IBM/Intel DC in the 90s... Too fn many non-technical stakes in an undefined landscape and RH attempting to "feel enterprise" with Intel - and it was a disaster. (two FN months to get IP allocations????) Yeah - I signed off on redhat way before this - but this is what made me hate RHEL. They got too smarmy, just as LinuxCare did... But to answer the Q -- They attempted to get 'too enterprisey with it' and emulate those who they were previously trying to take down. (want some linuxcare stories)

opk 3y ago

We use Oracle Linux where I work because we used to use Solaris. At one point it looked suspiciously like Redhat had explicitly removed support for some Oracle hardware from their kernel and the UEK saved us. Besides the kernel they also offer a few other extras and optional newer versions like dtrace, support for some extra filesystems and newer KVM. I get the impression that the fact that they're underdogs in the Linux market helps to keep them honest. Much of their key staff are clearly open source advocates even if Larry only cares about money. And while the Oracle support is not up to the standards of the old Sun support it really isn't a bad choice. And it'd be easy to switch away if we ever need to.

ineedasername 3y ago
>Redhat had explicitly removed support for some Oracle hardware Sure, if you already had Oracle as a hardware vendor then the cat is out of the bag, barn doors open w/ horse gone. After that, using their distro isn't going to increase your attack surface much, the killer is already inside the house.
- noizejoy 3y ago
  Off-topic, but I have to admit enjoying the liberally mixed metaphors of things leaving and entering barns, bags and buildings. And the music maker side of me thinks your post has the beginnings of a promising blues lyric. :-)
  - ineedasername 3y ago
    Thanks! I never saw a problem with mixing metaphors so long as it still conveyed your thoughts accurately. Language is fluid, let it flow.

ineedasername 3y ago

>Oracle Linux... set aside your automatic (and justified) repulsion I can't for two reasons: 1) The only way I avoid the repulsive side of Oracle in this is if I don't care about having a reliable guaranteed enterprise support contract. No one is going to have more expertise in how this distro is built & functions than Oracle, and that's the no-go zone. 2) It's not just about the current status of the project but also the future and what happens if this author is successful and lots of people switch over to Oracle Linux. Nothing stops them from switching things up at short notice (unless I pay them for a service guarantee!) and pulling the rug out from underneath the whole thing. And the more that people adopt it as their distro of choice, the more likely Oracle is to look around for some way to monetize it at gun-point (well, crippling legal bills at least)

monkeywork 3y ago
Lets not forget as well that as pointed out in this article OEL is essentially just a play at undercutting RHEL and is the main reason (imho) that RH/IBM took the steps they have over the last few months is because of OEL undercutting them (I think it had VERY VERY little to do with Rocky/Alma)

dangus 3y ago

Businesses only care about security fixes as far as compliance and a reasonable level risk. It’s more important to them type be able to tell auditors that they are following a mitigation process. In my case that means all I’m doing is pulling Amazon Linux images and running automatic security updates. Using Amazon Linux is the path of least resistance because it comes with AWS tooling and it’s already optimized for AWS. Everything else is basically “who cares?”

tkuraku 3y ago

I think that one of the biggest problems for my use case is license management for things outside the data center. With windows I can buy a pro license and have it attached to the hardware. I never have to worry about tracking or managing the license. With RHEL I have to make sure that the computer is registered and logged in to the license correctly. There is a lot of extra work for things like developer workstations or instrument controllers etc. If I could just buy a license of a major version of RHEL for $200-$500 and have it locked to the hardware like windows that would relieve a major administrative burden.

waweic 3y ago

As a former hobbyist that is now working a lot with RHEL as a sysadmin, I was really surprised to learn how little advantage there actually is with buying Red Hat Support. You are always limited to their opinionated decisions on what your deployment should look like* (or risk losing support), but at the same time, the support you actually get is next to none. If I can't fix it myself, it ain't gonna be fixed. At this point, I don't know what we are paying for anyways. *As an example, more recent versions of RHEL only allow the use of NetworkManager for permanent network configuration. In a production hypervisor system, NM is completely unsuitable in my opinion. It's full of footguns and that will bite us at some point

frankreyes 3y ago
> At this point, I don't know what we are paying for anyways. You get to shift your blame on someone else. It's a commercial "covering your behind as a service", that you just blame $vendor when things go bad. It's the game big corps play, sometimes called "compliance".

rwmj 3y ago

Erm, enterprise Linux systems backport major new features all the time. The version number of some software means very little. This article's whole premise is flawed.

l0b0 3y ago

Reminds me of CERN Linux, which was downstream of RHEL when I was there. It took several years to certify all the CERN software on a new release, which surely must've cost the org many more years of cumulative developer productivity.

cbmuser 3y ago

»Defenders will counter that this is a necessary tradeoff – you can’t have both stability and being fully up-to-date on security fixes. I’m not convinced that’s true as there are other Linux distributions that have shown you can have both. Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing. Additionally, distributions like Fedora Linux, while not technically a rolling release, do tend to hew close to upstream versions at a more regular cadence.« I'm sorry, but this paragraph proves that the author has not fully understood the purpose of enterprise distributions. The primary selling point is not the stability of the software, but the stability of the API and ABI. There is closed source software such as SAP that is compiled against the ABI of RHEL and SLES. And lots of expensive enterprise hardware ships Linux kernel drivers in binary form only, so that a stable, i.e. never-changing kernel ABI is required. Examples for such hardware is the SGI UV series whose drivers come in binary form only and therefore the hardware is supported on enterprise distributions only. Both RedHat and SUSE put a lot of effort into keeping their API and ABI stable, so that enterprise customers are guaranteed that no distribution update is going to break the software that they're running on the hardware they're using. Also, when you deploy Linux on hundreds or thousands of clients, a rolling release distribution would be a pure nightmare. Given the large amount of clients and applications, there will always be a combination of applications and use-cases that will break after a distribution package was updated to a completely new upstream version. Companies don't want software that changes all the time since they want to control the time for an update rollout themselves.

StillBored 3y ago
Right, and part two, is that in the case of RHEL they backport features required to run on newer hardware, or enable things required for newer software stacks (crypto protocols for example). It just depends on whether its possible to maintain API/ABI stability and support the newer feature. Vs, the other LTS distros that just track the upstream LTS kernel. That kernel both breaks things, as well as is "old" because its only generally taking security and bug fixes.
dralley 3y ago
Also when it comes to things like medical imaging, changes to the graphics stack (which includes userspace drivers) can and have altered how (e.g.) an MRI renders. So you can't go around making major updates to that stack every other week. It has to be certified and then re-certified after major updates and the vendor might need to be involved in that process. And you can't just skip it and "hope" that the MRI results are rendering properly. Expand that out to all the other similar kinds of applications.

rlpb 3y ago

The article claims that since security fixes land in latest releases first but must be individually backported to older releases, therefore staying on an older release is less secure. However I think this is in itself a false premise. The obvious counterargument is that it is newer releases in which new security vulnerabilities are introduced, so after a while, an older release with backported security fixes is more secure. Which argument is true depends on the relative rates of fixes vs. vulnerability introductions. This will differ per project, and therefore the balance may tip both ways too, depending. But the article fails to make any assessment of this at all.

frankreyes 3y ago

Enterprise stuff is decided by directors over fancy dinners, charity events, and business class plane flights by bureaucrats. Enterprise Linux needs to learn how to play the lobbying game This is sometimes called "compliance". At a previous job we had a bunch of servers that were reaching End of Life and the tech staff already was planning on upgrading, but someone in the bureaucracy with veto and signature powers decided to extend their life to an extended warranty. We had to keep running those slow and power hungry things for a few more years because we run out of budget.

hulitu 3y ago

> By locking packages to specific versions and only backporting select fixes, these distributions lag significantly behind upstream versions when it comes to security bugfixes, albeit unintentionally. And the "upstream versions" are how better from a security POV when they are mostly an incompatible redesign ? Or the OP means that GTK4 has fewer CVEs than GTK3 because it is newer and better ? A new version of a library (or a program) does not mean a more secure version. It is mostly "new features" and the bugs are left there to rot.

cryptos 3y ago

Doesn't Debian Stable have the very same problem? You get dated software with bugs that are solved in newer versions. The fundamental model of freezing and (endlessly) testing some state of the packages that makes up a distribution simply doesn't make sense in a world with so many and so big changes. This model is the "waterfall" model for operating systems.

LeoNatan25 3y ago

“The engineers, maintainers, and testers […] put in the enormous amount of thankless work” No, they actually get paid pretty well to do it. And pretty sure all the paying enterprise customers are “thankful” for the distribution’s stability and security, or they’d take their business elsewhere.

sharts 3y ago

Just move to BSD.