12 ms·
Disclaimer: I'm a former Red Hatter but my opinion is my own Red Hat, you NEED to fix this. I get that it sucks to have your business built on selling suppor
by freedomben 3y ago
Disclaimer: I'm a former Red Hatter but my opinion is my own
Red Hat, you NEED to fix this. I get that it sucks to have your business built on selling support, and a couple of companies have popped up selling cheaper support but not taking on the cost of building the thing. I get that (you think) it's going to cause problems for you (it might, I'm not sure yet). But you're torching an insane amount of good will that was built up over 20 years of good and mostly selfless deeds. You are making yourselves an enemy and a villain, rather than a friend. You need to stop the bleeding.
I don't have any answers, but I have some ideas:
1. Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support. If those don't agree, other projects will emerge who will agree and will take their place. Don't make their lives unnecessarily hard by throwing up roadblocks in their way. There will probably be popup companies selling third party support for Alma/Rocky etc here and there, but I doubt that type of support competition makes a serious dent to enterprise sales. In fact, I think it helps you by increasing adoption. A rising tide raises all boats, especially in this field.
2. Figure out a way to let individuals and small developers use real RHEL without having to dick with subscriptions or licensing or activation. Be liberal with this and make it easy
- acomjean 3y agoalmost 20 years ago when I took a Red Hat Kernel internals class (at Redhat) they basically told us to use CentOS if we wanted to experiment with RHEL. (We were preparing for a transition from the HPUX/Solaris machines.). But it worked, some of our architects did that and we could get the scheduler behaving similar to HPUX. I left that company, but I'm pretty sure they did switch and RedHat got a big contract out of it. We've come a long way.
- neilv 3y ago> 1. Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support. I don't know how the support business breaks down for this, but for the large customers, I can't imagine any of the little rebranding projects taking the support business that the IT dept. itself calls. "Nobody ever got fired for buying IBM," and in this case it's literally true again: RH still have some of the best technical experts. (Imagine being a huge company with a urgent mission-critical kernel problem: you want to be able to get a Red Hat commando team parachuting in, including a Sr. Principal Engineer who specializes in hard debugging tasks in that vast code base. Not have Bob's Discount Distro's support line be lost after you say you already tried rebooting and reinstalling your global operations.) At the same time, if anyone distro is rebranding RHEL not just as an open source hippie volunteer thing (that a lot of not-ready-to-pay-for-RHEL smaller businesses use), but also trying to make it a support business out of it, then that might be poking the lion. The hippie commune might get ripped apart in a rampage. Which might be what's happening now? (Credible support options from AWS and maybe Oracle are a different matter.) > 2. Figure out a way to let individuals and small developers use real RHEL without having to dick with subscriptions or licensing or activation. Be liberal with this and make it easy Maybe CentOS had been providing this market segmentation with upgrade path to enterprise support, until recently? RHEL is not easy to switch to, and I've had a client that was very capable shop look at switching to RHEL, and nope right back, because they didn't want to learn the new bureaucracy. If they'd been able to use RHEL for free since they were smaller, becoming an RH enterprise support customer as they grew would be easy and a no-brainer. (Docker and then K8s have shaken this up a bit.)
- Sylamore 3y ago>I don't know how the support business breaks down for this, but for the large customers, I can't imagine any of the little rebranding projects taking the support business that the IT dept. itself calls. "Nobody ever got fired for buying IBM," and in this case it's literally true again: RH still have some of the best technical experts. I support a very large telco company, and they've had a massive effort underway to convert all of their RHEL systems to Rocky (preferred) or Ubuntu because of the loss of CentOS as something they could use on non-production systems plus their (the telcos) financial issues making them look for any software licenses they can abandon. They've replaced thousands of RHEL boxes with Rocky boxes already, all public cloud images must use Rocky or Ubuntu, no RHEL allowed. The use of RHEL is limited to critical systems like those running an Oracle DB only (and even those they are investigation replacements for).
- biccboii 3y agowhy even go through all of this when debian is sitting right there, free and reliable.
- hintymad 3y agoNot sure if Red Hat will be able to fix this. This is not Red Hat but IBM. If IBM could fix problems like this, they wouldn't have been declining in the past decade.
- johnea 3y agoIt's not because they're not dumb, it's because they're IBM...
- bamfly 3y agoI think this is a key part of where they've been aiming for years. Since the '00s, even. Maybe earlier, IDK. I just have their public behavior to go on, but it sure looks like a long-term strategy. If it's not, it ought to be, because they're doing most of the work for it anyway. 1) Steer enough of the Linux ecosystem that Red Hat is the most-qualified support source for common Linux stacks, including maneuvering to make it difficult to decouple or avoid them in other distros (Systemd, Gnome, and Wayland are key plays here). 2) Cut off access to the blessed least-risky-for-enterprise version of your product. 3) You are Linux for much of the market. You own it. Profit.
- tester756 3y ago>You are Linux for much of the market. You own it. Profit. What a twist! So after all it wasn't Microsoft EEEing Linux, but actual "Linux Companies"?!
- anyoneamous 3y agoMicrosoft tried and failed because their product is strictly inferior for the market which Linux has ended up dominating. Marketing and evil behaviour can definitely win out over technical quality, but we were fortunate that it didn't go that way in this case.
- progmetaldev 3y agoI'm not sure if that argument holds water. You have Azure which is hugely successful, and Linux is still not anywhere close to becoming a standard desktop environment beyond the type of people you find on HN. Microsoft contributes quite a bit to Linux now, because they see the future. Becoming a leader in open source gives you the ability to steer where technology goes, and Red Hat definitely was able to play that hand as well (but under IBM seems to be losing quickly).
- anyoneamous 3y agoI'm not sure that any of your post actually contradicts mine; MS have accepted the dominance of Linux in the server market and have moved on to exploiting it where they can. That doesn't mean they wouldn't have rather had everyone running on Windows, if they could have pulled it off.
- pmontra 3y agoMaybe a solution for point 2 could be that the first five licenses or activations are free.
- freedomben 3y agoThat would help, but I think that only works optimally if they also do number 1. If there is any "activation" process or the like, it adds enough friction to cause problems, especially in a modern CI/CD environment. It's not an easy change by any stretch, but if they don't do something they're going to be the next Oracle.
- hinkley 3y agoI worked for a company that did the voting for American Idol for the first few seasons. Weird culture, a mix of bare metal and NIH that I thought didn’t quite function and some of us were trying to change that culture. Everyone resented working on that project and wanted to do other things, so at some point we disinvited ourselves (or so the story went, might have been sour grapes). The problem was we killed the cash cow without any viable livestock lined up to replace it, and the layoffs started about a year later. Of course 4/5ths of the people trying to change the culture were in the first round. Get rid of the rabble rousers first. While I think our proposed changes would have helped dramatically, I also think it might have been better if the company management had invested in making it more palatable to work on the project. Bribery if you have to. Conferences, toys… but they didn’t do either, and it flopped. Edit: on rereading your comments I may have missed your angle, but I’ll leave this anyway, since we are talking dumb failure modes.
- gymbeaux 3y agoThat I can’t run sudo yum update out of the box with RHEL is one of the big reasons I don’t bother with it. I love the “stability aspect” of RHEL but I love the I guess “convenience” of any other distro being ready to go out of the box, no activation required
- freedomben 3y agoI totally agree. Even when I worked at Red Hat and could get whatever licenses I needed, I still used CentOS for local stuff and for testing things because of the friction of the registration/activation process. It has gotten easier in recent years, but simply existing is enough to turn off a lot of people.
- pipo234 3y agoAlma, Rocky, fair enough. But isn't the elephant in the room here Oracle Linux?!
- freedomben 3y agoYes, very fair point. Oracle and Amazon both deserve to be on the list, probably much further up than Alma or Rocky
- pipo234 3y agoAmazon actually seems to have seen this coming. While Amazon Linux 1&2 was a basterdized CentOS6& derivative (with some newer packages), the latest breed (Amazon Linux 2022/2023) is based off Fedora. Also, I'm not convinced Amazon (or Oracle for that matter) have a very bright future. The opaque development process and (lack of) community support make development pretty hard. I know from experience, getting bugs resolved (or just getting Amazon to care) is very hard. At the same time, even for a large corporate maintaining your own distro brings with it considerable costs. Wouldn't it be great if instead they would work together, share some of the costs and profits of support and cloud mutually?!
- ilc 3y agoAmazon missed then, the company to bet on is SuSE here. Both Oracle and Amazon Linux have VERY bright futures for their use-cases. Do you run Oracle? Want to know your distro will be supported 100%. Oracle Linux seems like a damn good bet to use. Are you one of the largest companies on earth, who needs a distro to run your business? If so... guess what.. your distro is viable just on your own business. Amazon offering Amazon Linux is just them showing some internal tech. Nothing more.
- eddythompson80 3y agoAmazon (and Microsoft) offering a Linux distribution is about (a) raining in the explosion of distros their various teams depend on that they don't themselves control, and (b) answering insistent cloud customers requests for "What distro do you recommend? Do you support {distro}? Do you auto patch {distro}? How come update to {distro} broke your service/product? Do you not work with {distro}? Last year many services on Azure went down because they all used Ubuntu and Ubuntu pushed an update that broke for a DNS configuration that Microsoft uses. Having your own distro won't guard you from fuck-ups of course, but you can imagine the difficult conversation Microsoft had to have with their biggest customers about how Canonical can take down your 40 billion dollar business globally?
- rwmj 3y ago> Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support. That was CentOS (after acquisition by Red Hat but before CentOS Stream). Unfortunately killed off.
- freedomben 3y agoAgreed, although there is one important difference: Red Hat isn't behind Alma or Rocky like they were CentOS (yes they did that to themselves by acquiring CentOS, but that was in the past. It was a mistake, but we have to move forward from where we are today). A lot of people knew that CentOS was owned by Red Hat and that made them a lot more comfortable using it. They kind of had to change that relationship somehow. I'm not happy at all that they did it, but I can understand why. Option 1 above allows them to keep that "those are third parties, not Red Hat" relationship but still have the benefits of a free community build.
- m463 3y agoInternational Business Machines¹ should merge with the International Brotherhood of Magicians² to get a better chance of pulling this off. [1] https://en.wikipedia.org/wiki/IBM https://en.wikipedia.org/wiki/IBM [2] https://en.wikipedia.org/wiki/International_Brotherhood_of_Magicians https://en.wikipedia.org/wiki/International_Brotherhood_of_M...
- that_guy_iain 3y ago> Red Hat, you NEED to fix this. No, they don't. They don't need to do anything. > But you're torching an insane amount of good will that was built up over 20 years of good and mostly selfless deeds. Doesn't matter. That goodwill doesn't really benefit them. How many companies paid Red Hat because they liked the things they did for open-source? Realistically, very few if any. > Continue to make it so that rebuilders like Alma or Rocky can exist as bug-for-bug by having access to the source, but reach an agreement with them not to offer or sell support. Realistically, there are lots of companies that would pay for Red Hat but don't because they get can get it for free.
- redundantly 3y ago> Realistically, there are lots of companies that would pay for Red Hat but don't because they get can get it for free. If the free RHEL variants disappeared today I'd wager most of those companies would choose other distributions (Debian, Ubuntu, openSUSE, etc) or making the jump to containerisation or serverless before paying for RHEL. In the end less people would be using their kind of Enterprise Linux, which in turn will result in less conversion to the paid version of it.
- zb3 3y ago> In the end less people would be using their kind of Enterprise Linux, which in turn will result in less conversion to the paid version of it. Why would anyone want to convert to the paid version if their use case is covered by the free version? Out of generosity?
- yjftsjthsd-h 3y agoSometimes support - for the OS or from applications - is actually useful. I worked at a shop once that ran CentOS for most things but RHEL for the database servers, because the vendor only officially supported that.
- redundantly 3y ago
- janosdebugs 3y agoAlso former Red Hatter, disclaimer also applies. According to my observations, Red Hat does an unbelievable amount of stuff by hand. Lots of internal processes require someone to do something at a given point, otherwise these distros are not happening. I believe RH is now cutting costs at precisely these places: projects that don't directly contribute to the revenue get cut. However, as the post said, these changes make it incredibly hard to ship open source software for RHEL-likes to the point where I already decided to look for alternatives when doing integration work.
- avhception 3y agoI would like to add the perspective of a small-business infrastructure guy here. First things first: we're a Rocky Linux shop. About 100 instances, some bare metal, some VMs. A few local, a few at cloud providers like DigitalOcean. Held together by Ansible and kickstart installs. Too often the discussion about RHEL clones like Rocky paints a picture of people like me as freeloaders. And some of the folks with a similar setup might be. But most small-time Linux admins I know are real open-source believers. I would really like to funnel some money into the platform we're using. First and foremost because it's the right thing to do, the people writing all that source code deserve it. But also because we need platform continuity, and being able to open a support case from time to time would also be great. But it's really hard. I don't have the time and nerves to deal with the RHEL sales department, deal with the licensing, produce my own cloud images because DigitalOcean can't provide them, deal with our own internal purchasing process, deal with convincing my superiors that we need to. Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.
- _ea1k 3y agoHonestly, there are a ton of people like you out there. A big selling point for Red Hat has always been their contributions to the community and that funding them would help to drive that. It is a big motivator for people transitioning from your state to becoming a customer.
- Twirrim 3y ago> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out. Most of the major clouds have this pre-existing capability to charge more and revenue share the extra through their marketplace services etc; and they're in active use by other vendors. It's often a good source of revenue for both the cloud company and the vendor, and helps the customers by providing them with prebuild solutions that they don't have to build themselves (e.g. web application firewall vendors can/do provide closed source images that customers can use that way). It's not like this is expecting cloud companies to build whole new services from whole cloth just to support Red Hat.
- kbenson 3y agoI had a meeting with our RH reps last week, as a followup to Red Hat Summit and to talk about a potential purchase with them. I had a discussion about how we use free RHEL derivatives, and have for almost 30 years (when Red Hat itself was free), and we might have used one of the first public releases (I do know we were running 5.2 in '99). In explaining our position I told them we use the free products (which meant CentOS until stream, and now Rocky), I noted how I'm sure they have their own reasons for what they did, but the Stream change made us seriously consider other distros like Ubuntu, and while we settled on Rocky in the end after it came out, if we had chosen Ubuntu we wouldn't be having a discussion about possibly spending $100k with them now and more in the future, so that's how they're hurting themselves. Then again, from attending Red Hat Summit, it's apparent that they're really aiming at the enterprise and government anyway. I would estimate attendance was 60% government and education (including a lot of government contractors), 30% enterprise (Adobe, etc), and 10% small to medium business (and I'd bet many of them are partners offering services and want to support Red Hat). To me it looks like people making short sighted moved to increase their hold on their enterprise customers (this might be aimed more at Oracle Linux than Rocky or Alma), but at the expense of the surrounding ecosystem. The knock-on effects might be that they kill the ramp to RHEL and the training of admins in its use except for those companies willing to actually shell out for official training programs, as there will be little to no regular use by non-enterprise customers anymore if they keep this up.
- thesuperbigfrog 3y ago>> there will be little to no regular use by non-enterprise customers anymore if they keep this up. They are slowing transforming RHEL to become the new AIX (https://en.wikipedia.org/wiki/IBM_AIX https://en.wikipedia.org/wiki/IBM_AIX).
- leokennis 3y agoWhat is that line? “No one ever got fired for buying IBM”? I think the main appeal for enterprises is that if their VM’s start going down and thousands flush through the drain every second, someone at Red Hat is there to pick up the phone and hop on a call with their engineers. It makes them and their regulators happy and no other Linux distro offers that. I don’t think they care about open source at all.
- redundantly 3y ago> Red Hat, you NEED to fix this You should be addressing IBM. This is how they operate. Things will only get more strict and more difficult to work with from here. They don't care about good will. They don't care about OSS. The only thing that matters is dominance in the market place to take care of their bottom line.
- freedomben 3y agoA lot of Red Hat people pay attention to forums like this, and I would bet money there's a big discussion going on internally right now. IBM couldn't care less, but the people at Red Hat will. They're also not powerless (at least, they weren't when I left a few years ago).
- Yeroc 3y agoThere were similar discussions when the CentOS Stream thing was announced and Red Hat didn't change course so I'd guess they're not going to back off this change either.
- freedomben 3y ago> There were similar discussions when the CentOS Stream thing was announced and Red Hat didn't change course so I'd guess they're not going to back off this change either. True, and it's likely nothing will change this time too. But if there's no outrage at all and nobody tries, we guarantee nothing changes and probably invite further encroachment.
- jzb 3y agoObviously IBM owns Red Hat now and all that. But people overestimate how much involvement IBM has day-to-day with Red Hat. (Source: I'm a former Red Hatter, left last year.) At worst I'd imagine that IBM's influence is felt this way: IBM expects x% growth and other metrics every quarter. How that is achieved is up to Red Hat's executives. If the leadership at Red Hat decides it can do that with the status quo, then I don't think IBM is going to say "no, you must change the source distribution to fit our evil plan." The worst thing that IBM has done directly (IMO) was plucking Jim Whitehurst away from Red Hat and over to IBM, but not making him CEO. Whatever reason IBM chose to do that, it wasn't good for Red Hat or IBM long-term in my opinion.