Thoughts on RHEL Rebuilds
As a longtime RHEL/RHEL-like administrator at work and home, I have a few things to say about Red Hat’s latest rugpull.
Background
I have administered Red Hat/RHEL/RHEL-likes at every IT job I’ve had (and some Fedora as well, back when it first came out). Red Hat has been a standard of business/enterprise Linux since I started as a young vulpine sysadmin and I’ve built my career on it. Generally, the smaller businesses that I’ve worked for used RHEL-likes while the larger ones used RHEL as they can afford the support contracts, which, theoretically, is the value that Red Hat provides. Upper management is willing to pay for that “peace of mind” (though, in my experience, RHEL support has been useful exactly once in 20+ years).
In my current position, RHEL is the rule, though I used CentOS 8 in non-production/POC environments (before it got rugpulled on December 8, 2020). It was still somewhat viable until Alma and Rocky were ready. Both of these distros have been very useful to me on the job and at home, running most of my (admittedly small) set of servers.
Then, everything changed when the Fire Nation attacked Red Hat performed its latest rugpull.
The New Normal
This was a subtle change, only noticed a few days after Red Hat stopped publishing srpm/RHEL source code updates to git.centos.org. They then made it official in this blog post then kept digging themselves deeper. The latter post effectively says “Hey, Oracle, Alma, Rocky..you’re all a bunch of freeloaders and the party is over; you don’t contribute anything, you just steal our code (that we mostly pulled right out of open source projects). Everyone else, pay only us for RHEL and pay only us for support. All of you users are just freeloaders as well since you never contribute anything, and you need to pay up, now!”
Alma issued a pair of responses, refuting Red Hat’s “freeloader” claim. Rocky issued their rebuttal, along with a pair of posts further clarifying their stance and way forward. Oracle..had nothing to say. They seem likely to continue offering their RHEL rebuild and selling support for it. They could argue that they’ve contributed the Unbreakable Enterprise Kernel to the RHEL ecosystem, but I take their silence as evidence that they know about it but don’t care a bit.
(EDIT: On July 10, Oracle did have something to say..and it was very direct in its criticism of IBM (to the extent of stating that they would be willing to take over the development of RHEL if IBM wanted to become a downstream distributor of Oracle Linux. The audacity of that statement is breathtaking.) The post also promised to keep Oracle Linux source and binaries open and available and encouraged downstream community and commercial distros. (An invitation to Alma and Rocky, perhaps..)
I’ve been around long enough to not give Oracle the benefit of the doubt on anything..this is the same company that could have used a commercial and community model with Solaris and OpenSolaris, but they killed the latter off as soon as they could. I also wonder if this could bring about a schism between RHEL and Oracle Linux, where Oracle figures they can make a better RHEL than Red Hat can, hoping they can persuade Red Hat’s big customers that their Linux will be more cost effective for their workloads and hardware as Red Hat tightens their grip on RHEL.
It will be very interesting to see how this all plays out.)
(I have used the UEK in my professional work and I haven’t seen any drastic gains or losses from it, though it seems to work better with Oracle Database, though that might just be psychosomatic.)
Since then, there has been a lot of digital ink spilled about this drastic change, both for and against it (the former mostly from shills) along with a lot of uncertainty about what this means for companies that have based their infrastructure on RHEL-like distros. Do they keep the status quo and hope that Alma or Rocky (or other, lesser-known alternatives) can keep the updates and new major and minor releases coming in a reasonable amount of time? Do they switch to Oracle, reasoning that they’ll keep the distro current regardless of what Red Hat does? Who’s to say that Oracle won’t do a similar rugpull, considering their history? Can they afford to pay for RHEL either short or long-term, knowing that they could increase the pricing at any time, especially if they feel like their customers are locked in?
These are all difficult questions for a small IT department to figure out, especially if migrating to either Ubuntu, Debian, or SuSE isn’t a viable path. It would make sense for Red Hat to offer a lower/cheaper support tier for such companies, or, perhaps, a self-supported tier where the company pays a flat annual cost per 100/500/1,000 RHEL servers/virtual machines to get access to Red Hat’s distros, updates, and Satellite for system management but nothing else. If they could get past their short-sightedness and animosity towards the RHEL-likes, there is a path to giving a small company peace of mind with their Linux infrastructure while making it easy to move up to higher support tiers as the company grows.
The Path Ahead For Linux Servers At Home
The hobbyist/home server admin currently has the option to run up to 16 RHEL instances for free..but considering RHEL’s propensity for rugpulling, that can’t be relied upon, in my opinion. The RHEL-likes have always been great for running a stable, solid OS on one’s home servers with little fuss in obtaining them or oversight on how they’re used. They still are, as of this writing. (I was able to pull Alma updates for my 8.x server today, for instance). I have read speculation about using Debian at home (or in a small workplace) instead, as that is not subject to any kind of corporate shenanigans, and paid support is available from consultants or a company such as Debian Support.
My younger self was rather close-minded when it came to Debian, unfortunately, though I remember running sid as my workstation for a time very early in my career before replacing it with Red Hat Linux. I recently made a list of the daemons/other programs that are on my servers and was able to find either corresponding packages in Debian stable (some of which aren’t in RHEL or EPEL) or external Debian repos. Failing those, compiling it separately and using /usr/local is always an option.
As soon as I get a chance, I am going to migrate my Arch Linux ARM Raspberry Pi 4 instance to Debian ARM64 (not to be confused with the Debian-based Raspberry Pi OS, formerly known as Raspbian). Arch ARM (which isn’t an official port), runs what I want it to well enough, but has had bouts of instability along with several extended periods of no updates whatsoever, which defeats the purpose of running a rolling release. I’ll see how Debian stable fares in that department, but everything I’ve heard over the years indicates that’ll be solid, predictable, and boring..which, after getting a server set up properly, is a good thing. I’m not sure if updates and migrations will be the same, but I’ll find out..
Whether I’ll do the same for my other servers remains to be seen..they’re doing fine on RHEL-likes at the moment. There’s no guarantee that the RHEL-likes are going to be workable long-term now, though. Red Hat could very well escalate matters and plug whatever loopholes are being used, creating an ongoing cat-and-mouse game. If that doesn’t work, they can threaten legal action against the Alma Linux Foundation and the Rocky Enterprise Software Foundation. (Whether or not they’d engage in a heavyweight legal bout against Oracle…who knows).
Conclusion
This wasn’t something that Red Hat had to do, but they did it anyway. I don’t see them having a sudden change of heart in the near or far future, and their capriciousness seems likely to continue. Whether Red Hat can be trusted with any aspect of RHEL going forward also remains to be seen; they need to remember that they’re not the only game in town when it comes to running corporate Linux. (Canonical, for instance, hasn’t said anything publicly that I could find, but I wouldn’t be surprised if they’re using this as a new angle to persuade companies to migrate to Ubuntu Server.)
This doesn’t affect my work all that much, nor my home systems, but it adds an unnecessary layer of risk with the RHEL ecosystem in general and appears to be driving individuals and organizations to strongly consider alternatives to run their workloads. “Short-term gain, long-term pain” is, unfortunately, the modern way of corporations, and Red Hat is stumbling their way down that path like so many others, seemingly indifferent to the damage they’re doing to their reputation.
(All art on this page by Jasmine at myfurryart.com)