RHEL and Centos
February 28, 2018

So, You Think RHEL and CentOS Aren’t Cutting Edge Distros?

Operating Systems

Recently, we received feedback regarding one of our Twitter posts:

Why do IT leaders love CentOS

The response to this tweet stated that CentOS does not provide the “latest stable” releases of the published packages.

You’re not wrong

Since CentOS receives the package source from Red Hat’s Enterprise Linux, CentOS is typically bound to the package versions that Red Hat releases. Red Hat Enterprise Linux (RHEL) doesn’t upgrade the included packages to their latest stable releases very often; usually only when publishing a new major version release (RHEL 6 to RHEL 7, for instance). This is especially true if the package uses Semantic Versioning and the latest stable release has a different major version number than the currently included package. (Read more about Semantic Versioning to better understand why)

The fact that the package updates aren’t performed frequently builds uniformity and compatibility between minor version releases of both RHEL and CentOS, making them an excellent Linux distribution choice for third-party application vendors to support.

Red Hat published an article which further addresses when and why packages are updated - included in that article is the following statement:

One of the core goals of the Red Hat Enterprise Linux family of products is to provide a stable, consistent runtime environment for third-party applications. To support this goal, Red Hat seeks to preserve application binary compatibility, configuration file compatibility, and data file compatibility for all package updates issued within a major release.

It's likely that Red Hat’s dedication to providing a platform with a “…stable, consistent runtime environment…” is why so many third-party application vendors chose to support their products on RHEL and CentOS. In my experience, most third-party vendors offering distribution-specific support specify RHEL/CentOS as their preferred OS. Sure, some vendors may offer support for their products running on other Linux distributions but there are also vendors that don’t support any non-RHEL/CentOS deployments.

What about the waning security of these outdated packages?

Security is of obvious concern to anyone responsible for production environments. I can almost hear the speculative cries right now:

If RHEL/CentOS are not in the habit of updating to newer versions of packages between major releases, then I must not be getting security updates as they’re patched and released by the package maintainers!

Luckily, that is not the case! RHEL and CentOS backport security updates to the published package versions, so you're not trading security for stability.

Red Hat publishes security guides for administrators, and the following is from the RHEL 7 Security Guide:

As security vulnerabilities are discovered, the affected software must be updated in order to limit any potential security risks. If the software is a part of a package within a Red Hat Enterprise Linux distribution that is currently supported, Red Hat is committed to releasing updated packages that fix the vulnerabilities as soon as possible. Often, announcements about a given security exploit are accompanied with a patch (or source code) that fixes the problem. This patch is then applied to the Red Hat Enterprise Linux package and tested and released as an erratum update. However, if an announcement does not include a patch, Red Hat developers first work with the maintainer of the software to fix the problem. Once the problem is fixed, the package is tested and released as an erratum update.

In my article on RHEL to CentOS migration, I mention:

For the security-minded, when Red Hat publishes security updates, CentOS quickly turns those updates around and presents them to the CentOS community, usually in about 24 hours or less.

This is good news, indeed! RHEL and CentOS offer long-term stability, consistency, interoperability, and security!

But I want my systems to run on the cutting edge!

For those who prefer things on the cutting (or bleeding) edge, while wanting to stay somewhat close to the RHEL/CentOS world, Fedora may interest you. Fedora is often used to test out new packages, new features, and package updates before they are included in RHEL/CentOS. The Fedora Project does attempt to keep updates reasonable, though. They only use stable versions of packages and won’t use package revisions that are still in beta.

But, I don’t recommend running Fedora in production without first performing some serious internal testing *and* locking down automated updates. Such changes could break your finely-tuned environment, rendering your mission-critical applications unstable, unusable, or offline entirely.

Now, if you think about it, disabling automatic non-security updates to help ensure the long-term stability of your system is precisely what RHEL and CentOS are doing by not updating packages mid-major release!

There are other popular distributions that release updates WAY more often

So, you think that RHEL and CentOS are overprotective parents that shelter their kids (packages) and scoff at the other distros that update their packages with “…the frequency of a cheap ham radio”? ( Apologies to Dan Aykroyd)

Not quite…

Debian’s “stable” releases occur approximately every 2 years, and you’d choose the stable release “…If security or stability are at all important for you.” Debian also offers two other options: an “unstable” branch, which “…has the most recent (latest) versions. But the packages in unstable are not well tested and might have “bugs” and a “testing” branch, which fall between the older, reliable “stable” branch and the newest, untested “unstable” branch.

openSUSE also puts out major releases about every 2 years. Analogous to Fedora and Debian testing/unstable, there is a more current branch of openSUSE called “Tumbleweed” for those who prefer a distribution with a “rolling release” nature of constant updates and the potential stability and compatibility problems that come with them.

Even Ubuntu, which has a relationship with Debian that could be compared to that of CentOS and Red Hat, respectively, and publishes new releases every 6 months, offers Long Term Support (LTS) releases every 2 years. The Ubuntu wiki page for LTS releases includes statements like, “significantly limiting the number of new features”, “enterprise-focused,” and “more tested” when describing the LTS releases.

These sure sounds a lot like the stable release paradigm that RHEL/CentOS employs.

Of course, there are other distributions which have varying degrees of stability and freshness, but these are some of the more common ones that many folks will be familiar with, or at least aware of.

It’s your system…if you don’t like it, change it

I know that it can be frustrating if you’re looking for a newer version of a package than those shipped with RHEL or CentOS, but this frustration should be balanced with the knowledge that the OS will remain quite stable and maintain interoperability for a long time compared to distributions which update their packages relatively often.

That being said, let’s keep in mind that one of the great features of open source operating systems is that you have the freedom to modify the OS to fit your own needs! Sysadmins may choose to recompile the kernel and packages to fit their own version, feature, or security needs.

For instance, let’s say that RHEL/CentOS ships with a version of a fictitious “libFancyWidget*” library that is older than you’d like, and/or your application requires the “libFancyWidget” release that was posted last week by the package maintainers. It is usually quite easy to download the source RPM and modify it to use a different version of the upstream package.

What if RHEL/CentOS doesn’t ship with “libFancyWidget” at all? It may be available from a 3rd party repo like EPEL or RPM Fusion. There’s also a chance that Fedora offers the package (or perhaps Fedora has released the version that you need). If so, you could install the RPM from the Fedora or third-party repo. If that fails, you could see if a source RPM is available, grab that source RPM and compile a binary RPM specifically for your RHEL/CentOS system.

What if none of the above options work for you? You could obtain the “libFancyWidget” source and compile it as a standalone library…or go the extra mile and create a source RPM for the library and contribute it back to the open source community so that others can benefit from your work!

One consideration is that doing these things may negate some of the security and stability that Red Hat and the CentOS community invest in their releases during their QA phases.

*[The libFancyWidget library in this article is fictitious. No identification with actual libraries (current or EoL), packages, systems, and widgets is intended or should be inferred.]

I chose the distro less tested…

Did you decide to utilize a distro which offers bleeding edge updates for the published packages?

How confident are you (or not) after performing your internal testing? You did perform your own comprehensive testing, didn’t you?! Are you confident that the thoroughness of your testing meets or exceeds that of the entire team of Red Hat and CentOS release engineers? What if something unexpected goes wrong after passing all of your testing successfully?

These up-to-the-minute distributions aren’t usually known for their end-user support options!

At best, you’ll find that you need to upgrade or downgrade some package or other, and maybe not one of the ones you modified.

At worst, you’ll be ignored or told to fix it yourself and submit a patch or PR (Pull Request).

Somewhere in the middle, expect your bug report to be accepted and that someone will work on it (sometime), find the problem (eventually), post a patch (later), the patch will get merged into the next release (whenever that’ll be), and a new package may be built that solves your problem…usually sometime in the next year or so (if you’re lucky).

Of course, you could get really lucky and the package maintainers will recognize the seriousness of your bug and fix it right away and request your feedback. That’s a special type of lucky, right there, and you should immediately buy a lottery ticket!

I chose the distro more tested…

Even if you’ve determined that customizing an existing distribution is the path for your project, by starting with the solid foundation that is RHEL or CentOS, you can minimize the scope of the internal QA testing (both functional and regression) that will need to be invested before putting confidence into your custom mission critical solution.

Running CentOS? You can also get support for the portions of the OS that you haven’t modified through Rogue Wave Open Source Support. You may even be able to work with us on a professional services engagement for the packages that you have modified.