All they need to do, SUSE’s GM of the Business Critical Linux
team, Rick Spencer, said, is simply change their CentOS 7
update repository to SUSE’s, avoiding any disruptive migrations
or upgrades.
Oracle also was the first to release updates last time I compared the three of them side by side, which was about a year ago. I installed Rocky, Alma and Oracle 9 on three VMs, pointed them as close to upstream repositories as possible (to avoid mirroring delays), and just started them every day to see who was the first to update. I also checked RPM metadata which includes build time. The experiment ran for (IIRC) about two months.
Oracle was consistently the first, lagging behind RHEL for a few hours (for important stuff) to a couple of days (for less important ones). Alma was a very close second. Rocky would spend days to weeks and was by far the slowest.
No, I don't buy the argument that it's good for you because they receive additional testing. If you really need that much stability that you can't take an update after RHEL has done so, introduce your own delays and test your shit on staging. I'd like my upstream to be as quick as practically possible. Oracle and Alma cover that nicely.
If it were any other company, I might agree with you. But I just don't think I'd ever touch anything Oracle makes with a 10' pole except under duress. It's better to do the work up front of just going with the alternative than risk getting pulled into that particular tar baby.
That'll only buy you several months. IIRC OL7 EOL issue in December, unless you pay for extended support while you do your work to transition to a newer OS (please, please don't rely on extended support, unless desperate, switch to new versions.)
True but gives you some additional time, and yes don't rely on ext. support, neither from RH nor Oracle. It just gets harder and harder to upgrade, as i wrote further down my preferred upgrade time-frame is 3-5 years max.
Is Oracle committed to actually supporting the distro with security updates and so on? I was under the impression that it's simply a bug-for-bug compatibility rebuild from RHEL sources, which aside from rebuilding effort, the majority of support work was still done by RH.
It's a rebuild with a few more patches on top (like ksplice). If you need a RHEL rebuild, they're the first to release security updates of any of the rebuilds I've tested. See my other comment for more info.
It is and it isn't. They have a huge commercial interest in not unnecessarily sharing money with Red Hat for customers that basically need a supported/certified platform primarily for their Oracle products. SUSE has had fairly recent management changes (including ex-Red Hatters coming in). Red Hat should probably never have acquihired the CentOS team in the first place--not a comment on the people but bringing CentOS in-house with all that implied. And the others are fairly small.
For a public forum, let's just say there was a long history of expedient decisions which I had no hand in but many of us regret :-/ Understandable but there were almost certainly better paths.
Transitioning from CentOS 7 to RedHat 8 and 9 at my former company's private cloud has smooth for most teams, pareto-style, with 80% of migration-related incidents caused by the 20% of the teams that did some really weird changes to the VM's OS that was no longer allowed under RHEL 8 or 9.
At first, I thought it was just to reduce the complexity of managing hardening rules for several OS and OS versions.
Recently we had issues with RH 9 not having header packages for openssl 1.1 (which means e.g. you can't have erlang < 25). There's a potential for breakage for anything that is 3+ years old.
Not sure if that applies to your line of work but RHEL has been an increasingly annoying source of issues in low-latency space. Their kernels are a bizarre mixture of cherry picked stuff from upstream and a completely bonkers project structure.
Also, they break ABI across minor kernel versions which is unthinkable in mainline. What I'm trying to say is: if you're doing more funky stuff with your OS, just go with a distro built on top of mainline.
Scientific clusters have more options though, as Rocky/Alma and other alternatives to RHEL are available without dealing with "this distro does not ship FIPS-140-3 certified crypto" or similar.
Yes, that's true. However, tons of software was written solely for CentOS, so ironing out the small kinks will take some time.
The problem is, scientific software can silently error out in many ways (like slightly lost accuracy), and detecting and fixing those will take some time.
Considering some of these tools are developed over (almost two) decades, there's a lot of verification we have to go through.
Considering how many interesting issues that specific CentOS7-locked job had just with updating kernel to latest LTS (which required a lot of fun stuff as well, given that it didn't compile with included GCC)...
Rocky and Alma are okay-ish for scientific clusters. They work and drivers appear to mostly cooperate. Some changes, like switching to sssd, were suboptimal but to be expected.
CentOS just had a lot more testing being done by vendors. We still regularly hit problems with, e.g., Lenovo LESI being not aligned with our Rocky systems for many parameters.
Anecdotally, I am hearing scientific computing customers who are unhappy after moving to Rocky - the window where each minor OS version is "supported" is too small and their hardware vendors have trouble staying current. Have seen quite a few move to Ubuntu, and the rest switch to regular RHEL.
If you were using centos for fedramp, unless you got a variance from the feds, you were not in compliance. No one actually paid to have NIST evaluate centos's binaries (evaluation/certification must be of the compiled binary, not the source code, barring an exception made for openssl, and only openssl, not the kernel)
Talking firsthand, There is significant effort into ensuring that both 8.6.z (through to current) and 9.2.z (through to current) are fedramp compliant, this requirement eats my day, every day.
8.6 and 9.2 kernels are released more frequently than other streams, if you want more frequent updates for compliance reasons, these are the streams to use.
There was a time when RHEL FIPS certificate also applied to CentOS, and one of my former $DAYJOBs depended on that for a long time. Then Red Hat pulled the cert, and later there was a mad scramble to get rid of CentOS because of the EOL :)
Just a heads up, since I couldn't comment on the LWP article about this a few days ago.
The leapp-upgrade program had a reproducible bug as late as last week where after every upgrade it inserted net.ifnames=0 into the kernel cmdline. So when the new RHEL 8 system booted up it was using the wrong interface names (eth).
The fix was simply grubby --remove-args="net.ifnames=0" --update-kernel ALL but felt stupid since RH emphasize the interface name change and then they themselves sabotage it for us.
This has been a huge shitshow for us. Migration has been such a pain for our VPS boxes, so that we just spooled up new instances on AlmaLinux and migrated data by hand. It was a long week.
What a mess. Centos Stream 8 seemed to work at first but then after upgrade to 9 secure boot didn't work at all, I couldn't even boot the 9 install media.
I stopped trusting CentOS dates and assumed CentOS 7 was EOL a long time ago because their yum repos were down, or they didn't provide certain updates anymore, or something. I noticed inconsistencies across machines I tried to update simultaneously, noticed there wasn't a common update point available (the machines were on slightly different versions to begin with), and decided it was junk and time to move on.
You're sure that's not simply a matter of the client-side caching having expired on some machines (thus reaching out again for updates) vs. not having expired on other machines (thus not checking the repo a second time)?
Alternatively maybe they were hitting different mirrors. But in either case, that's not an issue with the distribution.
Oh yes, I checked caching, checked that the URLs were official by comparing them to the URLs CentOS canonically suggested, checked that the URLs did include security update repos, definitely dove into commands for "force check updates" type commands and the kernel rpms just weren't there. Old ones, new ones, nothing.
Now I could have made a mistake, and according to the article text CentOS 7 "isn't EOL yet" so if someone has a mirror that does work I'd be interested to see evidence that kernel updates are even available. I'm not checking myself though, I don't trust Oracle enough to risk spending time standing up a CentOS 7 machine.
We experienced a migration from CentOS7 to Rocky 8.
Rocky 8 running on VirtulBox produces different results on some numerical tools compared with Docker container. CentOS 7 did not use to have that problem. Same program and floating point differences.
Other than that Rocky 8 ran all right.
I chose Centos Stream 8 over Centos 8 because I figured that the stream version's rolling updates would result in a later EOL. When Red Hat announced the unexpectedly short EOL for 8 I was glad for my choice. That said, what has Red Hat said about an EOL for Stream, if any?
Some of us are forced by edict/policy to RHEL-only. I'm doing work for a client whose IT dept requires it, because it's all they 'support'.
Their 'support' largely consists of
a) giving ridiculously small and unchangeable /home and /usr partition sizes,
b) randomly logging in at unannounced times to add more logging and monitoring tools in to /home and /usr (then rebooting, also unannounced),
c) sending sporadic emails telling us to clean up our system because we are dangerously low on drive space in /home and /usr.
Got a message the other day stating that they power down our machine in 2 days unless we freed up drive space because 'low drive space can impact system availability and security'. Powering it down also impacts availability, but that point seemed lost on them.
EDIT: Realizing now that ... our RHEL 7 will expire at end of month. We've had 0 communication from them about this. Annoyed because when I'd requested this machine back in 2021, asking for RHEL 8, I was told "we don't officially support that", and now we're going in to EOL territory because of RHEL 7. I'm expecting an "hair on fire" email in early July indicating our machines will be shut down shortly without an immediate upgrade, that we will be forced to do ourselves (because upgrades fall outside 'support').
Only 5% of my work - I don't deal with them but a few times per year. I'm constantly confounded by how difficult they make relatively simple things, compared to other clients/projects I work with. But... this is dealing with state government. There are definitely sharp people that work in state govt - not dumping on all of them. But... this dept's incentives don't typically align with most others, and there's generally little that can be done about it. My eyes water when I see what my project is required to pay for this, relative to the 'support' we get.
Sort of does. Can't imagine it's a typical experience and makes me wonder what's so unusual in this case. There are always bad experiences with support but usually systematic tire fires indicate some other issues.
if you are going to stay, don't update to 8. Update to 9. Any 'significant' fixes you need fixed for 8 wont be possible in this stage of life and only 9 fits that requirement.
at this point, it will probably have to be a rebuild of a few machines - i'm not sure we can jump from 7->9 in an automated way (is it possible?) and i'm unsure how we'd recover if there was a hiccup in that process.
possible good news is that we'd been forced to use more machines a few years ago due to some IT policy about databases not being colocated on the same physical instance as the application(s) accessing it, which forced multiple servers when fewer were needed. I was told that's not a hard requirement any longer, which may allow us to consolidate some pieces during a rebuild.
I"m not sure how mature the LEAPP program is, but you could always simplify the process by running your apps in an el7 container for the initial migration while you ensure the rest of userspace catches up/adequately tested.
Debian was always there, they deserve what they get.
I'm saying this as someone who fought tooth and nail to migrate to Debian instead of Scientific in 2010 because you can't ever trust a for profit company.
But that was too hard, and what would a grad student know about the real world anyway, so here we are.
Sorry but that's an asinine reply. The people looking for the support are busy doing something and looking to pay a company for the support. If they had the time to volunteer for support, they wouldn't be looking for support themselves. You're the one touting Debian out of nowhere, you can't ask the people you're talking to, to volunteer to make your point better.
Debian handles all of these transitions by itself during upgrade process, and shows you a nice readme before starting all of them.
For example, Debian has finished two big transitions recently. Merging /usr and 64 bit time support. Both are done on testing, and even on testing nothing has broken.
Another big change (which also made HN front page via LWN) was /tmp behavior change. It's handled differently. If your system is already installed, it doesn't change the behavior, but new systems will behave differently.
All of these changes are again communicated via "NEWS" mechanism. If Debian changes a config file, it's replaced. If you modified this file, apt will ask what you prefer, and you can diff the file in place.
In the past, many similar changes are made, and all were transparent. If you're not using any external repositories which change tons of system packages with their own versions, nothing changes during upgrades for you.
While there's an extensive release note provided with every release like [0], the upgrades are pretty straightforward.
As a result, having a few or many Debian systems which are older than a decade is a norm, not an exception.
There are a lot of HPC sysadmins that are very busy lately. A lot of third party software started dropping CentOS 7 support this year, and they never upgraded their clusters to 8/9.
The situation is pretty backwards for us, ironically.
Upgrading the OS is easy. Initial work is a couple of days, rest of the deployment is an hour or so, but the software we need to run doesn't support CentOS 7+. Containerization might help or not, but most software is distributed as RPM packages and some of them are written with things like Python2.7.
As the IBM constructorships’ beams cut up this ecosystem you kind of see why the head Vogon has no sympathy because the plans have not been “in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.’”
Personally, I found RPM distros to be quite stable but have largely moved over to Ubuntu LTS for servers (technically Debian also has a LTS release, but it’s not as mainstream) and Linux Mint locally (largely Ubuntu without focus on snaps and the Cinnamon desktop is pleasant), it’s been working pretty well so far.
Then again, I run most software in Docker containers, so thankfully underlying OS changes usually aren’t too bad for me to deal with.
>Some people have actually had good luck with Oracle Linux but that’s very much an individual (or corporate) choice to consider.
Jup for enterprise, OracleLinux it is. I don't like Oracle as any normal person would, but they never did some shady stuff with their Linux, it even works perfectly on a Raspberry PI.
Debian 12, after a period of using Ubuntu that ended when Canonical decided to put advertising in the friggin cli. Oh, and pushing snap, but the advertising is what really nailed it for me. ;)
I realy like debian but is there any equivalent to yum ? It did some really nice advanced stuff I don't think you can replicate with apt. The shell mode avoided me some really big trouble several times (erlang updates often made me uninstall anything using it on update).
Current job doesn't give me many chances to use linux rn so I'm a bit out of touch. Recently took a look at rocky and it felt like a centos. also tried ubuntu but I recall I had to remove some ads package yeah.
Funnily enough, years ago, I migrated from Debian as my daily driver to (at the time) "Fedora Core" on my desktop.
My first question was "what's the replacement for aptitude", and people pointed me to "yum shell". It was not as good, but I got used to it, and went with it.
If you run "aptitude" on debian, without any argument, you end up in a TUI, you can use it to install or remove packages from your system, and then see the "preview" of the change, and apply/cancel the change. The same way people use "yum shell".
I'm used to new "dnf shell", so I don't miss aptitude anymore, but I think aptitude is what you're looking for.
Interesting, in my head aptitude was an ubuntu thing so I never tried in debian. Thanks for the tip.
I don't have anything against apt, it's just specific edge cases when it really saved me massive headaches by being able to remove and add during same change without having to remove all apps depending on it.
+1 Alma is great. I've interacted less with Rocky but both projects have great teams behind them, and both have a clear mission and promise to their users.
Which is probably the right answer (or Rocky) if you're set on having CentOS like it was before the team was acquired by Red Hat (which frankly didn't change the situation as much as some folks assume it did).
Switched to Debian since one of services we use is Debian supported only so it was a logical choice. Some clients are still requesting Oracle or RHEL but we are pushing towards Debian. It was a nice ride with CentOS.
That all of Debian will have a baby? Or that all of ArchLinux gets impregnated at once? Big projects are usually not led by a single person....oh whait...linux is, and now Linus cannot have sex anymore, thank you OP.
There are a ton of perfectly good Linux distros out there--many with some type of support available. The thing with CentOS was that, especially latterly after the CentOS team was acquired by Red Hat, many saw CentOS--as a downstream rebuild from RHEL sources, albeit with a different build system--as a semi-supported version of RHEL for free.
For people who don't want CentOS Stream, i.e. basically the nightly RHEL builds, the lack of having a versioned CentOS mirroring RHEL versions coming out of Red Hat (for about the last 10 years) puts them in somewhat uncharted waters even if they could pick one of the RHEL alternatives and "probably" be fine.
I'm always amazed and envious at how many successes people have had with the conversions.
This has been a huge problem for me and I only have just about 75 boxes. Only about 15 or so have been converted successfully to RHEL 7.9 since March, none (successfully) to 8.x after that.
1. Convert2rhel has been a nightmare, only a few have worked properly the first time. I've had 8 tickets opened with Redhat. Once I get one working, the next one will fail with different problems.
2. LEAPP from 7.9 to 8.9 hasn't worked successfully yet. When it does it breaks absolutely everything from PHP to mail sending to databases.
3. I thought Alma might solve the problem, nope. Similar issues as point number 2. Or I get Python errors, or any variety of anything else.
I'm not sure what our path is going to be (maybe Tuxcare or Suse or something), but I'm concerned about my job now since I was in charge of this and it's been a disaster. The anxiety is high.
>but I'm concerned about my job now since I was in charge of this and it's been a disaster.
Your boss should understand this, if you are a medium sized company with highly configured boxes (pet style) and have to change major versions, crazy stuff always happens, I can tell you story's (especially firmware and blob related but also cronjobs, "optimized"-shell scripts, to really old php stuff and so on) about upgrades that will drive you to the church before you touch any "dnf upgrade", however that's why i tend for much shorter upgrade times (within 3-5 years).
Customer had highly customized RHEL7 boxes, upgrade to 8 was "terror inducing", and went so far to rebuild nearly the whole infra (about 20 boxes but with really excellent documentation). You are not alone.
Is the problem ultimately to do with your configurations on your centos 7 boxes? Converting to rhel should be easy if you aren't doing anything that red hat wouldn't support in rhel.
Yeah, I thought so too (since they're only moving from COS signed to RH signed), but it's hard to describe. I've had everything from proxy issues; to python; to PHP-fpm breaking after the upgrade. It's just a constant flood of 1 step forward 3 back. Some from random gremlins. For example, last week I wanted to try a test conversion of a box to Alma linux and it worked fine, but broke PHP, so I had to revert the snapshot.
This week, the analysis blows up completely, despite nothing changing on the box. And that's just dev systems, I have zero confidence to move any prod systems.
Unrelated, but any major OS version change these days feels contrived and unnecessary. What is changing so drastically that we need a major version bump and hours of upgrading?
IMO, the versioning should follow a pattern of 2024.01, 2025.03, etc.. and updates should be seamless.
Enterprise linux distributions pin package versions through the entire "major" release. They get security backports and often feature backports (for the first few years) but nothing which would change an API or ABI - for 10+ years.
As such, the answer to your question is "pretty much everything". Wildly different kernel, libraries, etc.
In theory it would be nice to roll upgrades like you suggest, but there are huge swathes of industry for which it's not a practical solution.
... And we all know that rolling upgrades frequently introduce breaking changes that don't make sense for enterprise environments. To your (great) point: Customers pay companies like red hat for software stability, both in how it works and how their software interfaces to it.
Directly relevant:
https://techstrongitsm.com/itsm-news/suse-offers-lifeline-to...
HN article about it: https://news.ycombinator.com/item?id=40732016Or Oracle Linux that is free and don't need's a minimal investment of 2500$ like SUSE's Liberty Linux.
> Or Oracle Linux that is free and don't need's a minimal investment of 2500$ like SUSE's Liberty Linux.
Why would you put your business at the mercy of Oracle?
>Why would you put your business at the mercy of Oracle?
You don't, if Oracle does something fishy you go to your bank take at least 2500$ and above that, 25$/y/instance and switch to "Liberty" Linux ;)
But comparing the reliability and not giving up on their "Free Enterprise Linux" Oracle (since 2006) [1] is funnily by far the best.
SUSE: openSUSE Leap is dead
RH: Centos (not stream) is dead
Alma: We are Centos-Stream
Rocky: Too early to say anything about longstanding promises
EuroLinux: Pfff
[1] https://en.wikipedia.org/wiki/Oracle_Linux
Oracle also was the first to release updates last time I compared the three of them side by side, which was about a year ago. I installed Rocky, Alma and Oracle 9 on three VMs, pointed them as close to upstream repositories as possible (to avoid mirroring delays), and just started them every day to see who was the first to update. I also checked RPM metadata which includes build time. The experiment ran for (IIRC) about two months.
Oracle was consistently the first, lagging behind RHEL for a few hours (for important stuff) to a couple of days (for less important ones). Alma was a very close second. Rocky would spend days to weeks and was by far the slowest.
No, I don't buy the argument that it's good for you because they receive additional testing. If you really need that much stability that you can't take an update after RHEL has done so, introduce your own delays and test your shit on staging. I'd like my upstream to be as quick as practically possible. Oracle and Alma cover that nicely.
All this might have changed in the meantime.
CentOS stream is red hat.
Same reason why you put it at the mercy of RedHat. It's easier.
Oracle is a different beast[1].
If it were any other company, I might agree with you. But I just don't think I'd ever touch anything Oracle makes with a 10' pole except under duress. It's better to do the work up front of just going with the alternative than risk getting pulled into that particular tar baby.
---
1. https://www.youtube.com/watch?v=-zRN7XLCRhc&t=2115s
That'll only buy you several months. IIRC OL7 EOL issue in December, unless you pay for extended support while you do your work to transition to a newer OS (please, please don't rely on extended support, unless desperate, switch to new versions.)
True but gives you some additional time, and yes don't rely on ext. support, neither from RH nor Oracle. It just gets harder and harder to upgrade, as i wrote further down my preferred upgrade time-frame is 3-5 years max.
Is Oracle committed to actually supporting the distro with security updates and so on? I was under the impression that it's simply a bug-for-bug compatibility rebuild from RHEL sources, which aside from rebuilding effort, the majority of support work was still done by RH.
It's a rebuild with a few more patches on top (like ksplice). If you need a RHEL rebuild, they're the first to release security updates of any of the rebuilds I've tested. See my other comment for more info.
I wish they would add ZFS to it.
Yeah. It's super ironic. ;)
It absolutely is, kind of unreal ;)
It is and it isn't. They have a huge commercial interest in not unnecessarily sharing money with Red Hat for customers that basically need a supported/certified platform primarily for their Oracle products. SUSE has had fairly recent management changes (including ex-Red Hatters coming in). Red Hat should probably never have acquihired the CentOS team in the first place--not a comment on the people but bringing CentOS in-house with all that implied. And the others are fairly small.
In hindsight I wish I'd told KB "No, don't do it" at the meeting prior to them joining. :/
For a public forum, let's just say there was a long history of expedient decisions which I had no hand in but many of us regret :-/ Understandable but there were almost certainly better paths.
Transitioning from CentOS 7 to RedHat 8 and 9 at my former company's private cloud has smooth for most teams, pareto-style, with 80% of migration-related incidents caused by the 20% of the teams that did some really weird changes to the VM's OS that was no longer allowed under RHEL 8 or 9.
At first, I thought it was just to reduce the complexity of managing hardening rules for several OS and OS versions.
Recently we had issues with RH 9 not having header packages for openssl 1.1 (which means e.g. you can't have erlang < 25). There's a potential for breakage for anything that is 3+ years old.
Not sure if that applies to your line of work but RHEL has been an increasingly annoying source of issues in low-latency space. Their kernels are a bizarre mixture of cherry picked stuff from upstream and a completely bonkers project structure. Also, they break ABI across minor kernel versions which is unthinkable in mainline. What I'm trying to say is: if you're doing more funky stuff with your OS, just go with a distro built on top of mainline.
It's a big issue for companies that have substantial deployment of CentOS7... and FedRAMP or similar clientele.
It's a very big issue for scientific clusters which depend on CentOS, too.
We'll find a way, though.
Scientific clusters have more options though, as Rocky/Alma and other alternatives to RHEL are available without dealing with "this distro does not ship FIPS-140-3 certified crypto" or similar.
Yes, that's true. However, tons of software was written solely for CentOS, so ironing out the small kinks will take some time.
The problem is, scientific software can silently error out in many ways (like slightly lost accuracy), and detecting and fixing those will take some time.
Considering some of these tools are developed over (almost two) decades, there's a lot of verification we have to go through.
Considering how many interesting issues that specific CentOS7-locked job had just with updating kernel to latest LTS (which required a lot of fun stuff as well, given that it didn't compile with included GCC)...
... I can imagine.
Rocky and Alma are okay-ish for scientific clusters. They work and drivers appear to mostly cooperate. Some changes, like switching to sssd, were suboptimal but to be expected.
CentOS just had a lot more testing being done by vendors. We still regularly hit problems with, e.g., Lenovo LESI being not aligned with our Rocky systems for many parameters.
Anecdotally, I am hearing scientific computing customers who are unhappy after moving to Rocky - the window where each minor OS version is "supported" is too small and their hardware vendors have trouble staying current. Have seen quite a few move to Ubuntu, and the rest switch to regular RHEL.
If you were using centos for fedramp, unless you got a variance from the feds, you were not in compliance. No one actually paid to have NIST evaluate centos's binaries (evaluation/certification must be of the compiled binary, not the source code, barring an exception made for openssl, and only openssl, not the kernel)
This wasn't done yet for FedRAMP High, but it passed the audit for earlier customers demanding FIPS-140-2 compliance.
Not that I am not saying it wasn't a clusterfuck of epic proportions in general.
Talking firsthand, There is significant effort into ensuring that both 8.6.z (through to current) and 9.2.z (through to current) are fedramp compliant, this requirement eats my day, every day.
8.6 and 9.2 kernels are released more frequently than other streams, if you want more frequent updates for compliance reasons, these are the streams to use.
Why is it a problem with FedRAMP? CentOS 8 is FIPS certified, has STIG profiles etc.
Last I checked, CentOS Stream is not FIPS certified, and CentOS 8 is already past EOL, which makes it not allowed for FedRAMP.
And IIRC CentOS8 FIPS certificate was taken out by Red Hat (wouldn't have had to implement our own FIPS handling on CentOS7 if not for that move).
Centos was never FIPS certified https://csrc.nist.gov/projects/cryptographic-module-validati...
There was a time when RHEL FIPS certificate also applied to CentOS, and one of my former $DAYJOBs depended on that for a long time. Then Red Hat pulled the cert, and later there was a mad scramble to get rid of CentOS because of the EOL :)
Oh wow! Do happen to have a link to the old cert? I /really/ looked for a way for us to stay on centos when we started doing this kind of stuff.
Doh, sorry. Forgot it only hit Red Hat 8 (and derivatives).
Just a heads up, since I couldn't comment on the LWP article about this a few days ago.
The leapp-upgrade program had a reproducible bug as late as last week where after every upgrade it inserted net.ifnames=0 into the kernel cmdline. So when the new RHEL 8 system booted up it was using the wrong interface names (eth).
The fix was simply grubby --remove-args="net.ifnames=0" --update-kernel ALL but felt stupid since RH emphasize the interface name change and then they themselves sabotage it for us.
I used ELevate for a few VMs at home and it worked pretty well. I upgraded from Centos 7 to Alma 9 one release at a time
https://almalinux.org/elevate/
That looks very useful. Thank you for linking!
This has been a huge shitshow for us. Migration has been such a pain for our VPS boxes, so that we just spooled up new instances on AlmaLinux and migrated data by hand. It was a long week.
Disaster for us too, not VPS but VMs on-prem.
What a mess. Centos Stream 8 seemed to work at first but then after upgrade to 9 secure boot didn't work at all, I couldn't even boot the 9 install media.
Secure boot seems very problematic with stream 9. I was looking at their shim signing approval and it seemed like it wasn't there yet.
IMO, Stream is just there for testing and no one cares if it works or not.
I stopped trusting CentOS dates and assumed CentOS 7 was EOL a long time ago because their yum repos were down, or they didn't provide certain updates anymore, or something. I noticed inconsistencies across machines I tried to update simultaneously, noticed there wasn't a common update point available (the machines were on slightly different versions to begin with), and decided it was junk and time to move on.
You're sure that's not simply a matter of the client-side caching having expired on some machines (thus reaching out again for updates) vs. not having expired on other machines (thus not checking the repo a second time)?
Alternatively maybe they were hitting different mirrors. But in either case, that's not an issue with the distribution.
Oh yes, I checked caching, checked that the URLs were official by comparing them to the URLs CentOS canonically suggested, checked that the URLs did include security update repos, definitely dove into commands for "force check updates" type commands and the kernel rpms just weren't there. Old ones, new ones, nothing.
Now I could have made a mistake, and according to the article text CentOS 7 "isn't EOL yet" so if someone has a mirror that does work I'd be interested to see evidence that kernel updates are even available. I'm not checking myself though, I don't trust Oracle enough to risk spending time standing up a CentOS 7 machine.
Companies like CIQ, OpenLogic, and TuxCare offer extended support for CentOS 7, and Red Hat offers services for RHEL7.
https://linux.slashdot.org/story/24/05/10/2256230/red-hat-an...
We experienced a migration from CentOS7 to Rocky 8. Rocky 8 running on VirtulBox produces different results on some numerical tools compared with Docker container. CentOS 7 did not use to have that problem. Same program and floating point differences. Other than that Rocky 8 ran all right.
I chose Centos Stream 8 over Centos 8 because I figured that the stream version's rolling updates would result in a later EOL. When Red Hat announced the unexpectedly short EOL for 8 I was glad for my choice. That said, what has Red Hat said about an EOL for Stream, if any?
Stream 8 is already EOL: https://blog.centos.org/2023/04/end-dates-are-coming-for-cen...
D'oh! Thanks for the heads up.
You can use LEAPP to move to CS9: https://www.redhat.com/en/resources/leapp-explained-detail
Heh Heh Heh
"LEAPP" is an interesting name choice, in that one of SUSE's distro's is called "Leap".
Wonder if there are trademark issues around that?
LEAPP seems like it's been around for a few years now, so probably nothing serious.
CentOS Stream has a 5-year lifecycle. Support ends when the full support (prior to maintenance support) period of the corresponding RHEL release ends.
There are so many alternatives to CentOS and RHEL that their absence is inconsequential.
Some of us are forced by edict/policy to RHEL-only. I'm doing work for a client whose IT dept requires it, because it's all they 'support'.
Their 'support' largely consists of
a) giving ridiculously small and unchangeable /home and /usr partition sizes,
b) randomly logging in at unannounced times to add more logging and monitoring tools in to /home and /usr (then rebooting, also unannounced),
c) sending sporadic emails telling us to clean up our system because we are dangerously low on drive space in /home and /usr.
Got a message the other day stating that they power down our machine in 2 days unless we freed up drive space because 'low drive space can impact system availability and security'. Powering it down also impacts availability, but that point seemed lost on them.
EDIT: Realizing now that ... our RHEL 7 will expire at end of month. We've had 0 communication from them about this. Annoyed because when I'd requested this machine back in 2021, asking for RHEL 8, I was told "we don't officially support that", and now we're going in to EOL territory because of RHEL 7. I'm expecting an "hair on fire" email in early July indicating our machines will be shut down shortly without an immediate upgrade, that we will be forced to do ourselves (because upgrades fall outside 'support').
That all sounds like a reason to polish up the resume.
Only 5% of my work - I don't deal with them but a few times per year. I'm constantly confounded by how difficult they make relatively simple things, compared to other clients/projects I work with. But... this is dealing with state government. There are definitely sharp people that work in state govt - not dumping on all of them. But... this dept's incentives don't typically align with most others, and there's generally little that can be done about it. My eyes water when I see what my project is required to pay for this, relative to the 'support' we get.
Sort of does. Can't imagine it's a typical experience and makes me wonder what's so unusual in this case. There are always bad experiences with support but usually systematic tire fires indicate some other issues.
if you are going to stay, don't update to 8. Update to 9. Any 'significant' fixes you need fixed for 8 wont be possible in this stage of life and only 9 fits that requirement.
at this point, it will probably have to be a rebuild of a few machines - i'm not sure we can jump from 7->9 in an automated way (is it possible?) and i'm unsure how we'd recover if there was a hiccup in that process.
possible good news is that we'd been forced to use more machines a few years ago due to some IT policy about databases not being colocated on the same physical instance as the application(s) accessing it, which forced multiple servers when fewer were needed. I was told that's not a hard requirement any longer, which may allow us to consolidate some pieces during a rebuild.
I"m not sure how mature the LEAPP program is, but you could always simplify the process by running your apps in an el7 container for the initial migration while you ensure the rest of userspace catches up/adequately tested.
Interesting thought. I may consider that as a fallback approach if we hit any issues. Thanks.
There are whole niche ecosystems started on Scientific Linux and moved to CentOS over the last decade.
A change like this for thousands of bare metal machines, is not inconsequential. The complete inverse is true, in fact.
Debian was always there, they deserve what they get.
I'm saying this as someone who fought tooth and nail to migrate to Debian instead of Scientific in 2010 because you can't ever trust a for profit company.
But that was too hard, and what would a grad student know about the real world anyway, so here we are.
Debian has relatively short support.
Centos 7 was released in 2014 and the support ends next week.
Debian 10 was released in 2019 and the support also ends next week.
>Debian has relatively short support.
Then make it longer. The debian team is starved for volunteers and would love to have more long term maintainers.
What an inside out reasoning
So we want:
1. free/gratis Linux distribution
2. long support (~ 10 years)
3. not needing to contribute (as a community)
It seems that we can only pick two from this list (even a distribution with short support cycle needs community).
Sorry but that's an asinine reply. The people looking for the support are busy doing something and looking to pay a company for the support. If they had the time to volunteer for support, they wouldn't be looking for support themselves. You're the one touting Debian out of nowhere, you can't ask the people you're talking to, to volunteer to make your point better.
The people wanting to pay for support can absolutely move to rhel with no pain what so ever.
There's Freexian (as in company) if you want: https://wiki.debian.org/LTS/Extended
How good are Freexian / Debian ELTS compared to RHEL for their 10 years support and security fixes? Are they cost effective?
https://www.freexian.com/lts/extended/docs/cost-estimation/
>The people looking for the support are busy doing something and looking to pay a company for the support
I mean, if they are using CentOS, apparently not.
"what would a grad student know about the real world" is seeming like fair analysis given this.
Universities aren't the real world and I was right about redhat.
However, you can release-upgrade a Debian installation in five minutes or less, without any big issues cropping up.
Also, you can re-install a cattle worker node in 10 minutes or less, from power on.
What breaking changes do those upgrades introduce? Is there any compatibility guide?
Debian handles all of these transitions by itself during upgrade process, and shows you a nice readme before starting all of them.
For example, Debian has finished two big transitions recently. Merging /usr and 64 bit time support. Both are done on testing, and even on testing nothing has broken.
Another big change (which also made HN front page via LWN) was /tmp behavior change. It's handled differently. If your system is already installed, it doesn't change the behavior, but new systems will behave differently.
All of these changes are again communicated via "NEWS" mechanism. If Debian changes a config file, it's replaced. If you modified this file, apt will ask what you prefer, and you can diff the file in place.
In the past, many similar changes are made, and all were transparent. If you're not using any external repositories which change tons of system packages with their own versions, nothing changes during upgrades for you.
While there's an extensive release note provided with every release like [0], the upgrades are pretty straightforward.
As a result, having a few or many Debian systems which are older than a decade is a norm, not an exception.
[0]: https://www.debian.org/releases/stable/amd64/release-notes/
https://www.debian.org/releases/stable/amd64/release-notes/ This has sections for how to upgrade from the previous version and what major changes/issues.
I daily drive Debian close to 20 years. We'd prefer to have Debian on board, too.
However this is where we're now and what we have to go through.
> they deserve what they get.
Why the anger though?
There are a lot of HPC sysadmins that are very busy lately. A lot of third party software started dropping CentOS 7 support this year, and they never upgraded their clusters to 8/9.
Containers have been a big help.
The situation is pretty backwards for us, ironically.
Upgrading the OS is easy. Initial work is a couple of days, rest of the deployment is an hour or so, but the software we need to run doesn't support CentOS 7+. Containerization might help or not, but most software is distributed as RPM packages and some of them are written with things like Python2.7.
As the IBM constructorships’ beams cut up this ecosystem you kind of see why the head Vogon has no sympathy because the plans have not been “in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.’”
Just curious. What are you guys using in place of centos? This days, ny servers are all debian or ubuntu server based.
Some people have actually had good luck with Oracle Linux but that’s very much an individual (or corporate) choice to consider.
Aside from that, Rocky Linux also seems like a great choice for many: https://rockylinux.org/
Some also say that AlmaLinux is pretty good: https://almalinux.org/
Personally, I found RPM distros to be quite stable but have largely moved over to Ubuntu LTS for servers (technically Debian also has a LTS release, but it’s not as mainstream) and Linux Mint locally (largely Ubuntu without focus on snaps and the Cinnamon desktop is pleasant), it’s been working pretty well so far.
Then again, I run most software in Docker containers, so thankfully underlying OS changes usually aren’t too bad for me to deal with.
>Some people have actually had good luck with Oracle Linux but that’s very much an individual (or corporate) choice to consider.
Jup for enterprise, OracleLinux it is. I don't like Oracle as any normal person would, but they never did some shady stuff with their Linux, it even works perfectly on a Raspberry PI.
Worth mentioning that Rocky is made to match RHEL bug-to-bug, so an equal alternative to CentOS, Rocky's maintainer is former engineer in CentOS team.
Alma independently manage their updates though, started by Cloudlinux, who are also experts in maintaining EL distros
Debian 12, after a period of using Ubuntu that ended when Canonical decided to put advertising in the friggin cli. Oh, and pushing snap, but the advertising is what really nailed it for me. ;)
> put advertising in the friggin cli.
And the logs. I think it's every 30 minutes or so that "buy Ubuntu advantage" gets logged, you have to disable a systemd timer to get rid of it.
I realy like debian but is there any equivalent to yum ? It did some really nice advanced stuff I don't think you can replicate with apt. The shell mode avoided me some really big trouble several times (erlang updates often made me uninstall anything using it on update).
Current job doesn't give me many chances to use linux rn so I'm a bit out of touch. Recently took a look at rocky and it felt like a centos. also tried ubuntu but I recall I had to remove some ads package yeah.
Funnily enough, years ago, I migrated from Debian as my daily driver to (at the time) "Fedora Core" on my desktop.
My first question was "what's the replacement for aptitude", and people pointed me to "yum shell". It was not as good, but I got used to it, and went with it.
If you run "aptitude" on debian, without any argument, you end up in a TUI, you can use it to install or remove packages from your system, and then see the "preview" of the change, and apply/cancel the change. The same way people use "yum shell".
I'm used to new "dnf shell", so I don't miss aptitude anymore, but I think aptitude is what you're looking for.
Interesting, in my head aptitude was an ubuntu thing so I never tried in debian. Thanks for the tip.
I don't have anything against apt, it's just specific edge cases when it really saved me massive headaches by being able to remove and add during same change without having to remove all apps depending on it.
“apt install foo+ bar-” will install foo and remove bar, in one operation.
as someone who has used debian and centos 6 +7 extensively apt and dpkg are more than sufficient replacement for yum with centos.
i know its a whole environment change but debian really is the logical replacement for a centos style deployment.
I would love to be able to comment on rocky linux but i havent used that quite yet.
Hmmm, I've just been using apt and the dpkg-* tools (ie dpkg-query), and for my uses it's been fine. :)
AlmaLinux user here, happy with it, everything works.
+1 Alma is great. I've interacted less with Rocky but both projects have great teams behind them, and both have a clear mission and promise to their users.
+2 The transition was very smooth
As soon as Red Hat made the EOL announcement for CentOS I moved to Almalinux.
Which is probably the right answer (or Rocky) if you're set on having CentOS like it was before the team was acquired by Red Hat (which frankly didn't change the situation as much as some folks assume it did).
We're trying to migrate everything we can to Debian.
Switched to Debian since one of services we use is Debian supported only so it was a logical choice. Some clients are still requesting Oracle or RHEL but we are pushing towards Debian. It was a nice ride with CentOS.
RHEL is free for up to 16 systems (physical or VMs). For CI, you can run Rocky/Alma to ensure continuous compatibility if you grow past 16 beefy VMs.
>RHEL is free for up to 16 systems
If they don't change their mind, i would not build my business on a promise made by IBM ;)
But you'll build your business on software you get for free on the internet with absolutely no commitment behind it?
>absolutely no commitment behind it?
I think commitment to your baby (your project) is far more morally superior than commitment to just making your stakeholders happy...no?
BTW: I buy Github DVDs, I don't load it from the "Internet".
Until they have an actual baby and don't have free time to spend maintaining large projects for free.
>to spend maintaining large projects for free.
What is your point, anyway?
That all of Debian will have a baby? Or that all of ArchLinux gets impregnated at once? Big projects are usually not led by a single person....oh whait...linux is, and now Linus cannot have sex anymore, thank you OP.
CentOS Stream
There are a ton of perfectly good Linux distros out there--many with some type of support available. The thing with CentOS was that, especially latterly after the CentOS team was acquired by Red Hat, many saw CentOS--as a downstream rebuild from RHEL sources, albeit with a different build system--as a semi-supported version of RHEL for free.
For people who don't want CentOS Stream, i.e. basically the nightly RHEL builds, the lack of having a versioned CentOS mirroring RHEL versions coming out of Red Hat (for about the last 10 years) puts them in somewhat uncharted waters even if they could pick one of the RHEL alternatives and "probably" be fine.
Depends on if you have a heavy base using it. It’s not inconsequential then.
I'm always amazed and envious at how many successes people have had with the conversions.
This has been a huge problem for me and I only have just about 75 boxes. Only about 15 or so have been converted successfully to RHEL 7.9 since March, none (successfully) to 8.x after that.
1. Convert2rhel has been a nightmare, only a few have worked properly the first time. I've had 8 tickets opened with Redhat. Once I get one working, the next one will fail with different problems.
2. LEAPP from 7.9 to 8.9 hasn't worked successfully yet. When it does it breaks absolutely everything from PHP to mail sending to databases.
3. I thought Alma might solve the problem, nope. Similar issues as point number 2. Or I get Python errors, or any variety of anything else.
I'm not sure what our path is going to be (maybe Tuxcare or Suse or something), but I'm concerned about my job now since I was in charge of this and it's been a disaster. The anxiety is high.
>but I'm concerned about my job now since I was in charge of this and it's been a disaster.
Your boss should understand this, if you are a medium sized company with highly configured boxes (pet style) and have to change major versions, crazy stuff always happens, I can tell you story's (especially firmware and blob related but also cronjobs, "optimized"-shell scripts, to really old php stuff and so on) about upgrades that will drive you to the church before you touch any "dnf upgrade", however that's why i tend for much shorter upgrade times (within 3-5 years).
Customer had highly customized RHEL7 boxes, upgrade to 8 was "terror inducing", and went so far to rebuild nearly the whole infra (about 20 boxes but with really excellent documentation). You are not alone.
Is the problem ultimately to do with your configurations on your centos 7 boxes? Converting to rhel should be easy if you aren't doing anything that red hat wouldn't support in rhel.
Yeah, I thought so too (since they're only moving from COS signed to RH signed), but it's hard to describe. I've had everything from proxy issues; to python; to PHP-fpm breaking after the upgrade. It's just a constant flood of 1 step forward 3 back. Some from random gremlins. For example, last week I wanted to try a test conversion of a box to Alma linux and it worked fine, but broke PHP, so I had to revert the snapshot.
This week, the analysis blows up completely, despite nothing changing on the box. And that's just dev systems, I have zero confidence to move any prod systems.
Unrelated, but any major OS version change these days feels contrived and unnecessary. What is changing so drastically that we need a major version bump and hours of upgrading?
IMO, the versioning should follow a pattern of 2024.01, 2025.03, etc.. and updates should be seamless.
Enterprise linux distributions pin package versions through the entire "major" release. They get security backports and often feature backports (for the first few years) but nothing which would change an API or ABI - for 10+ years.
As such, the answer to your question is "pretty much everything". Wildly different kernel, libraries, etc.
In theory it would be nice to roll upgrades like you suggest, but there are huge swathes of industry for which it's not a practical solution.
... And we all know that rolling upgrades frequently introduce breaking changes that don't make sense for enterprise environments. To your (great) point: Customers pay companies like red hat for software stability, both in how it works and how their software interfaces to it.