4 ms·

The time (some years back now, it was a 5.10 patch and a 5.8 rpm) they mis-backported a perl patch to work around a bug in a deprecated CPAN module for one of t

by mst 3y ago

The time (some years back now, it was a 5.10 patch and a 5.8 rpm) they mis-backported a perl patch to work around a bug in a deprecated CPAN module for one of their enterprise customers and in the process caused a 2x-30x slowdown of lots of other newer code (including the library that had replaced it in the majority of production environments by that point) was 'fun'. Took me a couple years to get together a coalition of commercial support customers to apply sufficient pressure that they'd listen when I got the original author of the patch to break out the small words and crayon drawings and explain what they'd done wrong and how to backport it properly. Happily, they now generally snapshot Fedora's perl builds pretty directly and the current Fedora team have been fantastic to work with as a downstream but it was a spectacular mess at the time and there are plenty of people out there who still haven't forgiven them (I think I try to act -as if- I've forgiven them but I'm not sure I actually have).

bonzini 3y ago
To be honest it's a pretty good track record if you can only remember one instance of botching a backport, it's many years old, and it didn't have any security impact unlike Debian's ssh key generation.
- pm215 3y ago
  True. The one I remember is when they mis-backported some version of libncurses so it was the v4 ABI but had the v3 soname, but that was like two decades ago :-)
- mst 3y ago
  Given I work for https://shadow.cat/ https://shadow.cat/ and we do primarily perl/CPAN contracting, consultancy and commercial support (and don't run RHEL ourselves) I remember this one because I had to help multiple customers and probably by the end three digits' worth of companies via community support to work around the debacle. I suspect more man-hours were lost to the (now reversed for quite a while thanks to the Fedora team) decision to have their 'perl' package only be half a perl install (and to mass report the resulting problems that decision created to cpan authors without ever sending a single patch) but the original subject was things they botched by accident rather than things they broke deliberately. Though the uninformed arrogance behind the poor decisions and dismissive attitude to the resulting problems, even when they were impacting RHEL support customers, was very similar in both cases. Had they responded to realising they'd completely broken a bunch of paying customers' primary revenue generating applications by actually trying to do something about it I would have been happy to file the original mistake under "shit happens." As it is, I don't think 'pretty good track record' applies, I'm afraid.
scns 3y ago
"Forgive your enemies, but never forget their names." - John F. Kennedy
coldtea 3y ago
>The time (some years back now, it was a 5.10 patch and a 5.8 rpm) they mis-backported a perl patch to work around a bug in a deprecated CPAN module for one of their enterprise customers and in the process caused a 2x-30x slowdown of lots of other newer code (including the library that had replaced it in the majority of production environments by that point) was 'fun'. That ONE time might have been fun, but with Arch, Ubuntu and so on, you get to have such fun times all the time.
- ylyn 3y ago
  With Arch you don't get such fun times because Arch doesn't generally backport patches. They just update.
  - coldtea 3y ago
    That's where the fun times come from.
    - deleted 3y ago
      [deleted]
Spooky23 3y ago
That’s life. I can’t count the number of times some rando Microsoft patch has boned a massive global infrastructure.
ilyt 3y ago
I saw RedHat backport kernel bugs... we had one that bricked VLAN support on certain NIC that was put into Centos 5. For those that do not know, RHEL kernels are a bit of an abomination with both bugfixes and new features backported, just so they can tell to customers that the kernel version is "stable" (which is a bit of clown fiesta as Linux doesn't break backward compat on kernel level anyway) and that was one of the things mis-backported and not even tested, just copied bug from upstream. .... which half a year later got backported to Centos 6 too... They also do nasty shit like re-enabling/backporting legacy ciphers to software (like OpenSSH) that removed them or disabled them in code base. So you go thru audit, there are points about not using this and that now-insecure ciphers, but you look and see "hey, this one was disabled by default in <current version of OpenSSH in RHEL>, we're fine", and discover that you're NOT fine because RHEL in their infinite wisdom enabled it by default because some old legacy customers needed it and couldn't be bothered to edit their own configs. After near-two-decades I want to get off Mr. Red Hat weird ride. Just fucking use Debian
- chasil 3y ago
  Red Hat also removes drivers for older hardware. SAS RAID has been a recent victim. Two alternate kernels will restore most missing drivers, El Repo or the Oracle UEK.

by mst 3y ago

bonzini 3y ago

To be honest it's a pretty good track record if you can only remember one instance of botching a backport, it's many years old, and it didn't have any security impact unlike Debian's ssh key generation.

pm215 3y ago
True. The one I remember is when they mis-backported some version of libncurses so it was the v4 ABI but had the v3 soname, but that was like two decades ago :-)
mst 3y ago
Given I work for https://shadow.cat/ https://shadow.cat/ and we do primarily perl/CPAN contracting, consultancy and commercial support (and don't run RHEL ourselves) I remember this one because I had to help multiple customers and probably by the end three digits' worth of companies via community support to work around the debacle. I suspect more man-hours were lost to the (now reversed for quite a while thanks to the Fedora team) decision to have their 'perl' package only be half a perl install (and to mass report the resulting problems that decision created to cpan authors without ever sending a single patch) but the original subject was things they botched by accident rather than things they broke deliberately. Though the uninformed arrogance behind the poor decisions and dismissive attitude to the resulting problems, even when they were impacting RHEL support customers, was very similar in both cases. Had they responded to realising they'd completely broken a bunch of paying customers' primary revenue generating applications by actually trying to do something about it I would have been happy to file the original mistake under "shit happens." As it is, I don't think 'pretty good track record' applies, I'm afraid.

scns 3y ago

"Forgive your enemies, but never forget their names." - John F. Kennedy

coldtea 3y ago

>The time (some years back now, it was a 5.10 patch and a 5.8 rpm) they mis-backported a perl patch to work around a bug in a deprecated CPAN module for one of their enterprise customers and in the process caused a 2x-30x slowdown of lots of other newer code (including the library that had replaced it in the majority of production environments by that point) was 'fun'. That ONE time might have been fun, but with Arch, Ubuntu and so on, you get to have such fun times all the time.

ylyn 3y ago
With Arch you don't get such fun times because Arch doesn't generally backport patches. They just update.
- coldtea 3y ago
  That's where the fun times come from.
  - deleted 3y ago
    [deleted]

Spooky23 3y ago

That’s life. I can’t count the number of times some rando Microsoft patch has boned a massive global infrastructure.

ilyt 3y ago

I saw RedHat backport kernel bugs... we had one that bricked VLAN support on certain NIC that was put into Centos 5. For those that do not know, RHEL kernels are a bit of an abomination with both bugfixes and new features backported, just so they can tell to customers that the kernel version is "stable" (which is a bit of clown fiesta as Linux doesn't break backward compat on kernel level anyway) and that was one of the things mis-backported and not even tested, just copied bug from upstream. .... which half a year later got backported to Centos 6 too... They also do nasty shit like re-enabling/backporting legacy ciphers to software (like OpenSSH) that removed them or disabled them in code base. So you go thru audit, there are points about not using this and that now-insecure ciphers, but you look and see "hey, this one was disabled by default in <current version of OpenSSH in RHEL>, we're fine", and discover that you're NOT fine because RHEL in their infinite wisdom enabled it by default because some old legacy customers needed it and couldn't be bothered to edit their own configs. After near-two-decades I want to get off Mr. Red Hat weird ride. Just fucking use Debian

chasil 3y ago
Red Hat also removes drivers for older hardware. SAS RAID has been a recent victim. Two alternate kernels will restore most missing drivers, El Repo or the Oracle UEK.