5 ms·

I don't know why GCC-12 was in the picture, but I hope all of you can understand one thing: python's source code must be compatible with GCC 6. And no so called

by snnn 5y ago

I don't know why GCC-12 was in the picture, but I hope all of you can understand one thing: python's source code must be compatible with GCC 6. And no so called "Threading [Optional]" features. It is all because CentOS is dead. CentOS has been sold to IBM. There is no CentOS 8 anymore. The free lunch is ended. And there wouldn't be another OSS project to replace it. Linux kernel developers want to use C11. CPython developers want to use C11. But does your compiler support it? If you were a C++ developer, would you request C++17/C++20 also? CPython must be aligned with the manylinux project. manylinux build every python minor release from source. Historically manylinux only use CentOS. CentOS has devtoolset, which allows you use the latest GCC in an very old Linux release. Red hat has spent much time on it. Now you can't get it for free because CentOS is dead. So the manylinux community is switching to Ubuntu. Ubuntu can only provide GCC 6 to them. If anycode in CPython is not compatible with GCC6, then manylinux will have to drop ubuntu too. And they don't have any other alternative. As I said, for now Red Hat devtoolset is the only solution. When IBM discontinued the CentOS project, most people do not understand what it means to the OSS community. (I work for Microsoft. It doesn't stop me to love Linux.)

tomrod 5y ago
Huh. Could Python switch to Alpine instead of Ubuntu?
- LukeShu 5y ago
  The "manylinux" platform tag definition means GNU libc. musl libc systems are not within the "manylinux" Python platform, they are a separate "musllinux" platform tag. See: PEP 600, PEP 656. While the manylinux infrastructure project may use Alpine to produce "musllinux" binaries, Alpine will never be usable to produce "manylinux" binaries. So Alpine would only be useful in addition to another distro, not instead of another distro.
- snnn 5y ago
  See: https://github.com/pypa/manylinux/pull/1135 https://github.com/pypa/manylinux/pull/1135 . Alpine is a supplement. It can't replace the others. It hasn't been widely accepted by the community yet. I believe many python projects are not afforded to replace libc.
chc 5y ago
I'm a little confused by this. I'm on the last Ubuntu LTS version (20.04) and the default GCC version appears to be 9.3, with 10.3 available as an option. So why are they stuck on 6?
- snnn 5y ago
  As a user, if you build every python package from source, it's ok. But if you a maintainer of an OSS project and you need to publish binary packages for it, then you will hit the trouble. Binaries built on Ubuntu 20.04 can only support Ubuntu 20.04 and newer. So you'd better to choose an older Linux release to target broader users. Now most python packages choose CentOS 6 or 7. See https://github.com/pypa/manylinux/issues/1012 https://github.com/pypa/manylinux/issues/1012 for more details. They need help!
  - coldtea 5y ago
    >Binaries built on Ubuntu 20.04 can only support Ubuntu 20.04 and newer So? Who are those people building for servers they don't control and arbitrary old versions of OSes?
    - chc 5y ago
      Evidently the point of the manylinux project is to make it easy to distribute Python binary wheels for Linux, so that basically is their use case. Though they're not building for arbitrary old versions — they're building for a concrete set of old OS versions defined by Python standards. I think the idea is that you don't want to require everyone installing your Python module to have a full set of build tools installed, so they're providing a way to distribute a binary module that supports a guaranteed set of supported Linuxes.
    - oefrha 5y ago
      The manylinux wheels distributed on PyPI need to be built against the oldest glibc imaginable, which is defined to be the glibc on some ancient version of CentOS (a different version for each different manylinux platform tag). If you build your own wheels, of course no one’s going to stop you from building against anything.
formerly_proven 5y ago
(The reason this is significant is because the manylinux infrastructure is responsible for approximately 100% of the binary Python extension builds provided as wheels)
- uranusjr 5y ago
  What? Windows and macOS wheels are a thing, they are built on Windows and macOS respectively, and certainly are responsible for over 0%.
jcelerier 5y ago
> If you were a C++ developer, would you request C++17/C++20 also? yes absolutely. I think that it is wild that some people stay with older compilers solely because they happen to be on an old distro and don't want to update. Tying compiler versions to operating system versions is absolutely braindead. A compiler is just a program that takes text and outputs a binary which is supposed to work on anything with the expected ABI - you could even cross-compile from windows if you wanted ; at least I've done the opposite (cross-compile windows binaries from a linux host) a few times. e.g. personally I mostly use clang-13, soon 14, for development with every possible C++20 goodies, and ship software that works back to windows 7, mac os 10.13 and until recently centos:7-era linux userland (recently upgraded to 8). There is zero reason to use an older compiler. > I don't know why GCC-12 was in the picture, but I hope all of you can understand one thing: python's source code must be compatible with GCC 6. And no so called "Threading [Optional]" features. It is all because CentOS is dead. CentOS has been sold to IBM. There is no CentOS 8 anymore. The free lunch is ended. And there wouldn't be another OSS project to replace it. the day centos:8 ended I replaced centos:8 with rockylinux:8 in my docker image for my builds and everything continued working fine with the latest GCC and Clang versions. Honestly your pains regarding gcc-6 are entirely self-inflicted. Just ship a more recent GCC binary with the manylinux project or something, you can build it on centos 5 if you fancy in order to get it to work on older libcs
- kstrauser 5y ago
  I agree, and especially in the context of Python. My Mac comes with Python 2.7 and 3.8, but it seems like almost everyone uses Homebrew or pyenv to locally install a newer version. I rarely hear of anyone going out of their way to stick with the system version outside of specific scenarios (like "IT locks down developer laptops") or such. And while I'm keenly aware that Python and C++ are very different languages, if someone asked if I'd insist on using the "new" Python17 (aka 3.6), like that was wildly and unreasonably bleeding edge, I'd literally laugh at them.
  - snnn 5y ago
    If everyone just use the latest macOS and the latest iOS, I'm less concerned. But I haven't heard any IT manager enforcing that. macOS 12 has come out, but usually they still allows you using macOS 11 as long as you have installed all the security bug fixes. For macOS and iOS, the problem is more complicated. Let's say I want to publish a package to pypi.org. The package contains some binaries compiled from C++ and it requires C++17. Then what is the lowest macOS version the binary can support?? It's very tricky that if the build machine which generates the package is macOS 11, then it can have Xcode 12.5. Otherwise it has to use XCode 12.4 or lower. And in XCode 12.4 it says std::optional, which is a C++17 feature, is only supported in macOS 10.14+. But if the build machine has macOS 11 and XCode 12.5, then the binary would be able to run on 10.13 too. And in whatever case, it can't support 10.12 or lower. I believe you don't want to build tensorflow/pytorch packages from source. So the maintainers of these two OSS projects must consider the things above. If you wonder why they don't use C++17, this is the reason. For the same reason they want to avoid C11 optional features too.
wizee 5y ago
Rocky Linux provides a perfectly good and compatible drop in replacement for CentOS 8. So does Alma Linux.
- snnn 5y ago
  They don't handle bugs. If the bug is reproducible on RHEL, they will suggest you report the bug to RHEL. It scares people away. At this moment, the manylinux community is considering Rocky Linux and Alma Linux, but it's very hard for them to make the decision because of lacking direct support.
  - bonzini 5y ago
    It was the same for CentOS.
ciupicri 5y ago
Even if CentOS 8 is not anymore, we still have CentOS Stream 8.
- coldtea 5y ago
  Which has a different release model than Centos, so it's not a replacement for the business cases Centos was used (basically: long term stability and support).
arka2147483647 5y ago
As i see it, the real problem is the lack of a stable linux ABI for binary programs. Because linux does not have one, people need to play around with old distros and compilers to build something that will work on anything released after $SomeOldDistro.
- deleted 5y ago
  [deleted]
westurner 5y ago
> for now Red Hat devtoolset is the only solution. When IBM discontinued the CentOS project, most people do not understand what it means to the OSS community. "Compilers and Runtimes" > "CentOS sysroot for linux-* Platforms" https://conda-forge.org/docs/maintainer/infrastructure.html#compilers-and-runtimes https://conda-forge.org/docs/maintainer/infrastructure.html#... From https://conda-forge.org/docs/user/announcements.html https://conda-forge.org/docs/user/announcements.html : >> 2021-10-13: GCC 10 and clang 12 as default compilers for Linux and macOS >> These compilers will become the default for building packages in conda-forge. Conda-forge specifies enough to solve for CentOS sysroot compatibility and newer GCC is already specified. CentOS (RHEL SRPMs with RH trademarks removed + EPEL) lives on as {Rocky Linux, Alma Linux, CentOS Stream,} and SUSE is still RHEL-compatible. https://github.com/pypa/cibuildwheel https://github.com/pypa/cibuildwheel : >> Build Python wheels for all the platforms on CI with minimal configuration. >> Python wheels are great. Building them across Mac, Linux, Windows, on multiple versions of Python, is not. >> cibuildwheel is here to help. cibuildwheel runs on your CI server - currently it supports GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - and it builds and tests your wheels across all of your platforms.
WalterBright 5y ago
> But does your compiler support it? Yes, ImportC is a C11 compiler! https://dlang.org/spec/importc.html https://dlang.org/spec/importc.html `_Static_assert` and `_Generic`, too. No C compiler is complete without extensions. ImportC is no exception. It has modules, forward references, and compile time function execution.
saurik 5y ago
I don't understand any of this logic: you can trivially run new versions of gcc on any distribution you want, and you can trivially use new versions of gcc on new distributions (which is sane) to target arbitrarily-old distributions. I routinely use bleeding edge compilers to target pretty ancient systems, and have for decades now... like, the entire point of gcc is that it is easy to compile it to run on whatever crazy system you have and then it should be able to target whatever other crazy system you have as long as you have a sysroot for it (which is of course somewhat easy to obtain as it is generally equivalent to whatever system you would have used to compile "natively" for that system).
rurban 5y ago
Using an insecure C standard does not improve the situation, broken compilers neither. C is broken from C11 to C26 by committee. GCC was broken from 9 to 11. Python has at least the option to use GitHub, which can eventually detect PR's with unidentifiable identifiers. Linux will need to use linters to detect homoglyphs or bidi attacks, because reviewing emailed patches is impossible.

by snnn 5y ago

tomrod 5y ago

Huh. Could Python switch to Alpine instead of Ubuntu?

LukeShu 5y ago
The "manylinux" platform tag definition means GNU libc. musl libc systems are not within the "manylinux" Python platform, they are a separate "musllinux" platform tag. See: PEP 600, PEP 656. While the manylinux infrastructure project may use Alpine to produce "musllinux" binaries, Alpine will never be usable to produce "manylinux" binaries. So Alpine would only be useful in addition to another distro, not instead of another distro.
snnn 5y ago
See: https://github.com/pypa/manylinux/pull/1135 https://github.com/pypa/manylinux/pull/1135 . Alpine is a supplement. It can't replace the others. It hasn't been widely accepted by the community yet. I believe many python projects are not afforded to replace libc.

chc 5y ago

I'm a little confused by this. I'm on the last Ubuntu LTS version (20.04) and the default GCC version appears to be 9.3, with 10.3 available as an option. So why are they stuck on 6?

snnn 5y ago
As a user, if you build every python package from source, it's ok. But if you a maintainer of an OSS project and you need to publish binary packages for it, then you will hit the trouble. Binaries built on Ubuntu 20.04 can only support Ubuntu 20.04 and newer. So you'd better to choose an older Linux release to target broader users. Now most python packages choose CentOS 6 or 7. See https://github.com/pypa/manylinux/issues/1012 https://github.com/pypa/manylinux/issues/1012 for more details. They need help!
- coldtea 5y ago
  >Binaries built on Ubuntu 20.04 can only support Ubuntu 20.04 and newer So? Who are those people building for servers they don't control and arbitrary old versions of OSes?
  - chc 5y ago
    Evidently the point of the manylinux project is to make it easy to distribute Python binary wheels for Linux, so that basically is their use case. Though they're not building for arbitrary old versions — they're building for a concrete set of old OS versions defined by Python standards. I think the idea is that you don't want to require everyone installing your Python module to have a full set of build tools installed, so they're providing a way to distribute a binary module that supports a guaranteed set of supported Linuxes.
  - oefrha 5y ago
    The manylinux wheels distributed on PyPI need to be built against the oldest glibc imaginable, which is defined to be the glibc on some ancient version of CentOS (a different version for each different manylinux platform tag). If you build your own wheels, of course no one’s going to stop you from building against anything.

formerly_proven 5y ago

(The reason this is significant is because the manylinux infrastructure is responsible for approximately 100% of the binary Python extension builds provided as wheels)

uranusjr 5y ago
What? Windows and macOS wheels are a thing, they are built on Windows and macOS respectively, and certainly are responsible for over 0%.

jcelerier 5y ago

> If you were a C++ developer, would you request C++17/C++20 also? yes absolutely. I think that it is wild that some people stay with older compilers solely because they happen to be on an old distro and don't want to update. Tying compiler versions to operating system versions is absolutely braindead. A compiler is just a program that takes text and outputs a binary which is supposed to work on anything with the expected ABI - you could even cross-compile from windows if you wanted ; at least I've done the opposite (cross-compile windows binaries from a linux host) a few times. e.g. personally I mostly use clang-13, soon 14, for development with every possible C++20 goodies, and ship software that works back to windows 7, mac os 10.13 and until recently centos:7-era linux userland (recently upgraded to 8). There is zero reason to use an older compiler. > I don't know why GCC-12 was in the picture, but I hope all of you can understand one thing: python's source code must be compatible with GCC 6. And no so called "Threading [Optional]" features. It is all because CentOS is dead. CentOS has been sold to IBM. There is no CentOS 8 anymore. The free lunch is ended. And there wouldn't be another OSS project to replace it. the day centos:8 ended I replaced centos:8 with rockylinux:8 in my docker image for my builds and everything continued working fine with the latest GCC and Clang versions. Honestly your pains regarding gcc-6 are entirely self-inflicted. Just ship a more recent GCC binary with the manylinux project or something, you can build it on centos 5 if you fancy in order to get it to work on older libcs

kstrauser 5y ago
I agree, and especially in the context of Python. My Mac comes with Python 2.7 and 3.8, but it seems like almost everyone uses Homebrew or pyenv to locally install a newer version. I rarely hear of anyone going out of their way to stick with the system version outside of specific scenarios (like "IT locks down developer laptops") or such. And while I'm keenly aware that Python and C++ are very different languages, if someone asked if I'd insist on using the "new" Python17 (aka 3.6), like that was wildly and unreasonably bleeding edge, I'd literally laugh at them.
- snnn 5y ago
  If everyone just use the latest macOS and the latest iOS, I'm less concerned. But I haven't heard any IT manager enforcing that. macOS 12 has come out, but usually they still allows you using macOS 11 as long as you have installed all the security bug fixes. For macOS and iOS, the problem is more complicated. Let's say I want to publish a package to pypi.org. The package contains some binaries compiled from C++ and it requires C++17. Then what is the lowest macOS version the binary can support?? It's very tricky that if the build machine which generates the package is macOS 11, then it can have Xcode 12.5. Otherwise it has to use XCode 12.4 or lower. And in XCode 12.4 it says std::optional, which is a C++17 feature, is only supported in macOS 10.14+. But if the build machine has macOS 11 and XCode 12.5, then the binary would be able to run on 10.13 too. And in whatever case, it can't support 10.12 or lower. I believe you don't want to build tensorflow/pytorch packages from source. So the maintainers of these two OSS projects must consider the things above. If you wonder why they don't use C++17, this is the reason. For the same reason they want to avoid C11 optional features too.

wizee 5y ago

Rocky Linux provides a perfectly good and compatible drop in replacement for CentOS 8. So does Alma Linux.

snnn 5y ago
They don't handle bugs. If the bug is reproducible on RHEL, they will suggest you report the bug to RHEL. It scares people away. At this moment, the manylinux community is considering Rocky Linux and Alma Linux, but it's very hard for them to make the decision because of lacking direct support.
- bonzini 5y ago
  It was the same for CentOS.

ciupicri 5y ago

Even if CentOS 8 is not anymore, we still have CentOS Stream 8.

coldtea 5y ago
Which has a different release model than Centos, so it's not a replacement for the business cases Centos was used (basically: long term stability and support).

arka2147483647 5y ago

As i see it, the real problem is the lack of a stable linux ABI for binary programs. Because linux does not have one, people need to play around with old distros and compilers to build something that will work on anything released after $SomeOldDistro.

deleted 5y ago
[deleted]

westurner 5y ago

> for now Red Hat devtoolset is the only solution. When IBM discontinued the CentOS project, most people do not understand what it means to the OSS community. "Compilers and Runtimes" > "CentOS sysroot for linux-* Platforms" https://conda-forge.org/docs/maintainer/infrastructure.html#compilers-and-runtimes https://conda-forge.org/docs/maintainer/infrastructure.html#... From https://conda-forge.org/docs/user/announcements.html https://conda-forge.org/docs/user/announcements.html : >> 2021-10-13: GCC 10 and clang 12 as default compilers for Linux and macOS >> These compilers will become the default for building packages in conda-forge. Conda-forge specifies enough to solve for CentOS sysroot compatibility and newer GCC is already specified. CentOS (RHEL SRPMs with RH trademarks removed + EPEL) lives on as {Rocky Linux, Alma Linux, CentOS Stream,} and SUSE is still RHEL-compatible. https://github.com/pypa/cibuildwheel https://github.com/pypa/cibuildwheel : >> Build Python wheels for all the platforms on CI with minimal configuration. >> Python wheels are great. Building them across Mac, Linux, Windows, on multiple versions of Python, is not. >> cibuildwheel is here to help. cibuildwheel runs on your CI server - currently it supports GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - and it builds and tests your wheels across all of your platforms.

WalterBright 5y ago

> But does your compiler support it? Yes, ImportC is a C11 compiler! https://dlang.org/spec/importc.html https://dlang.org/spec/importc.html `_Static_assert` and `_Generic`, too. No C compiler is complete without extensions. ImportC is no exception. It has modules, forward references, and compile time function execution.

saurik 5y ago

I don't understand any of this logic: you can trivially run new versions of gcc on any distribution you want, and you can trivially use new versions of gcc on new distributions (which is sane) to target arbitrarily-old distributions. I routinely use bleeding edge compilers to target pretty ancient systems, and have for decades now... like, the entire point of gcc is that it is easy to compile it to run on whatever crazy system you have and then it should be able to target whatever other crazy system you have as long as you have a sysroot for it (which is of course somewhat easy to obtain as it is generally equivalent to whatever system you would have used to compile "natively" for that system).

rurban 5y ago

Using an insecure C standard does not improve the situation, broken compilers neither. C is broken from C11 to C26 by committee. GCC was broken from 9 to 11. Python has at least the option to use GitHub, which can eventually detect PR's with unidentifiable identifiers. Linux will need to use linters to detect homoglyphs or bidi attacks, because reviewing emailed patches is impossible.