6 ms·

A couple of years ago we reached out to the python community about wheels and arm64 - how it should be handled and whether they plan on embedding non-x86 blobs.

by smspf 6y ago

A couple of years ago we reached out to the python community about wheels and arm64 - how it should be handled and whether they plan on embedding non-x86 blobs. We received the standard "we'll think about it and let you know". Now that Apple switched to arm64, all communities are suddenly interested in porting things to arm64. And of course, Apple is not investing in these ports, at least as far as I know. They just rely on what other arm64 players did in the ecosystem before Apples rolled out M1; respectively lets developers figure out the remaining porting. As much as I hate to say this, IBM does handle porting things to ppc64 right - you can find IBM contributed code and optimizations anywhere you look. For many packages, porting to arm64 was a matter of "does it have ppc64 support? if so, it can be reused for arm64" ... Disclaimer - used to be a contractor porting stuff to arm64 for a couple of years.

Hamuko 6y ago
For what it's worth, I think Apple sent the Homebrew devs ARM Mac Minis for building and when the Homebrew devs asked for more, they sent more. So they're doing something, even if they're not doing as much as one would hope.
- smspf 6y ago
  To be honest, I don't blame Apple for this. I blame the ARM ecosystem which is very fragmented, each company working with ARM is contributing to the stuff they are interested in and that's it. Lots of contractors and always shuffling/changing projects they work on.
  - oblio 6y ago
    Isn't Apple doing the exact same thing with their proprietary ARM ISA extensions?
    - my123 6y ago
      Their ISA extension is an ML-specific one, and macOS runs fine with it disabled. Their public compilers do not support it either. You are supposed to use it through Accelerate.framework, which exposes a more traditional interface to that capability.
      - skissane 6y ago
        > Their ISA extension is an ML-specific one Apple Silicon has more than one ISA extension. There is also the x86 memory ordering extension used by Rosetta 2. Maybe there are yet others too.
        my123 6y ago
        > There is also the x86 memory ordering extension used by Rosetta 2. That one isn't really a requirement either, and is handled fully in kernel mode. Wkdm and friends are handled fully in kernel mode. APRR? Not a strict requirement for user-mode, JIT regions are just left as RWX without it.
        saagarjha 6y ago
        Apple enforces APRR on macOS; you need to use their APIs to work within the confines of W^X.
        saagarjha 6y ago
        There are more but they are not exposed to applications. Apple wants you to ship standard arm64 code.
- simonh 6y ago
  They also submitted some patches to MacPorts, and I think Homebrew though I couldn't find a reference offhand. So i suppose it's always possible to argue they could do more, but code and hardware is a long way from nothing.
- darksaints 6y ago
  Not doing as much as one would hope is a massive understatement. Apple has banked their future on this architecture, and they have $200 Billion with a B in cash laying around doing absolutely nothing. Sending a few thousand dollars worth of hardware to your largest open source software distributor barely breaks into the territory of self-interest. On the scale of self-destructive to self-interested, it rates somewhere in the realm of wiping your ass after taking a shit.
m463 6y ago
They can't. They've put themselves on the path where they can't abide by the GPL. GPL3 says you must preserve the right to run the software and apple takes that away from their customers then sells it back. So apple nopes out of it. They rely on the initial boost they got from GPL software. They do some MIT/BSD licensed stuff (and publish some of it but not everything). Every year less and less happens, unless it's their own language/etc. They do have nice gradients and round corners though. :)
- sdoering 6y ago
  Yes. That. I am done with the MacOS for good. My first Mac was a Classic II, back in '92. My last will be a MB Pro 2018 (actually 2020, but already repaired once and broken beyond working condition again). It was my third MB Pro in 3 years and every single one had to be repaired once before it crashed shortly thereafter. The company I work for has to replace around 1% - 2% of their MB Pro fleet per month. 6 - 10 devices are always in repair (additionally 1 - 2%). I have notified my superior, that I am not able to work properly in January when I am back from a 2 months LOA were I privately switched everything over onto a WIN10 with WSL2 running Ubuntu. I am not missing a thing currently. OK - I know at one point I will probably be missing Keynote.
- JimDabell 6y ago
  > They can't. They've put themselves on the path where they can't abide by the GPL. PyPy is MIT licensed.
  - m463 6y ago
    I'm talking about apple investing in non-apple/community software.
    - oefrha 6y ago
      You're talking about GPL3 software, which is irrelevant here. They clearly do contribute to MIT-licensed "non-apple/community software", as I demonstrated up thread, so what's the point of this "they can't" BS.
fouric 6y ago
Why would Apple invest in third-party toolchains? Aren't they incentivized to only support Xcode, Swift, and other tools that are as tightly bound to their ecosystem as possible?
- bla3 6y ago
  Because they sell hardware and commodize its complement. Having a good Dev experience is part of that, be that a good story for mac-only software (swift/xcode -- ymmv if this is better than other technologies, but that's the intent) or having your posix software mostly just work.
- thetinguy 6y ago
  Apple uses community tools internally for some things.
oefrha 6y ago
> And of course, Apple is not investing in these ports What count as "investing in these ports"? Does submitting patches to CPython (and many other open source projects, including NumPy) for macOS 11 and Apple Silicon count? Here's a list of Apple-submitted PRs on python/cpython: https://github.com/python/cpython/pulls?q=is%3Apr+author%3Alawrence-danna-apple https://github.com/python/cpython/pulls?q=is%3Apr+author%3Al... Also some co-authored patches excluded by the search. See https://bugs.python.org/issue41100 https://bugs.python.org/issue41100 for more related PRs. Your subsequent comments about IBM seem to imply that these do count. I'm also curious about who's "we" in "we reached out to the python community about wheels and arm64". --- Edit: Forgot to say, arm wheels have been supported for many years now (not sure about the specific timeline of aarch64 support, but if 32-bit arm was supported I don't see why aarch64 wouldn't be). Maybe most famously there's https://www.piwheels.org/ https://www.piwheels.org/ for RPis. Are you talking about aarch64 support on PyPI/warehouse?
- smspf 6y ago
  By investing in these ports I was thinking about any contribution that enables/improves arm64 support. So yes, PRs for python itself definitely qualify. However, the python ecosystem is not only python, there are lots of python packages that still lack arm64 optimizations or native arm64 support. Most of them work, being python, but every once in a while you'd run into a package that does not just work on arm64 without some tweaking. In my case, "we" referred to the working group of ARM contractors working a very specific higher level project (at the time we were doing some cloud/openstack/k8s stuff, which indirectly required some python packages). Note that the way contractors were assigned to projects at the time implied that sometimes you'd run into overlaps, e.g. some packages we needed had already been ported by a different team of contractors for a different project a while back. Sometimes multiple teams would hit the same missing port at the same time, most of the time we could sync and team up by escalating that through the right channels, but sometimes we'd just end up duplicating work items and end up with two different sets of patches/ports. About wheels, someone else in my team handled that, so I'm not that familiar with the issue - it might have been a particular package that was lacking arm64 wheels or it might have been impractical to use a different repository only for arm64 (in general we'd try avoiding using separate resources based on architecture and we definitely did prefer the arch-agnostic approaches, e.g. debian repository URLs are arch-independent, dockerhub manifests allowed using the same container image names etc.). We were interested PyPI at the time (couple of years ago), maybe the situation has improved meanwhile, but we just accepted compilation at install time as a good enough solution.

by smspf 6y ago

Hamuko 6y ago

For what it's worth, I think Apple sent the Homebrew devs ARM Mac Minis for building and when the Homebrew devs asked for more, they sent more. So they're doing something, even if they're not doing as much as one would hope.

smspf 6y ago
To be honest, I don't blame Apple for this. I blame the ARM ecosystem which is very fragmented, each company working with ARM is contributing to the stuff they are interested in and that's it. Lots of contractors and always shuffling/changing projects they work on.
- oblio 6y ago
  Isn't Apple doing the exact same thing with their proprietary ARM ISA extensions?
  - my123 6y ago
    Their ISA extension is an ML-specific one, and macOS runs fine with it disabled. Their public compilers do not support it either. You are supposed to use it through Accelerate.framework, which exposes a more traditional interface to that capability.
    - skissane 6y ago
      > Their ISA extension is an ML-specific one Apple Silicon has more than one ISA extension. There is also the x86 memory ordering extension used by Rosetta 2. Maybe there are yet others too.
      - my123 6y ago
        > There is also the x86 memory ordering extension used by Rosetta 2. That one isn't really a requirement either, and is handled fully in kernel mode. Wkdm and friends are handled fully in kernel mode. APRR? Not a strict requirement for user-mode, JIT regions are just left as RWX without it.
        saagarjha 6y ago
        Apple enforces APRR on macOS; you need to use their APIs to work within the confines of W^X.
      - saagarjha 6y ago
        There are more but they are not exposed to applications. Apple wants you to ship standard arm64 code.
simonh 6y ago
They also submitted some patches to MacPorts, and I think Homebrew though I couldn't find a reference offhand. So i suppose it's always possible to argue they could do more, but code and hardware is a long way from nothing.
darksaints 6y ago
Not doing as much as one would hope is a massive understatement. Apple has banked their future on this architecture, and they have $200 Billion with a B in cash laying around doing absolutely nothing. Sending a few thousand dollars worth of hardware to your largest open source software distributor barely breaks into the territory of self-interest. On the scale of self-destructive to self-interested, it rates somewhere in the realm of wiping your ass after taking a shit.

m463 6y ago

They can't. They've put themselves on the path where they can't abide by the GPL. GPL3 says you must preserve the right to run the software and apple takes that away from their customers then sells it back. So apple nopes out of it. They rely on the initial boost they got from GPL software. They do some MIT/BSD licensed stuff (and publish some of it but not everything). Every year less and less happens, unless it's their own language/etc. They do have nice gradients and round corners though. :)

sdoering 6y ago
Yes. That. I am done with the MacOS for good. My first Mac was a Classic II, back in '92. My last will be a MB Pro 2018 (actually 2020, but already repaired once and broken beyond working condition again). It was my third MB Pro in 3 years and every single one had to be repaired once before it crashed shortly thereafter. The company I work for has to replace around 1% - 2% of their MB Pro fleet per month. 6 - 10 devices are always in repair (additionally 1 - 2%). I have notified my superior, that I am not able to work properly in January when I am back from a 2 months LOA were I privately switched everything over onto a WIN10 with WSL2 running Ubuntu. I am not missing a thing currently. OK - I know at one point I will probably be missing Keynote.
JimDabell 6y ago
> They can't. They've put themselves on the path where they can't abide by the GPL. PyPy is MIT licensed.
- m463 6y ago
  I'm talking about apple investing in non-apple/community software.
  - oefrha 6y ago
    You're talking about GPL3 software, which is irrelevant here. They clearly do contribute to MIT-licensed "non-apple/community software", as I demonstrated up thread, so what's the point of this "they can't" BS.

fouric 6y ago

Why would Apple invest in third-party toolchains? Aren't they incentivized to only support Xcode, Swift, and other tools that are as tightly bound to their ecosystem as possible?

bla3 6y ago
Because they sell hardware and commodize its complement. Having a good Dev experience is part of that, be that a good story for mac-only software (swift/xcode -- ymmv if this is better than other technologies, but that's the intent) or having your posix software mostly just work.
thetinguy 6y ago
Apple uses community tools internally for some things.

oefrha 6y ago

> And of course, Apple is not investing in these ports What count as "investing in these ports"? Does submitting patches to CPython (and many other open source projects, including NumPy) for macOS 11 and Apple Silicon count? Here's a list of Apple-submitted PRs on python/cpython: https://github.com/python/cpython/pulls?q=is%3Apr+author%3Alawrence-danna-apple https://github.com/python/cpython/pulls?q=is%3Apr+author%3Al... Also some co-authored patches excluded by the search. See https://bugs.python.org/issue41100 https://bugs.python.org/issue41100 for more related PRs. Your subsequent comments about IBM seem to imply that these do count. I'm also curious about who's "we" in "we reached out to the python community about wheels and arm64". --- Edit: Forgot to say, arm wheels have been supported for many years now (not sure about the specific timeline of aarch64 support, but if 32-bit arm was supported I don't see why aarch64 wouldn't be). Maybe most famously there's https://www.piwheels.org/ https://www.piwheels.org/ for RPis. Are you talking about aarch64 support on PyPI/warehouse?

smspf 6y ago
By investing in these ports I was thinking about any contribution that enables/improves arm64 support. So yes, PRs for python itself definitely qualify. However, the python ecosystem is not only python, there are lots of python packages that still lack arm64 optimizations or native arm64 support. Most of them work, being python, but every once in a while you'd run into a package that does not just work on arm64 without some tweaking. In my case, "we" referred to the working group of ARM contractors working a very specific higher level project (at the time we were doing some cloud/openstack/k8s stuff, which indirectly required some python packages). Note that the way contractors were assigned to projects at the time implied that sometimes you'd run into overlaps, e.g. some packages we needed had already been ported by a different team of contractors for a different project a while back. Sometimes multiple teams would hit the same missing port at the same time, most of the time we could sync and team up by escalating that through the right channels, but sometimes we'd just end up duplicating work items and end up with two different sets of patches/ports. About wheels, someone else in my team handled that, so I'm not that familiar with the issue - it might have been a particular package that was lacking arm64 wheels or it might have been impractical to use a different repository only for arm64 (in general we'd try avoiding using separate resources based on architecture and we definitely did prefer the arch-agnostic approaches, e.g. debian repository URLs are arch-independent, dockerhub manifests allowed using the same container image names etc.). We were interested PyPI at the time (couple of years ago), maybe the situation has improved meanwhile, but we just accepted compilation at install time as a good enough solution.