14 ms·
PyPy Project looking for sponsorship to add support for Apple Silicon
- mkj 6y agoThis seems like the kind of thing that Apple themselves are in a good position to help with. Seems pretty improbable unfortunately.
- tgv 6y agoAlso: the people or organizations making these requests. They know making software isn't a free lunch.
- praptak 6y agoIt is, if someone else funds it. It's a commons dilemma of sorts.
- deleted 6y ago[deleted]
- Hamuko 6y agoI imagine that PyPy is way too obscure for Apple to care about it. Amazon would probably care a lot more if they're planning on adopting more ARM in AWS.
- zelphirkalt 6y agoIn the Python community it is a pretty well known implementation.
- jjice 6y agoFor sure, but it's not used as much on user Arm machines compared to server applications, which Apple doesn't have much of a stake in ATM.
- pjmlp 6y agoUnfortunately no, JITs in other communities get much more love than PyPy does.
- vips7L 6y agoI was under the impression that Apple doesn't really care for JIT compilers.
- pjmlp 6y agoThey do, in macOS and in what concerns iOS, JSCore. PyPy doesn't get much love from Python community overall, too focused on CPython + C, no matter on which platform. Hence why efforts like Julia and Chapel spring to life.
- saagarjha 6y agoThey ship multiple JITs, so I don’t really understand your comment.
- vips7L 6y agoThey don't allow anyone else to ship a JIT on iOS and you set special security settings in your app on macOS to ship one.
- yen223 6y agoI don't doubt that many Python people have heard of Pypy, but in my experience not many companies actually use Pypy in production
- person_of_color 6y agoWho uses PyPy in production?
- sanxiyn 6y agoWho uses Apple Silicon in production?
- _ix 6y agoTruly my own ignorance... help me out! Is this port strictly for Apple silicon, ~ARM64, or is that the same thing? Where would this leave support for say, AWS Graviton instances?
- mattip 6y agoPyPy already supports ARM64. This is additional work needed to support macOS 11 on M1. Apple changed some things that impact PyPy, like the register uses and ffi calling conventions. They updated clang to handle this, but PyPy's JIT emits assembler directly and so requires work to support M1.
- cesarb 6y ago> Apple changed some things that impact PyPy, like the register uses and ffi calling conventions. I thought everyone who used 64-bit ARM used ARM's AAPCS64 (https://github.com/ARM-software/abi-aa/blob/master/aapcs64/aapcs64.rst https://github.com/ARM-software/abi-aa/blob/master/aapcs64/a...), so the register usage and FFI calling convention should be the same as on Linux and Windows. What did Apple do differently that would affect the PyPy JIT?
- mattip 6y agoThere are some slight nuances that affect the JIT. https://developer.apple.com/documentation/xcode/writing_arm64_code_for_apple_platforms https://developer.apple.com/documentation/xcode/writing_arm6...
- Phemist 6y agoI wish I could contribute more generally, but I think the PyPy project is a great cause and will put down a small amount to help with this. I've come to the realization that, to get more performance out of python, the most pythonic way is to simply use PyPy, rather than try and hack your way around issues like attributes requiring dict lookups, access to locals() being faster than globals(), etc.
- dingaling 6y ago> and will put down a small amount to help with this I'd politely ask you to reconsider. You'd basically be subsidising work that benefits the World's highest-capitalised company, to whom the investment in this port would be a fraction of a second's profit.
- mhh__ 6y agoFor references, Apple's piggy bank (Braeburn) currently has about $250E9 on the books.
- chrisseaton 6y agoThe money isn't for Apple, it's for PyPy developers, who don't work for Apple.
- Avamander 6y agoIndirectly it pretty much is. If software works on Apple hardware people wish to buy it. Boycotting noncollaborating hardware vendors by FOSS is LONG overdue. In my opinion there's too much catering to companies that do not wish to be helped. Though, we can't forget that, aarch64 support also helps other ARM vendors, open hardware, theoretically. So there's that.
- oefrha 6y agoBy your logic every hardware vendor should pay for the development of all software (or at least all open source software) that aims to run on that hardware. Which is plain ridiculous. Apple is pretty collaborating in this transition. Not paying for development of every remotely popular project under the sun does not make them “noncollaborating”. Apple ships a Python.framework and a user-facing python3 installation through CLT, so I expect them to contribute to CPython. And they did. They don’t ship anything remotely related to PyPy, so I don’t expect them to do anything. A gesture would be nice, but that’s being nice.
- buffalobuffalo 6y agoI've never been able to find a good use case for pypy. Why would you ever want to run CPU bound tasks in python? I'm sure there are arguments about the huge ecosystem, not having to rewrite code in another language, etc. But most of the widely used stuff in python already has underlying compiled code for the heavy lifting. Also, given the large amount of parallelism in modern hardware architecture, and python's lackluster concurrency support, I just don't see a reason for using it.
- chrisseaton 6y ago> Why would you ever want to run CPU bound tasks in python? ... But most of the widely used stuff in python already has underlying compiled code for the heavy lifting. Haven't you just answered your own question? We know that people want to run CPU bound tasks in Python so much that they went to the effort of writing native modules because they couldn't do it in Python. > python's lackluster concurrency support This is a common misconception - Python actually has fully concurrent threads already.
- seunosewa 6y agoNot fully. The GIL is always held when executing Python bytecode, because it isn’t threadsafe. Dropping the GIL (parallelism) only happens in native code that explicitly drops it.
- chrisseaton 6y ago> Not fully. This isn't true - they are fully concurrent. The GIL prevents parallelism - not concurrency.
- woadwarrior01 6y ago> The GIL prevents parallelism - not concurrency. When a Python thread is holding on to the GIL (running Python bytecode), how many other Python threads can concurrently run in the same process? The answer is zero. Sure the interpreter releases the GIL every n bytecode ops, and C extensions can release the GIL before doing anything IO bound and reacquire it (i.e wait for it) afterwards, but that isn’t full concurrency, in my books.
- smspf 6y agoA couple of years ago we reached out to the python community about wheels and arm64 - how it should be handled and whether they plan on embedding non-x86 blobs. We received the standard "we'll think about it and let you know". Now that Apple switched to arm64, all communities are suddenly interested in porting things to arm64. And of course, Apple is not investing in these ports, at least as far as I know. They just rely on what other arm64 players did in the ecosystem before Apples rolled out M1; respectively lets developers figure out the remaining porting. As much as I hate to say this, IBM does handle porting things to ppc64 right - you can find IBM contributed code and optimizations anywhere you look. For many packages, porting to arm64 was a matter of "does it have ppc64 support? if so, it can be reused for arm64" ... Disclaimer - used to be a contractor porting stuff to arm64 for a couple of years.
- Hamuko 6y agoFor what it's worth, I think Apple sent the Homebrew devs ARM Mac Minis for building and when the Homebrew devs asked for more, they sent more. So they're doing something, even if they're not doing as much as one would hope.
- smspf 6y agoTo be honest, I don't blame Apple for this. I blame the ARM ecosystem which is very fragmented, each company working with ARM is contributing to the stuff they are interested in and that's it. Lots of contractors and always shuffling/changing projects they work on.
- oblio 6y agoIsn't Apple doing the exact same thing with their proprietary ARM ISA extensions?
- my123 6y agoTheir ISA extension is an ML-specific one, and macOS runs fine with it disabled. Their public compilers do not support it either. You are supposed to use it through Accelerate.framework, which exposes a more traditional interface to that capability.
- captain_price7 6y ago> Either we get sufficient money, and maybe support, and then we can do it quickly; or PyPy will just remain not natively available on M1 hardware for the next 3-5 years Out of curiosity, what is a ballpark, a rough estimate of the "sufficient money" for a project like this?
- franciscop 6y agoFrom the rest of the blog post it seems enough to buy a single M1 machine with 8+ GB of RAM, so my estimate is that it should be a bit over $1k for a basic Mac Mini + shipping + Apple Care.
- deleted 6y ago[deleted]
- captain_price7 6y agoI don't think we should exclude fund for developer time from the calculation, at least that's what I understood from this sentence in post: > I would do it myself for a small amount of money.
- franciscop 6y agoAh true, I got confused with "It can be either a machine provided and maintained by a third party, or alternatively a pot of money big enough to support the acquision of a machine and ongoing work of one of us." and conflated the "pot of money" for the machine with the total being asked for, my bad.
- mhh__ 6y agoThat assumes they work for free
- anigbrowl 6y agoI too became frustrated reading the article and trying to figure out what they were asking for. Protip: patrons and sponsors do not want to know all the details, and certainly not up-front. What they want is to know what you're asking for, what you offer in return and then (briefly) how one leads to the other. Thus: We make Pypy, which does X and is used by Y people, including projects such as Z, z, and z'. We'd love to bring Pypy to Apple's exciting new M1 hardware, but that needs resources. We're looking for $CASH_MONEY to buy some M1 hardware that we can develop, test against, and eventually run the repository on. With #NUM_MACHINES available, we believe was can have PyPy available to all M1 developers by %CALENDAR_MONTH. We also like eating and drinking and if you send us more money we will spend some of it on !GOOD_TIME(HAVING). The whole approach of 'we're not asking for much, just x would help' is how volunteers talk to each other, minimizing their ask so that reorienting effort is not too burdensome for anyone. But how you talk to each other is not a good way to talk to external readers, because it requires effort to parse what you are getting at and then further effort to negotiate and arrange everything which is a distraction from their own concerns. Many people would rather just give you some cash and sign up for a monthly progress update. There's a reason services Kickstarter etc. work so well: they save the donors time and entanglement. Just set up : PyPy on Mac M1 for $5000?' or similar and make it easy for people to throw money at you.
- gigatexal 6y agoThere are enough engineers posting here from well paid FAANG roles to fund this effort...
- _alex_ 6y agoHow much money and how do we contribute?
- superkuh 6y agoIt's pretty incredible that people are actually considering using Apple's new architecture given that it can't actually run anything that exists natively except apple's OS and core applications.
- shagie 6y agoJetbrains has DataGrip and RubyMine already... and apparently just got IntelliJ on M1. https://blog.jetbrains.com https://blog.jetbrains.com ... and reading that list, there's a fair bit more that's been built. The other thing is... all if the iPhone and iPad apps "work" (once you extract them from the iDevice) - its a lot more than just the core apps.
- superkuh 6y agoThat's just my point. A select handful of applications that had to be ported over by large corporations. But that's it. None of the extremely long tail of actual available applications or even anything that isn't "big". It's like a soviet grocer's window where it's full of specially prepared "food" in the front display and then empty behind. These announcements that large institutions and corporations are looking to try to get their application running aren't signals of something good. They indicate something very bad that requires large scale resources to even hope to deal with. The vast majority of applications will never be ported.
- katbyte 6y agoMy coworkers who switched over did so for the extra long batter life; responsiveness, and faster compilation times
- fiddlerwoaroof 6y agoYeah, the battery life is amazing: so far, I just don’t worry about whether my laptop is plugged in.
- 6y ago
- mattip 6y agoFor those asking "who uses PyPy?": the truth is that very few people using it get back to us and say how it helps them or what they are doing with it. For instance crossbar.io uses it [0]. Some shops use it as a second step until they refactor code into a compiled language, but often that second step never materializes. But the value of a second implementation of a language goes beyond the immediate "who uses this in practice". It can be a fertile bed for innovation and for new ideas, and provides a contrast to the nay sayers. For instance, the recent pitch to vastly improve CPython's speed has some roots in ideas that were tested out in PyPy. CFFI [1], revdb [2] and vmprof [3] all started as PyPy projects. Some of these turned out to be very popular, some less so. The next project in this line is HPy [4] (still alpha-quality), which is trying to re-think the C-API for Python to make it even easier to interface with. RPython [5], the language behind PyPy, is also an accessible playground for dynamic language research. [0] https://crossbar.io/about/FAQ/#python-runtime https://crossbar.io/about/FAQ/#python-runtime [1] https://cffi.readthedocs.io/en/latest/ https://cffi.readthedocs.io/en/latest/ [2] https://morepypy.blogspot.com/2016/07/reverse-debugging-for-python.html https://morepypy.blogspot.com/2016/07/reverse-debugging-for-... [3] https://vmprof.readthedocs.io/en/latest/ https://vmprof.readthedocs.io/en/latest/ [4] https://hpy.readthedocs.io/en/latest/ https://hpy.readthedocs.io/en/latest/ [5] https://rpython.readthedocs.io/en/latest/examples.html https://rpython.readthedocs.io/en/latest/examples.html
- bobajeff 6y agoTo add I believe PyPy also inspired asm.js the precursor to Web Assembly.
- Recursing 6y agoAFAIK pypy improvements also led to cpython compact dictionaries (which led to dictionaries being ordered) and many many cpython performance improvements (the most recent one being LOAD_ATTR caching, with huge performance gains). I wonder if sponsoring pypy instead of the PSF might be a way for people that want "a focus on performance improvements, instead of fancy features" to vote with their wallets
- ngfellow 6y ago
- a012 6y agoI'd add GRPC to the wishlist too: https://github.com/grpc/grpc/issues/4221 https://github.com/grpc/grpc/issues/4221
- alien_ 6y agoAmazon AWS will soon add support for the M1 hardware to the macOS EC2 managed offering announced a few weeks ago. AWS also offers free credits to prominent Open Source projects. It's not available yet but it should be a good option once they become available, you can immediately start the process of applying for the credits. This way you would get access to the required hardware without any costs and without having to maintain it. They are designed from the ground up towards such CI/development use cases. You can also use the same credits for getting access to Intel and ARM Linux, as well as Windows EC2 instances. This may improve your project's CI and build times. Disclaimer: I work at AWS.
- uncledave 6y agoThis is no use unless you reduce the minimum tenancy. Might as well buy a Mac Mini and shove it on a shelf.
- alien_ 6y agoI'm not working closely with the Mac EC2 nor Open Source departments, so I may be wrong but the project should get credits covering a certain monthly bill, the amount would be requested on application for the credits. It shouldn't matter what type of instances are launched and the credits cover any other services, not just the EC2 compute, as long as the bills are less than the monthly credits.
- sgao08 6y agoUse cases like this are far more than you can imagine. The longer I work for AWS the more I realize this...
- barkingcat 6y agowhat if the free credits cover 24x7 tenancy
- dontbecheap 6y agoPyPy volunteers are being too nice asking for a couple of thousand dollars. Don't be afraid to ask for enough money to support development and maintenance for this new platform, purchase the hardware, as well as earning replacement income for opportunity loss. $50K minimum is more reasonable. Get the funding you deserve from these free riding companies using the software.
- mlinksva 6y agoI guess they're not asking for individual donations in this post, but still, the "donations page" link in the sidebar 404s. It appears from https://www.pypy.org/howtohelp.html https://www.pypy.org/howtohelp.html that https://opencollective.com/pypy https://opencollective.com/pypy is the correct link.
- selimnairb 6y agoSo they just need $900?
- outside1234 6y agoApple?
- stuaxo 6y agoApple should pay some of their vast horde of cash.