8 ms·
This is wonderful. This could become the best way to move Python projects to Rust: initially just run on the RustPython interpreter, but then optimize low level
by hathawsh 8y ago
This is wonderful. This could become the best way to move Python projects to Rust: initially just run on the RustPython interpreter, but then optimize low level routines in Rust. In 15 years I wouldn't be surprised if this or something like it surpasses CPython in popularity.
Still, no discussion about Python implementations is complete without some mention of the infamous GIL (global interpreter lock). :-) CPython has it, Pypy doesn't (I think) (EDIT: yes it does), Jython doesn't, etc. What is the GIL plan for RustPython?
- lykr0n 8y agoPyPy has the GIL.
- schmichael 8y agoPyPy does have a GIL: http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-why http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-... Jython, as you state, does not: http://www.jython.org/jythonbook/en/1.0/Concurrency.html http://www.jython.org/jythonbook/en/1.0/Concurrency.html
- znpy 8y agoJython is however stuck with Python 2.x compatibility only as far as I know (I'd be happy to be proven wrong).
- mooreds 8y agoJython 3.x development is "in progress": https://wiki.python.org/jython/JythonFaq/GeneralInfo https://wiki.python.org/jython/JythonFaq/GeneralInfo Last commit was over a year ago: https://github.com/jython/jython3 https://github.com/jython/jython3
- tln 8y agoAgreed, this looks like a great project! Grumpy was supposed to accomplish the same for Python->Go, and although now abandoned, probably holds some lessons in how to design a platform to help Python projects get Rusted. Grumpy compiled python code to fairly unreadable Go, and then quickly compiled the result. One effect of this is that a programmer could theoretically refactor the resulting Go code gradually.
- amelius 8y agoWriting a concurrent runtime system including garbage collector is a serious effort, and that's why all those other versions of Python don't support it and are stuck with a GIL. Hence, I highly doubt that this Rust version of Python has gotten rid of the GIL. I'd love to see a better separation of language and VMs. I think it's a bit sad that a language designer has to either implement their runtime system from scratch, or has to run it on top of a VM that was designed for another language (Java in the case of Jython). Therefore, the thing I'm looking forward to most is a concurrent, generic and portable VM written in Rust.
- pdpi 8y ago> I'd love to see a better separation of language and VMs. I think it's a bit sad that a language designer has to either implement their runtime system from scratch, or has to run it on top of a VM that was designed for another language (Java in the case of Jython). Wasn't Perl 6's Parrot kind of meant to fulfil that role?
- lizmat 8y agoI'd say not originally. But over time, as the Perl 6 project got delayed in the 2000's, it was decided that Parrot would be a runtime for all scripting languages. Which in turn meant it couldn't cater well enough for any. Which led to its demise.
- chromatic 8y ago> Wasn't Perl 6's Parrot kind of meant to fulfil that role? Yes, that was an original project goal. You can see this as far back as Larry's State of the Onion 2003: https://www.perl.com/pub/2003/07/16/soto2003.html/ https://www.perl.com/pub/2003/07/16/soto2003.html/ ... the "Parrot: Some Assembly Required" article written by Simon Cozens in September 2001: https://www.perl.com/pub/2001/09/18/parrot.html/ https://www.perl.com/pub/2001/09/18/parrot.html/ ... or, if you trust Git commits rather than articles which could have been edited in the meantime, the same article revised as introductory docs in the Parrot repository in December 2001: https://github.com/parrot/parrot/blob/9bc8687beb5180e4cc89712dc69d0b828670d5ba/docs/intro.pod https://github.com/parrot/parrot/blob/9bc8687beb5180e4cc8971...
- std_throwawayay 8y agoThe GIL comes with great convenience as you don't have to worry about a whole host of data races. It's no silver bullet but it's mighty convenient if you only do some multi-threading like in a web-server. Many libraries are not prepared for the disappearance of the GIL and while it's not a general problem for python per se it will be a great amount of work to make every library compatible with GILless python. Therefore I think that you must always provide an option for the GIL that is enabled by default in order to provide backward compatibility.
- pdonis 8y ago> The GIL comes with great convenience as you don't have to worry about a whole host of data races. This is true, but it doesn't mean that a GIL-less Python would need to have an option for "enable GIL so I don't have to worry about data races". It means that a GIL-less Python would have to ensure that there are no data races, without having to have a GIL. > it will be a great amount of work to make every library compatible with GILless python No, it won't; libraries won't have to change at all. The current interpreter makes a guarantee that those libraries rely on: "you don't have to worry about data races". A GIL-less interpreter would have to make the same guarantee; it just wouldn't have to have a GIL to do it. That requirement is what makes a GIL-less Python interpreter hard to do.
- std_throwawayay 8y agoRight now you have one mutex for everything (the GIL itself) and everything else doesn't need locking. In order to achieve similar convenience without the GIL you would have to trade this for one mutex for every single data structure. Because the data structures in python are so flexible, every single variable needs its own mutex then. Locking every single variable access would be enormously costly. Other languages achieve a good compromise by clustering data structures into fewer parts with only a handful of mutexes that are locked while other threads work on different datasets. This is usually done manually and with great care as it is the heart of both safety and performance. I don't know if there is an automatic solution to this problem that is compatible with the way python programs are structured. The libraries basically assume that, while you call them, nothing else changes. In order to ensure that you need to lock everything down. Because you don't know what these libraries do and what data they access it needs to be everything (like it is today). It should be possible to only lock the GIL when such a library is called, so there should be kind of a middle way forward.
- antpls 8y ago> This is wonderful. This could become the best way to move Python projects to Rust: initially just run on the RustPython interpreter, but then optimize low level routines in Rust. In 15 years I wouldn't be surprised if this or something like it surpasses CPython in popularity. What you are describing is simply a JIT compiler. Maybe are you suggesting to rewrite PyPy (its C part) in Rust?
- hathawsh 8y agoActually I was referring to manual translation of Python code to Rust. My experience with PyPy has been rather bad, unfortunately (it runs my code at less than half the speed of CPython) and I figure human translation should be a lot more effective than JIT optimization.
- antpls 8y agoMaybe you should send your code to the PyPy team. I bet they will look into it and see how they could improve their JIT compiler. Last time I checked the project, they had a full benchmark running for each release to spot regressions and to track improvements
- nickpsecurity 8y ago"What you are describing is simply a JIT compiler." The user is describing the opposite of a JIT compiler: a gradual rewrite of Python apps in Rust to feed into an ahead-of-time, highly-optimizing compiler. A JIT compiler would do quick compiles of Python code while it's running. The performance, reliability, and security capabilities of JIT vs AOT vary considerably with context. For predictability and security, I avoid JIT's wherever possible in favor of AOT's.
- loeg 8y agoIt's basically impossible to implement a CPython-compatible language without a GIL (or you lose single thread performance by using very fine-grained atomics/locking). Python has very specific multithreading semantics that are a function of the CPython bytecode and the GIL, and programs rely on this.
- int_19h 8y agoOther than refcounting (which is not a part of the Python language spec - it even specifically says that conforming code shouldn't rely on it), what other semantics did you have in mind?
- sametmax 8y agoYou can list.append from 2 threads without worrying about crashing, although I would not recommend it. It's possible to do without the GIL, but up to now, it's been a damn to way of doing that.
- int_19h 8y agoI don't think that's guaranteed, though.
- jlokier 8y agoloeg says, "programs rely on this". Guarantee or not, it constrains whether something is usable as a drop-in replacement interpreter, especially if people can't tell which programs will break, and doubly so if the breakage is a subtle data corruption race that doesn't show up in tests.
- tanilama 8y agoIf we are ever going to get rid of GIL, we need to get rid of Python's C extensions all together. It is not a real Python implementation if not compatible with C extension, it is just embedded DSL that has Python flavor syntax.
- bakery2k 8y agoCompatibility with C extensions seems to be the most difficult thing for an alternative implementation to achieve. PyPy struggled with this for a long time, and IIRC extension compatibility also caused the failure of Dropbox's Pyston. Is there really no situation in which an alternative implementation that only supports "pure Python" would be useful?
- tanilama 8y ago> Is there really no situation in which an alternative implementation that only supports "pure Python" would be useful? This really depends on your definition of 'being useful'. Jython is useful in a sense, it is being used in many Big Data solutions as a way to embed Python as DSL/UDF, like Pig/Hive, etc. However, if without support for C extensions, it is not really a Python implementation, in a sense, I can't run a python script I just gripped from internet using the so-called 'alternative' implementation. So if the point of being useful is to be a replacement, then sadly, the answer is no, it is an everything-or-nothing situation.
- pjmlp 8y agoIn the old days, before Eclipse had a scratchpad area or Java finally got a shell, it was useful to me as means to explore some Java APIs, as I did not want to spend time with either Beanshell or Grovvy since I know Python superficially since 1.6.
- sametmax 8y agoIndeed, I'm very excited about this. Rust + Python seems a natural combinaison to me, and being able to have one single dev env (and maybe in the end, one single deployment mechanism) to do both is a killer feature. And actually, I think having Python written in Rust would provide some other very nice properties: - limit the number of bug you can introduce in the implementation because of the rust safety nets; - can still expose a C compatible ABI and hence be compatible with existing extensions; - the rust toolchain being awesome, it may inspire people to make it easy to compile a python program. Right now I use nuikta, which is great, but has to convert to C then compile, which make it a complex toolchain.
- woolvalley 8y agoI would personally try to move away from python at this point for greenfield projects. The GIL is so baked into the language, if you removed it a bunch of current python code will probably break in subtle ways. Modern languages need proper multithreading support, static types and fast compile speeds. Use golang, use kotlin, use dart, use anything but python & javascript.
- sametmax 8y agoI want it to have the GIL, because I want it to maintain compat with C extensions. We already have a plan to bypass the GIL: multi interpreters. Having an implementation in Rust may make future improvement to Python easier, so it's better to have something exactly similar first, then start to hack it.
- FartyMcFarter 8y agoBy multi interpreters do you mean several interpreter processes without shared memory? (or only limited memory sharing) CPython can do that too, but this isn't really multi-threading, and it only bypasses the GIL in a very trivial sense. But yeah, keeping the GIL is probably the only reasonable way to go if you want compatibility with existing extensions.
- deleted 8y ago[deleted]
- sametmax 8y agoYes. Actually you get one gil per interpretter, and can spawn your thread in sub interpretter. Cpython could do it, but currently can't provide much since the api to do it is only accessible from c.
- FartyMcFarter 8y agoDoesn't CPython do this with the multiprocessing module? https://docs.python.org/2/library/multiprocessing.html https://docs.python.org/2/library/multiprocessing.html
- sametmax 8y agoNo, this spawns several process, while sub interpretters allow you to have several of them in one single process, making data sharing much cheaper.