8 ms·

Now this is something I will donate money to… Python with Rust as its foundation sounds like the best idea ever. I’m curious to know whether or not it would b

Now this is something I will donate money to… Python with Rust as its foundation sounds like the best idea ever. I’m curious to know whether or not it would be possible (or reasonable) to eventually get the same or better performance as CPython.

Waterluvian 8y ago
CPython is really quite slow by design. The reference implementation is meant to be obvious and pretty easy to interoperate with C. I think a RustPython implementation would be pretty cool. You could definitely take that opportunity to worry about performance more than CPython does while also worrying about interoperability more than PyPy does. Or I'm missing your point and you're suggesting a drop-in replacement for CPython that supports all the same C-based libraries as CPython does.
- gonational 8y ago
  Awesome! No, that was my point basically, although I could imagine something like this shipping with some basic libraries and package support, and then having a similar ecosystem to the current python ecosystem.
- lykr0n 8y ago
  From my limited experience with Rust, having support for C-based libraries shouldn't be too hard to do- as Rust (like Go) has support for embedding C. I'm not sure how portable the solution would be- I would guess you would need to compile the VM with the modules you want baked in. I think not having support for C modules would hamper long term adoption. I would absolutely love to adopt this for my stuff, but off the top of my head- I use uvloop and confluent-Kafka, both of which are largely written in in C. Moving away from those would be hard-ish.
  - weberc2 8y ago
    Calling into and out of C (whether dynamically or statically linked) isn't the hard part; the hard part is implementing the somewhat arbitrary interface against which C extensions are written in a way that is functionally compatible with CPython without losing the advantages of not being CPython.
- Twirrim 8y ago
  PyPy is doing really well on compatibility these days. It can use all the C libraries natively, and a lot of the speed issues there have been resolved.
  - chrisseaton 8y ago
    > It can use all the C libraries natively Literally all of them, without any issues? I'm working on implementing support for Ruby's C extensions in an alternative implementation and it's a right slog.
    - mattip 8y ago
      PyPy dev here. Yes to both - all of them and it was a right slog. If it doesn’t work report an issue. We have a proposal looking for funding to make it fast.
      - Waterluvian 8y ago
        This is wonderful news to me. So broadly speaking I should be able to drop in replace cpython with pypy for my fairly not so special projects? I have a Django app that does some heavy data serialization and I'm not yet ready to optimize those serializers in another language. I can't wait to try this out. Crumb. I didn't realise pypy is on 3.5.3. Loves me my f strings.
        orangecat 8y ago
        f-strings were "backported" to pypy and work fine.
        mattip 8y ago
        We backported fstrings to 3.5. We also have an alpha quality linux 3.6 available on nightly builds
        Waterluvian 8y ago
- std_throwawayay 8y ago
  > CPython is really quite slow by design. The data structures are slow by design. The way we use these data structures is quite inefficient. Having a fast interpreter only about doubles the execution speed in most programs leaving another factor of 50 open for future generations.
  - sametmax 8y ago
    Really ? I though Python dicts and lists were actually quite fast for a dynamic language. Ints and classes can be slow though. Anything I don't know ?
    - std_throwawayay 8y ago
      They are probably the fastest dicts and lists we can imagine today. If you really need lists and dicts for your data, you need them and won't gain much. But if you don't need them and instead could use something simpler, an array of C structs would be much faster than a list of dicts.
      - sametmax 8y ago
        I though you were stating that Pythn were slow compared to what it could be, being a dynamic language. Of course if you compare to static languages it's slow. Of course you can write low level specialized DS. Duh.
Zelmor 8y ago
>Python with Rust as its foundation sounds like the best idea ever stop putting lead in your drinking water
- joshmarlow 8y ago
  I'll bite - why do you believe this is a bad idea?
  - coldtea 8y ago
    He doesn't. He just thinks it's a stupid idea. There's a mature 30 year old codebase, with huge industry update and tons of major players using it, and some random project to rewrite it in a new fashionable language, that will more likely than not go nowhere (as 99% of such attempts do) and be abandoned when the committers lose interest and recognize how big the full task is.
    - Zelmor 8y ago
      I'd send you a bottle of wine for saving me the time and effort.
johnisgood 8y ago
I'm curious to know that why isn't it the case right now? Isn't that one of the selling points of Rust, being blazing fast? Or perhaps just this particular piece of software has been implemented poorly?
- pdpi 8y ago
  Even if RustPython is faster at the architectural level (which it probably does, CPython is infamously slow), CPython has the advantage of a mature codebase and two and a half decades of accumulated performance tuning.
  - coldtea 8y ago
    >and two and a half decades of accumulated performance tuning. Or two and a half decades of performance neglect. I've read about things like dicts, sort and the like getting faster implementations, but I've never seen a big effort to make CPython faster in general. In fact the first versions of 3.x were even allowed to regress to slower than 2.x.
    - FartyMcFarter 8y ago
      This is certainly the case when it comes to startup time (an important thing in a scripting language). On my linux system, python2 starts in about 10ms, python3 takes 20-25 ms.
- klodolph 8y ago
  That’s a false dilemma. We’ve had almost three decades to optimize CPython. It will take a while to catch up.
  - johnisgood 8y ago
    It's not, and I'm curious as to why, I even suggested a reason very similar to your answer. I guess people assumed malice. Oh well.
grok2 8y ago
> Python with Rust as its foundation sounds like the best idea ever. Why do you think it's the "best idea ever"? What are the benefits over any other Python implementation?
- nickpsecurity 8y ago
  Python has massive use in a diverse array of fields with lots of educational resources helping beginners. There is and will be for a while lots of Python code. It's increasingly used in business-critical systems, like at Bank of America. That it's built on an unsafe language puts that all at risk of unreliability and security vulnerabilities. Long ago, I wanted to rewrite it in Ada with all safety features on to mitigate that risk while maintaining its convenience. Rust is another safe, systems language with extra measures for temporal errors. So, this project is doing the same thing. Full security for Python apps would require consideration of each layer of abstraction: 1. User's code in Python. 2. The interpreter and extensions. 3. How these interact. 4. If added for performance or security, any assembly code plus its interactions. Rewriting Python interpreter in Rust mainly addresses No 2. An example of a method to address all of them would be Abstract, State Machines which can represent simultaneously language semantics, software, and hardware. Tools like Asmeta exist to make them like programming languages. The verification would probably be manual, specialist work. Whereas, Rust's compiler gives you key properties with just annotations for many and working with borrow-checker for a few.
  - deleted 8y ago
    [deleted]
  - dnautics 8y ago
    > That it's built on an unsafe language puts that all at risk of unreliability and security vulnerabilities Has this actually been a problem, though? I'm no lover of python, but tons of people seem to use, for example, Django, without incident.
    - sametmax 8y ago
      Now but you need an incredible high level of mastery to work on such a big C project so widely deployed. Rust will allow to safely invite a broader range of contributors, because there are so many things you don't need to check. This also means a smaller number of required tests, and because Rust uses higher level constructs that C, more productivity in general. So basically, on the long run, more people, able to do more things. Besides, on of the goals of the main implementation is to stay simple, which is hard to do in C. For those reasons, and because of the potential for unreliability and security, CPython is quite slow. We can't optimize it, because it would make it too complex. But with a rust implementation, one can hope to suddenly be able to apply more optimizations. It's all theorical of course, but it's a nice hope.
deleted 8y ago
[deleted]

Waterluvian 8y ago

CPython is really quite slow by design. The reference implementation is meant to be obvious and pretty easy to interoperate with C. I think a RustPython implementation would be pretty cool. You could definitely take that opportunity to worry about performance more than CPython does while also worrying about interoperability more than PyPy does. Or I'm missing your point and you're suggesting a drop-in replacement for CPython that supports all the same C-based libraries as CPython does.

gonational 8y ago
Awesome! No, that was my point basically, although I could imagine something like this shipping with some basic libraries and package support, and then having a similar ecosystem to the current python ecosystem.
lykr0n 8y ago
From my limited experience with Rust, having support for C-based libraries shouldn't be too hard to do- as Rust (like Go) has support for embedding C. I'm not sure how portable the solution would be- I would guess you would need to compile the VM with the modules you want baked in. I think not having support for C modules would hamper long term adoption. I would absolutely love to adopt this for my stuff, but off the top of my head- I use uvloop and confluent-Kafka, both of which are largely written in in C. Moving away from those would be hard-ish.
- weberc2 8y ago
  Calling into and out of C (whether dynamically or statically linked) isn't the hard part; the hard part is implementing the somewhat arbitrary interface against which C extensions are written in a way that is functionally compatible with CPython without losing the advantages of not being CPython.
Twirrim 8y ago
PyPy is doing really well on compatibility these days. It can use all the C libraries natively, and a lot of the speed issues there have been resolved.
- chrisseaton 8y ago
  > It can use all the C libraries natively Literally all of them, without any issues? I'm working on implementing support for Ruby's C extensions in an alternative implementation and it's a right slog.
  - mattip 8y ago
    PyPy dev here. Yes to both - all of them and it was a right slog. If it doesn’t work report an issue. We have a proposal looking for funding to make it fast.
    - Waterluvian 8y ago
      This is wonderful news to me. So broadly speaking I should be able to drop in replace cpython with pypy for my fairly not so special projects? I have a Django app that does some heavy data serialization and I'm not yet ready to optimize those serializers in another language. I can't wait to try this out. Crumb. I didn't realise pypy is on 3.5.3. Loves me my f strings.
      - orangecat 8y ago
        f-strings were "backported" to pypy and work fine.
      - mattip 8y ago
        We backported fstrings to 3.5. We also have an alpha quality linux 3.6 available on nightly builds
        Waterluvian 8y ago
std_throwawayay 8y ago
> CPython is really quite slow by design. The data structures are slow by design. The way we use these data structures is quite inefficient. Having a fast interpreter only about doubles the execution speed in most programs leaving another factor of 50 open for future generations.
- sametmax 8y ago
  Really ? I though Python dicts and lists were actually quite fast for a dynamic language. Ints and classes can be slow though. Anything I don't know ?
  - std_throwawayay 8y ago
    They are probably the fastest dicts and lists we can imagine today. If you really need lists and dicts for your data, you need them and won't gain much. But if you don't need them and instead could use something simpler, an array of C structs would be much faster than a list of dicts.
    - sametmax 8y ago
      I though you were stating that Pythn were slow compared to what it could be, being a dynamic language. Of course if you compare to static languages it's slow. Of course you can write low level specialized DS. Duh.

Zelmor 8y ago

>Python with Rust as its foundation sounds like the best idea ever stop putting lead in your drinking water

joshmarlow 8y ago
I'll bite - why do you believe this is a bad idea?
- coldtea 8y ago
  He doesn't. He just thinks it's a stupid idea. There's a mature 30 year old codebase, with huge industry update and tons of major players using it, and some random project to rewrite it in a new fashionable language, that will more likely than not go nowhere (as 99% of such attempts do) and be abandoned when the committers lose interest and recognize how big the full task is.
  - Zelmor 8y ago
    I'd send you a bottle of wine for saving me the time and effort.

johnisgood 8y ago

I'm curious to know that why isn't it the case right now? Isn't that one of the selling points of Rust, being blazing fast? Or perhaps just this particular piece of software has been implemented poorly?

pdpi 8y ago
Even if RustPython is faster at the architectural level (which it probably does, CPython is infamously slow), CPython has the advantage of a mature codebase and two and a half decades of accumulated performance tuning.
- coldtea 8y ago
  >and two and a half decades of accumulated performance tuning. Or two and a half decades of performance neglect. I've read about things like dicts, sort and the like getting faster implementations, but I've never seen a big effort to make CPython faster in general. In fact the first versions of 3.x were even allowed to regress to slower than 2.x.
  - FartyMcFarter 8y ago
    This is certainly the case when it comes to startup time (an important thing in a scripting language). On my linux system, python2 starts in about 10ms, python3 takes 20-25 ms.
klodolph 8y ago
That’s a false dilemma. We’ve had almost three decades to optimize CPython. It will take a while to catch up.
- johnisgood 8y ago
  It's not, and I'm curious as to why, I even suggested a reason very similar to your answer. I guess people assumed malice. Oh well.

grok2 8y ago

> Python with Rust as its foundation sounds like the best idea ever. Why do you think it's the "best idea ever"? What are the benefits over any other Python implementation?

nickpsecurity 8y ago
Python has massive use in a diverse array of fields with lots of educational resources helping beginners. There is and will be for a while lots of Python code. It's increasingly used in business-critical systems, like at Bank of America. That it's built on an unsafe language puts that all at risk of unreliability and security vulnerabilities. Long ago, I wanted to rewrite it in Ada with all safety features on to mitigate that risk while maintaining its convenience. Rust is another safe, systems language with extra measures for temporal errors. So, this project is doing the same thing. Full security for Python apps would require consideration of each layer of abstraction: 1. User's code in Python. 2. The interpreter and extensions. 3. How these interact. 4. If added for performance or security, any assembly code plus its interactions. Rewriting Python interpreter in Rust mainly addresses No 2. An example of a method to address all of them would be Abstract, State Machines which can represent simultaneously language semantics, software, and hardware. Tools like Asmeta exist to make them like programming languages. The verification would probably be manual, specialist work. Whereas, Rust's compiler gives you key properties with just annotations for many and working with borrow-checker for a few.
- deleted 8y ago
  [deleted]
- dnautics 8y ago
  > That it's built on an unsafe language puts that all at risk of unreliability and security vulnerabilities Has this actually been a problem, though? I'm no lover of python, but tons of people seem to use, for example, Django, without incident.
  - sametmax 8y ago
    Now but you need an incredible high level of mastery to work on such a big C project so widely deployed. Rust will allow to safely invite a broader range of contributors, because there are so many things you don't need to check. This also means a smaller number of required tests, and because Rust uses higher level constructs that C, more productivity in general. So basically, on the long run, more people, able to do more things. Besides, on of the goals of the main implementation is to stay simple, which is hard to do in C. For those reasons, and because of the potential for unreliability and security, CPython is quite slow. We can't optimize it, because it would make it too complex. But with a rust implementation, one can hope to suddenly be able to apply more optimizations. It's all theorical of course, but it's a nice hope.

deleted 8y ago

[deleted]