4 ms·

The problem is with CPU intensive tasks. Being as expressive as Python is, it would be nice if CPU intensive algorithms could be written in it, without blockin

by saghul 9y ago

The problem is with CPU intensive tasks. Being as expressive as Python is, it would be nice if CPU intensive algorithms could be written in it, without blocking all other Python threads while the algorithm runs. For example.

crdoconnor 9y ago
>it would be nice if CPU intensive algorithms could be written in it It would be nicer if instead of trying to turn python into something it is not (a language for writing CPU intensive tasks), it was made better at what it is (glue code). It would be especially nice if there were tighter, more seamless integration between python and rust so that hot paths in python can easily be migrated to rust.
- mjburgess 9y ago
  No to threading, but yes to tighter integration with rust. Ah yes, let's not focus on something python is not. Let's appeal to the majority use case. What minor planet are people here from?
- Walkman 9y ago
  Python is far more than glue code.
- sevensor 9y ago
  I'm not a Rust guy, so take this with a grain of salt, but I was under the impression that Rust did a great job with C interoperability, which is all you need for writing Python extensions. Some hasty googling brings up https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for. In other words, it's not heavily advertised, but it appears to be entirely feasible to build Python extensions in Rust, and not spectacularly complicated either.
  - crdoconnor 9y ago
    >Some hasty googling brings up https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for. Yeah, I've tried it and it didn't work. It's certainly feasible it's just not very well developed.
  - steveklabnik 9y ago
    The language does do great at the core of it, but there's a lot of stuff that makes it easier that's not the language's job. compare https://doc.rust-lang.org/1.2.0/book/rust-inside-other-languages.html#ruby https://doc.rust-lang.org/1.2.0/book/rust-inside-other-langu... to http://usehelix.com/ http://usehelix.com/ for example; the former is only using what the language knows, the latter is a nice library built on top. There's been some effort for nicer Python integration, but I haven't seen anything super super slick yet.
xorcist 9y ago
If it's really CPU intensive, you can always fork.
- srean 9y ago
  The problem is the overhead of memory copy. Linux is somewhat smart about it with its copy on write semantics (a double edged sword really). However, the moment you refer to an existing Python object the underlying data-structure gets mutated because of reference counting and boom now you have to copy the whole thing. This can be worked around with creative stealing and by keeping a sharp lookout for references but it gets really painful and annoying. Even if that were resolved its still not as flexible as one would want. One cant really go about spawning and terminating process too frequently. This forces one to move larger and larger parts of the logic into C.

by saghul 9y ago

crdoconnor 9y ago

>it would be nice if CPU intensive algorithms could be written in it It would be nicer if instead of trying to turn python into something it is not (a language for writing CPU intensive tasks), it was made better at what it is (glue code). It would be especially nice if there were tighter, more seamless integration between python and rust so that hot paths in python can easily be migrated to rust.

mjburgess 9y ago
No to threading, but yes to tighter integration with rust. Ah yes, let's not focus on something python is not. Let's appeal to the majority use case. What minor planet are people here from?
Walkman 9y ago
Python is far more than glue code.
sevensor 9y ago
I'm not a Rust guy, so take this with a grain of salt, but I was under the impression that Rust did a great job with C interoperability, which is all you need for writing Python extensions. Some hasty googling brings up https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for. In other words, it's not heavily advertised, but it appears to be entirely feasible to build Python extensions in Rust, and not spectacularly complicated either.
- crdoconnor 9y ago
  >Some hasty googling brings up https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for. Yeah, I've tried it and it didn't work. It's certainly feasible it's just not very well developed.
- steveklabnik 9y ago
  The language does do great at the core of it, but there's a lot of stuff that makes it easier that's not the language's job. compare https://doc.rust-lang.org/1.2.0/book/rust-inside-other-languages.html#ruby https://doc.rust-lang.org/1.2.0/book/rust-inside-other-langu... to http://usehelix.com/ http://usehelix.com/ for example; the former is only using what the language knows, the latter is a nice library built on top. There's been some effort for nicer Python integration, but I haven't seen anything super super slick yet.

xorcist 9y ago

If it's really CPU intensive, you can always fork.

srean 9y ago
The problem is the overhead of memory copy. Linux is somewhat smart about it with its copy on write semantics (a double edged sword really). However, the moment you refer to an existing Python object the underlying data-structure gets mutated because of reference counting and boom now you have to copy the whole thing. This can be worked around with creative stealing and by keeping a sharp lookout for references but it gets really painful and annoying. Even if that were resolved its still not as flexible as one would want. One cant really go about spawning and terminating process too frequently. This forces one to move larger and larger parts of the logic into C.