4 ms·
Why call python from Rust? I think calling rust from python would make more sense. Use it to optimize python functions like how python libraries do with with C
by hypewatch 6y ago
Why call python from Rust? I think calling rust from python would make more sense. Use it to optimize python functions like how python libraries do with with C (i.e. numpy)
- FridgeSeal 6y agoSuppose I’m building an application in Rust that processes a lot of data, one of the steps in processing that data I maybe want to run through a Python tool like SpaCy or Flair. How would I go about doing that? I could put the Python code behind a little http API and call it that way, but that’s a bunch of overhead and extra stuff to maintain just to analyse some text. If I embed said Python tools in my Rust code then I can call those tools with significantly less overhead and complexity.
- kccqzy 6y agoWhy does it have to have a little http API? Why not just spawn the python tool?
- staticassertion 6y agoOk, at that point, why not just write the Python directly in your Rust application?
- dbsmith83 6y agoBecause the code smell reeks? You've greatly increased complexity. Did you read about debugging the whitespace? What a mess.
- FridgeSeal 6y agoAnd communicate with it how? If you’re suggesting spawning things over shell/cmd line I’m of the opinion that this a generally bad idea.
- sfkdjf9j3j 6y agoWhat's wrong with forking and using a pipe or socket?
- kccqzy 6y agoJust fork and exec. Or if you fancy, posix_spawn a separate process. Then communicate over pipes, or if you have a lot of outputs from these tools, collect results from the file system. Really, where does this fear of spawning things come from?
- jononor 6y agoPython module loading tends to be quite slow. Loading Tensorflow on our Heroku production environment takes 10 seconds (5 on my laptop). A client of mine running Rust and Python on Embedded Linux device found that loading numpy/pandas modules took over 5 seconds (laptop was under 1), and their computation took just x00 milliseconds. 10x overhead... So the spawning of a short-lived subprocess approach has massive overhead, only suitable for multi-second workloads.
- umanwizard 6y agoSpawning an external binary does not necessarily involve the shell (if it did, how could the shell itself spawn things? It’d be shells all the way down)
- zetalemur 6y ago> And communicate with it how? The simplest approach is via a pipe to the process' stdin/stdout I guess. Of course you have to (de)serialize your data, but you would have to to the same if you go the HTTP route, which seems far more complex. Furthermore the suggested solution probably has a nice wrapper in the language's stdlib (e.g. `check_output()` in Python land).