11 ms·
Writing Python inside Rust
- ptato 6y agoWhat is the use case for embedding Python code in a Rust program?
- djohnston 6y agoMaybe interfacing w an ml lib like pytorch or something?
- RcouF1uZ4gsC 6y agoFor pytorch it would probably be easier to do something based off the pytorch C++ interface.
- djohnston 6y agoSure, I've never even used pytorch. But I'm just saying python has a fair bit of data analysis and ml tools that probably haven't found native homes in rust
- qchris 6y agoI suppose you could, but I believe that there are direct bindings from Rust to Pytorch at https://github.com/LaurentMazare/tch-rs https://github.com/LaurentMazare/tch-rs. I haven't used it personally, but I've heard fairly good things about it's correctness and responsive maintainer.
- danieldk 6y agoI have written a transformer-based (BERT, XLM-RoBERTa) sequence labeler/lemmatizer/dependency parser in Rust with tch-rs [1] and it is great! Some parts do not feel rusty. E.g. dynamic typing of tensors, you get a Tensor rather than Tensor<f32>, but that's more due to how libtorch itself works. But it's a very straightforward API that exposes most of libtorch. The maintainer is also very nice and responsive. I was also amazed how much is implemented in libtorch itself as opposed to the Python wrapper, which makes much of the Torch functionality available to other languages. [1] https://github.com/stickeritis/sticker2 https://github.com/stickeritis/sticker2 https://github.com/stickeritis/sticker-transformers https://github.com/stickeritis/sticker-transformers
- random32840 6y agoSomeone might have jumped that hurdle but the point is it circumvents that class of hurdle. What about the next framework which doesn't have bindings? What about leveraging existing Pytorch code that's written in Python?
- amanzi 6y agoI only had a quick scan through the article, but could this be used to create an executable for some existing Python code?
- choward 6y agoWhat would the point of that be versus wrapping it in a bash script? Obfuscation?
- lmm 6y agoPython is missing a nice simple way to ship native executables. If you have to send people a package to unpack that's an extra step. In theory you can use freeze, but configuring it all is a pain.
- trishmapow2 6y agoI've only used it for small GUIs (a couple of files & dependencies) but `pyinstaller --onefile` is about as easy as it gets.
- londt8 6y agoYou can use pip to install the executable source files. Pip is preinstalled on most Libux distributions and MacOS.
- choward 6y agoThat's exactly what I thought when I read the title. It's very frustrating how many landing pages, project readmes, blog posts, etc that don't answer the question "why?". I usually need to know that before going further when I come across something that I've never heard of before. If they don't have the "why?" somewhere easy to find I just close the tab.
- mehrdadn 6y agoYou know, that might not be unintentional. They might want those who don't see the value to just close the tab and move on. That way only those who see the value keep reading. I imagine it helps avoid having to engage in (often endless) arguments about the validity of the use case.
- deleted 6y ago[deleted]
- elpatoisthebest 6y agoIt's a bit of a guilty pleasure for me to read these kinds of weird things with no practical application. It's just a little refreshing to read about something technically possibly, but frivolous (no offense meant to the OP, of course) just because it can be done.
- owl57 6y ago"Because it can be done" is a very solid answer to "Why?"
- aspenmayer 6y agoIt helps me to step outside my own context and view a familiar use case, problem, or solution with an unfamiliar perspective.
- brainless 6y agoI am quite surprised by your message. This is HN and even before or outside of that, for years I have myself dived into totally random projects simply because "why not?". To try random ideas is part of learning and when you have something to share, just do that. There will always be like minded someone to pick up.
- granzymes 6y agoIf imports worked I could see it being very useful for making graphs. I've heard of people serializing data in other languages and then using Python to plot it.
- qchris 6y agoI can confirm this--Rust's data visualization and plotting libraries are still pretty early stage. You can generally do 2D bar charts and scatter plots pretty well, but once you start jumping into more complex representations or three dimensions, you start running into some serious barriers.
- m-ou-se 6y agoImports work fine. The main reason I wrote this, was to be able to use the Python matplotlib library in Rust: https://twitter.com/m_ou_se/status/1120577172438233088 https://twitter.com/m_ou_se/status/1120577172438233088
- memco 6y agoI’m just throwing out ideas but what if I wanted to take working python code and convert it to rust? Could I use this to start as the baseline and start replacing bits of it and see that it still produces the same output?
- zerkten 6y agoThere was a recent ATP.fm podcast with Chris Lattner of Swift fame. He talked about the topic of embedding Python code in Swift for TensorFlow and it overlaps with your suggestion.
- NobodyNada 6y agoSwift for Tensorflow has implemented Python interoperability, rather than embedded Python code as in this article. A Swift program can import a Python module and interact with Python objects and functions as if they were native Swift objects (minus strong typing, of course). See https://github.com/tensorflow/swift/blob/master/docs/PythonInteroperability.md https://github.com/tensorflow/swift/blob/master/docs/PythonI...
- akiselev 6y agoRust has had Python interop through the Python C API since around 2015 [1]. It's pretty low hanging fruit for any language that supports the C FFI. Rust-cpython has had simple macros for interop for ages [2] and there's even a library that uses serde macros to encode/decode pickled object [3]. This article is just a fun hack using Rust macros. [1] https://github.com/dgrunwald/rust-cpython https://github.com/dgrunwald/rust-cpython [2] http://dgrunwald.github.io/rust-cpython/doc/cpython/macro.py_class.html http://dgrunwald.github.io/rust-cpython/doc/cpython/macro.py... [3] https://docs.rs/serde-pickle/0.6.0/serde_pickle/ https://docs.rs/serde-pickle/0.6.0/serde_pickle/
- deleted 6y ago[deleted]
- rtpg 6y agoPlugins for example? Game scripts? Think of use cases for embedding Lua into a C program.
- charliesome 6y agoThere could be no reasonable use case for something like this, and yet it would still have artistic value and this would still be an interesting article. It's a brilliant hack and we are on Hacker News after all.
- deleted 6y ago[deleted]
- tedmielczarek 6y agoA friend of mine has a similar project called PyOxidizer that's intended for building standalone executables with Python+Rust: https://pyoxidizer.readthedocs.io/en/stable/ https://pyoxidizer.readthedocs.io/en/stable/ His primary use for it is building distributable binaries for Mercurial, which is written primarily in Python.
- hypewatch 6y agoWhy call python from Rust? I think calling rust from python would make more sense. Use it to optimize python functions like how python libraries do with with C (i.e. numpy)
- FridgeSeal 6y agoSuppose I’m building an application in Rust that processes a lot of data, one of the steps in processing that data I maybe want to run through a Python tool like SpaCy or Flair. How would I go about doing that? I could put the Python code behind a little http API and call it that way, but that’s a bunch of overhead and extra stuff to maintain just to analyse some text. If I embed said Python tools in my Rust code then I can call those tools with significantly less overhead and complexity.
- kccqzy 6y agoWhy does it have to have a little http API? Why not just spawn the python tool?
- staticassertion 6y agoOk, at that point, why not just write the Python directly in your Rust application?
- dbsmith83 6y agoBecause the code smell reeks? You've greatly increased complexity. Did you read about debugging the whitespace? What a mess.
- FridgeSeal 6y agoAnd communicate with it how? If you’re suggesting spawning things over shell/cmd line I’m of the opinion that this a generally bad idea.
- 6y ago
- amrx101 6y agoSo I am not the only one with questions regarding its use case? I cant seem to conjure any.
- random32840 6y agoQuickly introducing battle-tested data science frameworks, snippets and patterns without having to replicate them in Rust.
- deleted 6y ago[deleted]
- superdisk 6y agoThere's a typo where stringify! is referred to as strinfigy! Cool article btw!
- m-ou-se 6y agoThanks! Updated.
- bsder 6y agoAm I the only person having trouble reading this blog? That color scheme makes my eyes bleed. It seems like it's a light gray background with a slightly (not much) darker gray text. The contrast and thin font weight is terrible. And the pink is practically vibrating on the page. At least the blue is okay.
- marvy 6y agoSo, at first I thought this approach is doomed, because you lose comments, and what looks like a comment to Rust may not look like one to Python. Example: x = y // 2 # floor division But then I decided to look at the docs: https://doc.rust-lang.org/proc_macro/struct.Span.html https://doc.rust-lang.org/proc_macro/struct.Span.html And I noticed source_text, which "preserves the original source code, including spaces and comments"!!! Why not just use this from the start then?? Seems like the easy way out, no? (Disclaimer: I don't know Rust, can't even write hello world.)
- uranusjr 6y agoThe author likely just barely missed its introduction. While the article is written recently, the implementation it talks about was published first in early April 2019, right about when source_text was first introduced into nightly.
- ogoffart 6y agoI am the one who contributed Span::source_text to rust. My motivation for doing that was exactly the same as the author of the blog post, and was to help implementing the cpp! macro which embeds C++ within rust just like this python! macro. https://docs.rs/cpp/ https://docs.rs/cpp/ However, I still can't make use of it today because it is not stable yet, and even proc_macro2 does not implement it (and this is not trivial to implement there)
- marvy 6y agoWow. The fact that you care about this being stable implies that you plan to use this "for real"; the fact that you actually implemented this implies you care enough to put in some real work. So you must have a use case in mind, and I am not imaginative enough to guess what it might be. Is it just about convenience so you don't need to bother with putting your C++ code in a different file? Or is there more to it?
- ogoffart 6y ago
- ComputerGuru 6y agoI wrote this [0] up a couple of weeks ago, which is a hack to put shell code in your rust, with reasons why you would want to. [0]: https://neosmart.net/blog/2020/self-compiling-rust-code/ https://neosmart.net/blog/2020/self-compiling-rust-code/
- oon 6y agoUsing Span to get the token location is exactly what I needed! Thanks. I wrote a blog post[0] the other day about making a css macro that compiles into rust for the use in Yew, a front end react like framework. > However, in rust, there’s no way to differentiate .a.b with .a .b Now I know that the above is incorrect. I would have never thought of spans so I thank you again [0] https://conradludgate.com/posts/yew-css/#what-are-the-downsides-of-yew https://conradludgate.com/posts/yew-css/#what-are-the-downsi...
- 6gvONxR4sf7o 6y agoI know this project may just be for fun, but with WASM targets for all sorts of languages, I'm hoping we get to a future where mixing and matching different languages for different parts of your program will be seamless. Imagine starting a project in an easy language, then migrating pieces to a faster "bare metal" language as needed in a super piecemeal way. Same with moving pieces to a safer language as the project grows, slowly expanding the boundaries of the safe bits as appropriate.
- pjmlp 6y agoJust like the JVM and CLR I guess. https://en.wikipedia.org/wiki/List_of_JVM_languages https://en.wikipedia.org/wiki/List_of_JVM_languages https://en.wikipedia.org/wiki/List_of_CLI_languages https://en.wikipedia.org/wiki/List_of_CLI_languages https://www.graalvm.org/docs/ https://www.graalvm.org/docs/
- 6gvONxR4sf7o 6y agoSame idea, yeah, but hopefully with better common adoption. For various reasons users didn't adopt those (e.g. numpy on JVM would be a ton of effort, especially a decade ago). But the web is an irresistable force, maybe. I'm hopeful (though not necessarily optimistic).
- pjmlp 6y agoBut they did, just not with the languages that are loved on HN as "taking over the world". Most people on JVM land, use a mix of Java, Kotlin, Scala, Clojure, Groovy, JRuby. Whereas on CLR land it is C#, F#, VB.NET and C++/CLI (for low level stuff). Naturally those that want to kill Java, or rather not touch Windows, aren't aware of this.
- tasogare 6y agoI know you know this, but I’ll write it for other readers: there is even more languages on CLR if we count the one not supported by MS. Also, C native libraries and thus any language that can target it, can by used as well on Windows, Mac OS and Linux with PInvoke.
- pansa2 6y agoIs it common to embed one language's code directly inside another language like this? Lua is often used tightly-coupled to C, and it doesn't have the significant-whitespace issues that Python shows here. Even so, I've only ever seen it used in separate `.lua` files, never embedded directly within `.c` files.
- iddan 6y agoI don’t think it’s common nor it’s a best practice as you can’t apply static analysis on your embedded code. But it’s really cool Rust allows that and it can be a ad-hoc solution when you need to write a piece of code fast inside of rust
- kzrdude 6y agoLua is a better fit for Rust, because it doesn't have the threading restrictions of (C)Python. With Lua you can run a thousand parallel vms in the same process if you want. But the macro hacks are impressive!
- axegon_ 6y agoWhile I've seen people doing this, I've never done it. I've done the opposite in a small open source project of mine(arguably abandoned at this point due to lack of time and contributors). That said, there is a huge potential here - your code may require some small operation that is cheap to execute but would take you 2 minutes to write in python while it would take you an hour of doing the same in rust. I can think of some use cases for this. Nice article!
- uijl 6y agoAt FOSDEM there was a talk about boosting Python with Rust [1]. Might be interesting for the people checking this. [1] https://fosdem.org/2020/schedule/event/python2020_rust/ https://fosdem.org/2020/schedule/event/python2020_rust/
- deleted 6y ago[deleted]
- rav 6y agoThe blog post shows two Python snippets starting with "if True:" and different indentation and says the snippets "have a different meaning". However, in this case the difference between the snippets is mainly in their syntax and not in their meaning. The example would have been better if "if True:" was replaced by "if False:" or "if foo:". if True: x() y() if True: x() y()
- m-ou-se 6y agoGood point! Updated.
- zengid 6y agoI'm having issues seeing the whole post, when I scroll down i just see white, and when I click the subject headers from the menu it jumps down half the page and the mouse wheel scroll gets locked. I'm using Chrome on MacOS.