14 ms·
Using Rust in Mercurial
- the_mitsuhiko 9y agoIt’s not being rewritten in Rust. They want to use rust instead of C for extension modules.
- oconnor663 9y agoCould we get a mod edit on the title here? The article itself is called "Using Rust in Mercurial".
- jxcl 9y agoOne of us misread the article. I understood it to say that they want to rewrite the core of the Python code to Rust so that they don't have to wait for the Python interpreter to start every time someone uses the hg command. The Rust binary will be able to call Python code, which is basically a complete reversal of what they have now.
- JDevlieghere 9y agoThis is how I understood it as well.
- oconnor663 9y agoI read it as they want it to be possible for some commands to be written in pure rust, but that they expect to maintain the majority of their current Python code.
- seba_dos1 9y agoIf it's already possible to write C extensions, then it's already possible to write Rust extensions, so that would be a non-news.
- oconnor663 9y agoI think the idea is that in order to avoid Python interpreter startup time on commands that are primarily written in Rust, they also want to have the main function written in Rust, with Python interpreter startup being an optional codepath (that a majority of the codebase still depends on).
- the_mitsuhiko 9y ago> One of us misread the article. I understood it to say that they want to rewrite the core of the Python code to Rust so that they don't have to wait for the Python interpreter to start every time someone uses the hg command. It would still use the Python interpreter internally but they want to skip it for some functionality like the basic command line interface. So they would embed the Python interpreter in their new distribution. However the Python code still just calls out to other Rust code.
- merb 9y ago> However the Python code still just calls out to other Rust code. this will probably be hard. calling back and forth is always harder than into a single direction.
- ahupp 9y agoFrom the page: "Desired End State: hg is a Rust binary that embeds and uses a Python interpreter when appropriate (hg is a Python script today)"
- m12k 9y ago> Desired End State: hg is a Rust binary that embeds and uses a Python interpreter when appropriate (hg is a Python script today) I'd say it seems like they want to also flip the ratio of the core around, so it's Rust for the central parts and Python only as needed (whereas today it's Python for the core and C for certain performance critical things).
- wtetzner 9y agoI'm not sure they want to flip the ratio as much as the architecture. The entry point would be Rust, but the majority of the code would still be Python.
- neandrake 9y agoI quoted this elsewhere but this part makes it sound like they are intending to move over large parts of the implementation into rust in the long run, migrating away from python due to the size of the project becoming more difficult to maintain > In addition to performance concerns, Python is also hindering us because it is a dynamic programming language. Mercurial is a large project by Python standards. Large projects are harder to maintain. Using a statically typed programming language that finds bugs at compile time will enable us to make wide-sweeping changes more fearlessly. This will improve Mercurial's development velocity.
- sctb 9y agoWe've updated the submission title, which was “Mercurial (hg and C extensions) being rewritten in Rust” to that of the article. It's surprisingly easy to mislead, which is one of the reasons we ask submitters to use the original title and not to editorialize.
- deleted 9y ago[deleted]
- dragonwriter 9y agoNo, they want to flip the relationship of Python and native code; instead of hg being Python with C extensions, they want it to be Rust with an embedded Python interpreter. It's not a total rewrite in Rust, but it's also a more significant change than just replacing C extensions with Rust extensions while maintaining a Python core. Quoting the source material on the “Desired End State”: “hg is a Rust binary that embeds and uses a Python interpreter when appropriate (hg is a Python script today)”
- deleted 9y ago[deleted]
- steveklabnik 9y agoI was wondering how serious this was. I don't know a lot about how mercurial is developed, but https://twitter.com/indygreg/status/937527180292014080 https://twitter.com/indygreg/status/937527180292014080 https://gregoryszorc.com/work.html https://gregoryszorc.com/work.html > I am a significant contributor to the Mecurial open source version control system. > I serve on the Mercurial Steering Committee, which is the governance group for the Mercurial Project. I also have reviewing privileges, allowing me to accept incoming patches for incorporation in the project. So not sure, but it seems like at least one person on the team is into it?
- masklinn 9y agoAugie Fackler seems surprised, but not necessarily negatively, they're a pretty big contributor (top 5~10 by commits).
- ngoldbaum 9y agoThis is a proposal not a plan. Right now it’s 100% vaporware.
- steveklabnik 9y agoYeah, totally; I was wondering if it's a proposal that came from the team themselves, or some random person.
- durin42 9y agoThere's pretty broad consensus that we'd like to do Rust and not C for future extension work, but the current plan is that a pure-Python hg will also be something we support. The "rust binary that embeds Python" approach looks like a straightforward win on Windows, where we have a native .exe that embeds Python anyway. I'm not sure if that'll make sense on non-Windows, but we'll see. I've done some poking around with milksnake, which seems extremely promising for writing native speedups.
- tonfa 9y ago
- gcb0 9y agoreally wonder why not Go. original code was python with lots of C. the same can be done with Go, while keeping much of the same philosophy of python. This change will probably alienate most of the contributors since rust and python or C are worlds apart.
- rkangel 9y agoI'm curious, why is Go+Python easier than Rust+Python?
- Uther 9y agoIt is not. Go alone is easier than Rust. But since Rust has no GC, it is easier to embed, especially with a language with a Garbage collector like Python.
- oconnor663 9y agoMy understanding is that Go would be a fine candidate if you wanted to rewrite the entire codebase in it, but not for writing compiled extensions that Python can dynamically link, and they want one language to do both of those things?
- jhugg 9y agoOne reason might be that it seems easier to mix languages with Rust vs. Go. The C interop is pretty sane and Go has a few more gotchas in that department. The rest of the Rust v Go debate may be similar to the 1000 times people have had it before.
- kevin_thibedeau 9y agoNim is an even closer match to Python and interop is trivial.
- deleted 9y ago[deleted]
- rubber_duck 9y ago
- ruke 9y agoI suddenly remember FbExperiment on building Mercurial server using Rust https://github.com/facebookexperimental/mononoke https://github.com/facebookexperimental/mononoke And yes.. it's for extension modules (maybe just the beginning)
- kyrra 9y agoFor those that don't want to deal with the Python startup time, the Mercurial team already has an attempt at fixing this with a tool called CHg[0]. It is a C binary that interacts with the Mercurial CommandServer[1], which is just a long running version of Mercurial CLI that you can interact with over a pipe. Using Rust as the primary client would simplify this a lot, but is a lot more work than what CHg accomplished. [0] https://www.mercurial-scm.org/wiki/CHg https://www.mercurial-scm.org/wiki/CHg [1] https://www.mercurial-scm.org/wiki/CommandServer https://www.mercurial-scm.org/wiki/CommandServer
- ngrilly 9y agoThis is already explained in the linked article.
- kyrra 9y agoI noticed this after the fact. Thanks for pointing it out. I had previously looked into understanding how mercurial worked and what options there were for different frontends or backends. (And what it would take to write my own backend just to understand a very basic flow of a clone or pull). It's a lot of work for sure and there mercurial team (at least mpm a few years back) preferred one true implementation.
- yeukhon 9y agoSo my understanding is the hg developers are planning to rewrite a large part of hg in Rust. If so, this is an unfortunate blow to Python, because often I hear (and I do as well) Python developers cite hg as one of the largest Python-based application (regardless of the C-extension). I certainly feel sad if that is the case. I studied hg quite in depth in undergraduate for a semester, when I was implementing "bitbucket" myself. To be really honest, the codebase was easy to navigate, and function names were pretty consistent with the actual hg commands/internal spec. While the code itself is probably hard to write any true unit tests (you'd have to monkeypatch like crazy) using mock -- which means the function has a lot of code, overall the codebase quality was pretty good for a complex software. I just had to know the variable name abbreviations, get used to them and referred back to the Hg paper. In anyhow, whatever the decision is, I'd learn a bit Rust to help out :-) P.S. I am still trying to find an answer to this: if FB uses Hg, then what about their git code?
- nindalf 9y ago> what about their git code? Could you elaborate?
- yeukhon 9y agoFB has git repositories on GitHub, for example, React.
- chme 9y agoYou don't necessarily go to github just to use git, but for the community, advertisement, ... Also facebook is a big company, and like all bigger companies many people in it prefer different stuff.
- yeukhon 9y agoI am not sure if I wasn't clear. I have never worked for FB, so I don't know how internally FB handles this. But my understanding is, FB uses Hg for its main repository. However, FB has developed open source projects which most of them are published on Github, so for those development teams they work with Git. The question is, how do they reconcile some dev teams are context switching between hg and git? How common is that? Personally, I would allow one DVCS only, so my internal development platform doesn't have to support both Git and Hg (who knows, maybe FB also have teams using SVN for good reasons).
- bedros 9y agohow would they deal with hg extensions written in Python?
- krschultz 9y agoLeaving aside the actual project at hand, this is a great example of a well thought out project plan. There is a clear rationale, clear end state, a bunch of known problems to tackle, a front loading of risk, and it delivers incremental value along the way.
- puddums 9y agoInitial reaction to headline was it sounded like (a) no more python, and (b) this is a decided future direction. Instead, it sounds like this is a proof-of-concept for flipping the main 'hg' command from being python + C extensions, to instead being a rust binary with an embedded python interpreter. Part of the rationale appears to be performance, but also smoothing out cross platform experience, especially on Windows. Pulling out some related snippets: ----- While Python is still a critical component of Mercurial and will be for the indefinite future, I'd like Mercurial to pivot away from being pictured as a "Python application" and move towards being a "generic/system application." In other words, Python is just an implementation detail. ----- Desired End State hg is a Rust binary that embeds and uses a Python interpreter when appropriate (hg is a Python script today). Python code seemlessly calls out to functionality implemented in Rust. Fully self-contained Mercurial distributions are available (Python is an implementation detail / Mercurial sufficiently independent from other Python presence on system) ----- "Standalone Mercurial" is a generic term given to a distribution of Mercurial that is standalone and has minimal dependencies on the host (typically just the C runtime library). Instead, most of Mercurial's dependencies are included in the distribution. This includes a Python interpreter. ----- This patch should be considered early alpha and RFC quality.
- puddums 9y ago... and now looks like the HN headline was improved to be more representative of the content compared to the earlier headline.
- bulldoa 9y agowhat does embed python interpreter mean? are they actually writing a python interpreter using rust so they can write python code and compile to rust?
- puddums 9y agopython has a concept of "extending" and also "embedding". It looks like they are looking at embedding[0], which enables you use the normal CPython interpreter from within another program. (So no, not writing a new Python interpreter in Rust). Sample snippet from python docs: ----- So if you are embedding Python, you are providing your own main program. One of the things this main program has to do is initialize the Python interpreter. At the very least, you have to call the function Py_Initialize(). There are optional calls to pass command line arguments to Python. Then later you can call the interpreter from any part of the application. There are several different ways to call the interpreter: you can pass a string containing Python statements to PyRun_SimpleString(), <...etc..> ----- [0] https://docs.python.org/3/extending/embedding.html https://docs.python.org/3/extending/embedding.html
- bla2 9y ago> The nice things we want to do in native code are complicated to implement in C because cross-platform C is hard. The standard library is inadequate compared to modern languages. While modern versions of C++ are nice, we still support Python 2.7 and thus need to build with MSVC 2008 on Windows. It doesn't have any of the nice features that modern versions of C++ have. Things like introducing a thread pool in our current C code would be hard. But with Rust, that support is in the standard library and "just works." Having Rust's standard library is a pretty compelling advantage over C/C++ for any project, not just Mercurial. Sounds like the main reason for rust is that Python has a weird dep on a fixed MSVC version.
- Niten 9y agoWhich they mention they'd still have to implement workarounds for if they adopt Rust, so I'm not sure I understand that selling point.
- pornel 9y agoSwitching to Rust removes a lot of papercuts from dealing with MSVC. These things could be solved in many other ways, so it's not exactly the reason, just something you also get from adopting Rust. For C programs Windows happens to be the odd one out for lots of things. It's annoying with its unloved C compiler, missing unix-ish headers and tools, string encoding pains, very different packaging story, etc. So I think the motivation here is they'll solve just that one CRT problem now, and will have fewer Windows problems to worry about later. I've done that for pngquant. I can build things with Cargo rather than explain to users that `make` may be `mingw32_make`. I can parse options with Rust's getopts, rather than getting bugreports that Visual Studio can't find `getopt.h`. These aren't hard problems. I could have solved all of them, but I don't have to!
- wtetzner 9y agoIt sounds more like they don't want to be stuck using an old version of C++ for writing their fast-path, and would prefer to use Rust, because of the nice features it provides.
- neandrake 9y ago
- agentgt 9y agoPerhaps childish and trite but I do like the use of the word “oxidation” for porting to Rust
- nnethercote 9y agoThere is precedent! https://wiki.mozilla.org/Oxidation https://wiki.mozilla.org/Oxidation
- johnny_1010 9y ago"Rewrite everything in Rust, exactly the same way but in Rust it solve all problem." - some Rust programmer