4 ms·

I know this is a tired subject, but as Mercurial is nearing its multi-year effort to upgrade to Python 3, patience wears thin: https://twitter.com/indygreg/sta

I know this is a tired subject, but as Mercurial is nearing its multi-year effort to upgrade to Python 3, patience wears thin: https://twitter.com/indygreg/status/1187069501424062464 https://twitter.com/indygreg/status/1187069501424062464 Mercurial was in fact one of important reasons for why the Python 2 deprecation was pushed back five years. We met with Guido at Pycon 2014 in Montreal and asked for an extension, which he granted. Back then Python itself was using Mercurial, so maybe that's why our opinion carried some weight. We weren't the only ones, but we were one of the vocal ones.

pbhjpbhj 7y ago
Could you precis why it's taking multiple years? Surely there's a formal equivalence that could be automatically implemented wrt the code itself?
- lidHanteyk 7y ago
  Python 2 and Python 3 are distinct languages with a lot of commonalities. It's a difficult port that cannot be automated.
  - kevin_thibedeau 7y ago
    It's easy to gradually upgrade Py2 code. Start introducing the future imports. Unicode handling is the only tricky one (though not so bad when the codecs module is consistently used). Then start running files through 2to3 and making the code forward compatible. This can be done over a long time span while maintaining full 2.7 compatibility.
    - marmada 7y ago
      These types of comments are confusing. The developer of Mercurial has stated that it took/is taking them several years to port to Python 3. I'm inclined to believe them compared to some random stranger who asserts that it should be easy for Mercurial to be ported to Python 3. Unless you were saying that most other/non-Mercurial programs are easy to port to Python 3, which could be true.
    - lidHanteyk 7y ago
      Have you ever carried an application using Django, or Twisted, or Zope, through this process? It is not easy. Many different libraries with various incompatibilities and versions can lead to real difficulties merely in upgrading for security releases. Additionally, unless one is containerized, one must contend with the underlying platform's issues, as the platform must also switch from Python 2 to Python 3. I understand your opinion: It is the party line. But it doesn't actually help much when the process as described will take years and people only have months, and this is further worsened by the fact that PSF did not have to deprecate the entire Python 2 ecosystem in order to stop supporting CPython 2.7.
      - kevin_thibedeau 7y ago
        The is more the fault of certain libraries playing non-Pythonic games with monkeypatches and magic code on import. Mercurial is not burdened by those troubles. The writing was on the wall ten years ago. Failure to take action is not Python's fault.
- jordigh 7y ago
  Mercurial treats everything as bytes. Whatever you store in your repo, that's what Mercurial reads -- bytes. There are no encodings, there are no strings. Only bytes. You don't want your VCS to start having opinions about which language you should speak and how you should encode it. PEP 461 was the bare minimum we needed before porting to Python 3 could happen: http://legacy.python.org/dev/peps/pep-0461/ http://legacy.python.org/dev/peps/pep-0461/ We needed a way to put things into bytestrings. And after that, it's just nothing but work. Lots of it. Little of which could be automated. I get a little impatient, personally, with people who think that upgrading software is easy. We don't actually care about Python. We care about version control. Python is just some thing in the middle that is helping us produce version control. A multi-year effort to upgrade to Python 3 (or 3.8 or whatever) isn't giving us or our users a net benefit. In fact, until recently, it was a net loss, because Python 3.5 or 3.6 had slower warmup time, which is terrible for a tool like hg that needs to pay that cost every time you run any command. Instead, Python 3 created busy work. Almost five years of it.
  - armitron 7y ago
    May I ask why you think it's worth doing all this work, especially now that Git has - clearly - won. How do you motivate yourself to keep working on mercurial and at what point do you cut your losses short? I assume you're not getting paid for it, so besides bragging rights or a nice CV entry, what else is there for you to look towards?
    - jordigh 7y ago
      Git is horrible to use. Few people like it, most only tolerate it. Regardless of how you feel about git, there's plenty of literature regarding how awful the git user experience is. Mercurial's greater modularity also has made it easier to implement improvements for monorepos, although personally that's less important to me. That's why I still use and work on Mercurial.
      - bipson 7y ago
        Highly subjective I would say. Git is not horrible to use. And even before I was a proficient Git user, mercurial was giving a team of 5 people a lot of work trying to fix stuff that just works on Git.
- pjc50 7y ago
  Not really, no. There are some automatic transforms that "2to3" will do for you, and it's mostly automated, but the string handling is fundamentally different in subtle ways and there are a few places where you have to alter your entire architecture (e.g. make all your exceptions inherit from Exception). If your architecture makes too many assumptions about the equivalence of characters and bytes problems can be severe. The decision to make it not backward-compatible and thereby force-deprecate millions of lines of working Python code was extremely expensive.
  - BJBBB 7y ago
    Tend to agree. My previous employer did most of the ATE, quality, and production control in Python; and hardware drivers mostly in C. It worked reliably for about 15 years. During late 2017, the new 'code boss' mandated Python 3 for new stuff, and to convert as required for existing stuff. Production continued with little or no problems until an important customer did a major spec change, plus the new safety and EMC regulations and standards became effective, so we had to get some stuff re-certified and some processes accredited (again). After three months, the boss fired the three programmers and assigned the task to the quality, manufacturing and regulatory engineers. We looked at the code for a few days, then said nope and re-designed and wrote in C/C++. When I left in November, things were humming along well, but the parent company saw this as a huge future liability and commenced to shut down that part of the company that made that class of widgets. Think about - a group of less than 40 production people and a few engineers that pumped 25 million USD profit annually into the mother company was dropped because of potential future code shenanigans. Because of these experiences, I discourage use Python professionally (unless the contract allows for some specific indemnity clauses) for quality and manufacturing, and know of other manufacturing and hardware activities abandoning Python for new stuff and 'returning' to C and C++. I continue to enjoy and use Python (mostly 3) for personal projects, where support and making money do not always matter.
ken 7y ago
Back when I looked at it, at least, Mercurial had the strangest and most non-idiomatic Python I'd ever seen. It doesn't surprise me it's been trouble to upgrade. Mercurial's curious implementation is also the reason I gave up trying to do anything with it. I love the idea of a popular DVCS written in a HLL, but it looked halfway to C.
- jordigh 7y ago
  > Mercurial had the strangest and most non-idiomatic Python I'd ever seen. It's not so bad. We just use a lot of abbreviations, but you get used to them. There's also a reason for a lot of the ways it's written: backwards compatibility. We supported Python 2.4 until a couple of years ago. This meant you had to do "x and y or z" instead of "y if x else z" because the ternary operator wasn't available in older releases. This wasn't because we like old software, but because our users do. A data-storage solution needs to be stable over very long periods of time. You don't want to be unable to access your repo because you're on an old machine. A DVCS is too important to force your users to be chasing after the latest release all the time. Another reason for Mercurial's style is to avoid unnecessary function lookups or calls and other quirks of the Python interpreter. Something as simple as for x in y: thing.doit(x) can have a noticeable speedup if you instead do doit = thing.doit for x in y: doit(x) because Python just takes a while doing function lookups. Our users notice those milliseconds of difference because they want "hg status" or "hg log" to be nearly instantaneous.
  - bipson 7y ago
    May I ask why you are even using Python, if you need to work around the language in such ways? These non-idiomatic uses might be shedding a few ms, yet they ruin the readability and maintainability of the code, most of all all tools like 2to3? Why not another (non-interpreted and thus faster) language? One where the idoms work in your favor? Also, no need to support releases from 2004 and having to groan about it.

pbhjpbhj 7y ago

Could you precis why it's taking multiple years? Surely there's a formal equivalence that could be automatically implemented wrt the code itself?

lidHanteyk 7y ago
Python 2 and Python 3 are distinct languages with a lot of commonalities. It's a difficult port that cannot be automated.
- kevin_thibedeau 7y ago
  It's easy to gradually upgrade Py2 code. Start introducing the future imports. Unicode handling is the only tricky one (though not so bad when the codecs module is consistently used). Then start running files through 2to3 and making the code forward compatible. This can be done over a long time span while maintaining full 2.7 compatibility.
  - marmada 7y ago
    These types of comments are confusing. The developer of Mercurial has stated that it took/is taking them several years to port to Python 3. I'm inclined to believe them compared to some random stranger who asserts that it should be easy for Mercurial to be ported to Python 3. Unless you were saying that most other/non-Mercurial programs are easy to port to Python 3, which could be true.
  - lidHanteyk 7y ago
    Have you ever carried an application using Django, or Twisted, or Zope, through this process? It is not easy. Many different libraries with various incompatibilities and versions can lead to real difficulties merely in upgrading for security releases. Additionally, unless one is containerized, one must contend with the underlying platform's issues, as the platform must also switch from Python 2 to Python 3. I understand your opinion: It is the party line. But it doesn't actually help much when the process as described will take years and people only have months, and this is further worsened by the fact that PSF did not have to deprecate the entire Python 2 ecosystem in order to stop supporting CPython 2.7.
    - kevin_thibedeau 7y ago
      The is more the fault of certain libraries playing non-Pythonic games with monkeypatches and magic code on import. Mercurial is not burdened by those troubles. The writing was on the wall ten years ago. Failure to take action is not Python's fault.
jordigh 7y ago
Mercurial treats everything as bytes. Whatever you store in your repo, that's what Mercurial reads -- bytes. There are no encodings, there are no strings. Only bytes. You don't want your VCS to start having opinions about which language you should speak and how you should encode it. PEP 461 was the bare minimum we needed before porting to Python 3 could happen: http://legacy.python.org/dev/peps/pep-0461/ http://legacy.python.org/dev/peps/pep-0461/ We needed a way to put things into bytestrings. And after that, it's just nothing but work. Lots of it. Little of which could be automated. I get a little impatient, personally, with people who think that upgrading software is easy. We don't actually care about Python. We care about version control. Python is just some thing in the middle that is helping us produce version control. A multi-year effort to upgrade to Python 3 (or 3.8 or whatever) isn't giving us or our users a net benefit. In fact, until recently, it was a net loss, because Python 3.5 or 3.6 had slower warmup time, which is terrible for a tool like hg that needs to pay that cost every time you run any command. Instead, Python 3 created busy work. Almost five years of it.
- armitron 7y ago
  May I ask why you think it's worth doing all this work, especially now that Git has - clearly - won. How do you motivate yourself to keep working on mercurial and at what point do you cut your losses short? I assume you're not getting paid for it, so besides bragging rights or a nice CV entry, what else is there for you to look towards?
  - jordigh 7y ago
    Git is horrible to use. Few people like it, most only tolerate it. Regardless of how you feel about git, there's plenty of literature regarding how awful the git user experience is. Mercurial's greater modularity also has made it easier to implement improvements for monorepos, although personally that's less important to me. That's why I still use and work on Mercurial.
    - bipson 7y ago
      Highly subjective I would say. Git is not horrible to use. And even before I was a proficient Git user, mercurial was giving a team of 5 people a lot of work trying to fix stuff that just works on Git.
pjc50 7y ago
Not really, no. There are some automatic transforms that "2to3" will do for you, and it's mostly automated, but the string handling is fundamentally different in subtle ways and there are a few places where you have to alter your entire architecture (e.g. make all your exceptions inherit from Exception). If your architecture makes too many assumptions about the equivalence of characters and bytes problems can be severe. The decision to make it not backward-compatible and thereby force-deprecate millions of lines of working Python code was extremely expensive.
- BJBBB 7y ago
  Tend to agree. My previous employer did most of the ATE, quality, and production control in Python; and hardware drivers mostly in C. It worked reliably for about 15 years. During late 2017, the new 'code boss' mandated Python 3 for new stuff, and to convert as required for existing stuff. Production continued with little or no problems until an important customer did a major spec change, plus the new safety and EMC regulations and standards became effective, so we had to get some stuff re-certified and some processes accredited (again). After three months, the boss fired the three programmers and assigned the task to the quality, manufacturing and regulatory engineers. We looked at the code for a few days, then said nope and re-designed and wrote in C/C++. When I left in November, things were humming along well, but the parent company saw this as a huge future liability and commenced to shut down that part of the company that made that class of widgets. Think about - a group of less than 40 production people and a few engineers that pumped 25 million USD profit annually into the mother company was dropped because of potential future code shenanigans. Because of these experiences, I discourage use Python professionally (unless the contract allows for some specific indemnity clauses) for quality and manufacturing, and know of other manufacturing and hardware activities abandoning Python for new stuff and 'returning' to C and C++. I continue to enjoy and use Python (mostly 3) for personal projects, where support and making money do not always matter.

ken 7y ago

Back when I looked at it, at least, Mercurial had the strangest and most non-idiomatic Python I'd ever seen. It doesn't surprise me it's been trouble to upgrade. Mercurial's curious implementation is also the reason I gave up trying to do anything with it. I love the idea of a popular DVCS written in a HLL, but it looked halfway to C.

jordigh 7y ago
> Mercurial had the strangest and most non-idiomatic Python I'd ever seen. It's not so bad. We just use a lot of abbreviations, but you get used to them. There's also a reason for a lot of the ways it's written: backwards compatibility. We supported Python 2.4 until a couple of years ago. This meant you had to do "x and y or z" instead of "y if x else z" because the ternary operator wasn't available in older releases. This wasn't because we like old software, but because our users do. A data-storage solution needs to be stable over very long periods of time. You don't want to be unable to access your repo because you're on an old machine. A DVCS is too important to force your users to be chasing after the latest release all the time. Another reason for Mercurial's style is to avoid unnecessary function lookups or calls and other quirks of the Python interpreter. Something as simple as for x in y: thing.doit(x) can have a noticeable speedup if you instead do doit = thing.doit for x in y: doit(x) because Python just takes a while doing function lookups. Our users notice those milliseconds of difference because they want "hg status" or "hg log" to be nearly instantaneous.
- bipson 7y ago
  May I ask why you are even using Python, if you need to work around the language in such ways? These non-idiomatic uses might be shedding a few ms, yet they ruin the readability and maintainability of the code, most of all all tools like 2to3? Why not another (non-interpreted and thus faster) language? One where the idoms work in your favor? Also, no need to support releases from 2004 and having to groan about it.