3 ms·

The real reason is that the Python community has been TOO NICE. NodeJS and Ruby broke everything. What did they do ? They said "move your butt" and gave a sm

The real reason is that the Python community has been TOO NICE. NodeJS and Ruby broke everything. What did they do ? They said "move your butt" and gave a small windows of time. Everybody moved. Python ? We gave 10 years. Created tools. Tutorials. A way to write Python 2/3 compatible programs. Then gave 5 more years. The result ? Nobody moved for 9 years and spread FUD on how hard it is. But the reality is: most people are not Google. They don't have thousands of 2.7 files to convert. If you have a 20k LOCS Python project, you can convert that in 2 days. After what you will get a MUCH, MUCH better Python version anyway so it's largely worth it.

upofadown 9y ago
The way I remember it is that people were complaining about all the changes in Py2 that were breaking their code all the time. So it was decided to leave Py2 alone and do all the breaking stuff on Py3. The staticness of Py2 made it an attractive target for new development. As a result Python became very popular. So if they had of just broke everything that very easily could of been the straw the broke the camel's back. Python would not of achieved the popularity it did. I think the fundamental error here was the idea that languages become popular because of some fundamental quality of the language as a language. That never really happens. It is always something else. In the case of Python it was the "Batteries Included" thing. It came with a whole whack of functionality in the form of libraries written in other languages. No one much cared about the details of the framework for getting at that functionality.
eesmith 9y ago
I have a Python project with 28KLOC of Python, 22KLOC for a Python/C extension, and 26KLOC for the tests, so about 4x larger. It took me about 6-8 weeks to port. Part of it was because I had to change the API. I had an API with a method like "X.to_string(fmt)" where the format specifier could be something like 'csv' for text and 'csv.gz' for gzip-compressed text. I had to split that out into a "X.to_string()" and "X.to_bytes()" variants. I also deal with formats which almost always only use ASCII but where people end up putting in Latin-1 or UTF-8. The API was something like "X.get_field(name)", where name was a byte string, and it returned a byte string. (GIGO and it's your responsibility to deal with your byte strings.) In Python 3, X had to acquire an "encoding" and "errors" parameter, so it could know the expected encoding for the entire record. But to make it work, in practice each field could have a different encoding, so I ended up also adding a "X.get_field_as_bytes(name)". The 'name' can be a Unicode or byte string. These changes to return a Unicode string from a byte string ended up adding a lot of code of the form: try: field = field_bytes.decode(self.encoding, self.encoding_errors) except UnicodeDecodeError as error: die("Cannot decode field %r (%r): %s" % (field_name, field_bytes, error) All of these new branches then meant adding unit tests to ensure coverage. Parts of the code supported passing in a byte string, Unicode string containing only ASCII characters, or buffer object. There's no easy way to handle that in the API, so I eventually dug into Python's own source code to copy how Modules/binascii.c supports it. I also had to switch all my code to use the new buffer API. Only to find I missed a PyBuffer_Release(), which caused a slow memory leak. Then the performance in Python 3 was slower than Python 2, so I pushed more of the Python-level code into a C extension. I'm not convinced the new API is all that much better. It's certainly harder to implement and therefore maintain, and I don't have all that many users. But I can say that "you can convert that in 2 days" feels like an optimistic statement. The test suite contains over 2000 byte string constants, and even at 5 changes per minute that one change alone would take 6 hours. Admittedly, part of the reason it took 6-8 weeks was because the result works under both Python 2.7 and Python 3.5+. It is not so easy to make compatible Python/C extensions. But as literally all of my users were using Python 2.x, I wasn't going to drop support for at least Python 2.7. (I learned a few months ago that one potential customer was still using Python 2.6 on their compute cluster. They have since upgraded to Python 2.7. I warned them about 2020.)
- sametmax 9y ago
  You took my "port 20K LOC python code" to "take 70K LOC of mixed of Python and C and make it compatible with 2 and 3". That's an entirely different story, and is definitely the average project.
  - eesmith 9y ago
    Agreed, but are you saying the scaleup is non-linear in LOC? Because simple linear scaleup suggests 6 days of work, not 30-40 days of work. I also pointed out that just the test suite, which not much bigger than 20KLOC, required hours of work to tweak all of the b"" strings correctly. One of the other tedious difficulties is that this code base started during Python 2.5, so there were many submodules with print statements which needed to turned into print functions. (Annoyingly, some "print >> f, s" debug lines, enabled with an undocumented environment variable, made it to release, because that is still valid syntax under Python 3 ("unsupported operand type(s) for >>: 'builtin_function_or_method'") and it wasn't part of the test suite.) I looked at another code base of mine, which is 18.5 KLOC (9.5KLOC my source, 4.5KLOC for a third-party submodule which already supported Python 3, and 4.5KLOC of a machine generated table which needed no porting). This one started under Python 2.7, and made sure to use __future__ compatibility even when it was Python 2.7-only. That helped reduce the porting cost. There are 23 commits which specifically mention fixing compatibility problems. However, I did the Python 3 update during a major refactor, and included Python 3 changes as part of the refactoring. I then re-tested under Python 2 to check for compatibility. This means I can't identify all of the changes which are 2->3 specific. These are mostly the usual iteritems()->items(), zip()->list(zip()), itertools.izip()->zip(), a compatibility shim for isinstance(obj, basestring), open(name, "U") vs. open(name, "r", newline=None), etc. Some of the trickier ones were: how to read bytes from stdin (sys.stdin.buffer); explicit flushes on sys.stderr (not needed for Python 2.x); a StringIO-like wrapper around BytesIO so I could write() both bytes and Unicode strings containing only ASCII (which is what cStringIO.StringIO did); and a few workarounds for changes in argparse behavior. The trickiest is that I allow the user to pass in an eval-able mathematical expression on the command-line. My code first compiles the expression and checks that all of the global variables are known, in order to generate a more detailed error message than Python itself would. In Python 3, True and False are keywords, while in Python 2 they are built-ins. I didn't originally allow "True"/"False" in the expression, so there could be simple expressions which were valid under the Python 3 version which weren't valid under the Python 2. While the change was easy, figuring out that I needed the change was harder. I don't know how long it took for this code base. Based on the bunching of the commit messages, it looks like I spent at least 16 hours (= 2 days) on it, figuring from first commit of each bunch to the last. % hg history | grep --before 1 -i 'python.*[23]' | grep date date: Fri Jun 02 05:04:30 2017 +0200 date: Fri Jun 02 03:51:28 2017 +0200 date: Thu Jun 01 03:37:37 2017 +0200 date: Thu Jun 01 01:16:33 2017 +0200 date: Mon May 29 17:07:00 2017 +0200 date: Sun May 28 04:48:14 2017 +0200 date: Sun May 28 03:38:34 2017 +0200 date: Sun May 28 03:18:56 2017 +0200 date: Sat May 27 00:28:08 2017 +0200 date: Fri May 26 16:29:03 2017 +0200 date: Fri May 26 16:28:43 2017 +0200 date: Fri May 26 15:37:40 2017 +0200 date: Fri May 26 14:26:47 2017 +0200 date: Fri May 26 13:56:42 2017 +0200 date: Thu May 25 04:38:45 2017 +0200 date: Thu May 25 03:52:04 2017 +0200 date: Thu May 25 03:51:29 2017 +0200 date: Thu May 25 03:51:01 2017 +0200 date: Thu May 25 03:36:34 2017 +0200 date: Thu May 25 01:33:26 2017 +0200 date: Wed May 24 18:44:42 2017 +0200 date: Wed May 24 16:52:40 2017 +0200 date: Wed May 24 16:51:47 2017 +0200 This is about twice as long as your estimate would suggest. (I'm excluding the auto-generated data table and the third-party package from my LOC count.)

upofadown 9y ago

The way I remember it is that people were complaining about all the changes in Py2 that were breaking their code all the time. So it was decided to leave Py2 alone and do all the breaking stuff on Py3. The staticness of Py2 made it an attractive target for new development. As a result Python became very popular. So if they had of just broke everything that very easily could of been the straw the broke the camel's back. Python would not of achieved the popularity it did. I think the fundamental error here was the idea that languages become popular because of some fundamental quality of the language as a language. That never really happens. It is always something else. In the case of Python it was the "Batteries Included" thing. It came with a whole whack of functionality in the form of libraries written in other languages. No one much cared about the details of the framework for getting at that functionality.

eesmith 9y ago

I have a Python project with 28KLOC of Python, 22KLOC for a Python/C extension, and 26KLOC for the tests, so about 4x larger. It took me about 6-8 weeks to port. Part of it was because I had to change the API. I had an API with a method like "X.to_string(fmt)" where the format specifier could be something like 'csv' for text and 'csv.gz' for gzip-compressed text. I had to split that out into a "X.to_string()" and "X.to_bytes()" variants. I also deal with formats which almost always only use ASCII but where people end up putting in Latin-1 or UTF-8. The API was something like "X.get_field(name)", where name was a byte string, and it returned a byte string. (GIGO and it's your responsibility to deal with your byte strings.) In Python 3, X had to acquire an "encoding" and "errors" parameter, so it could know the expected encoding for the entire record. But to make it work, in practice each field could have a different encoding, so I ended up also adding a "X.get_field_as_bytes(name)". The 'name' can be a Unicode or byte string. These changes to return a Unicode string from a byte string ended up adding a lot of code of the form: try: field = field_bytes.decode(self.encoding, self.encoding_errors) except UnicodeDecodeError as error: die("Cannot decode field %r (%r): %s" % (field_name, field_bytes, error) All of these new branches then meant adding unit tests to ensure coverage. Parts of the code supported passing in a byte string, Unicode string containing only ASCII characters, or buffer object. There's no easy way to handle that in the API, so I eventually dug into Python's own source code to copy how Modules/binascii.c supports it. I also had to switch all my code to use the new buffer API. Only to find I missed a PyBuffer_Release(), which caused a slow memory leak. Then the performance in Python 3 was slower than Python 2, so I pushed more of the Python-level code into a C extension. I'm not convinced the new API is all that much better. It's certainly harder to implement and therefore maintain, and I don't have all that many users. But I can say that "you can convert that in 2 days" feels like an optimistic statement. The test suite contains over 2000 byte string constants, and even at 5 changes per minute that one change alone would take 6 hours. Admittedly, part of the reason it took 6-8 weeks was because the result works under both Python 2.7 and Python 3.5+. It is not so easy to make compatible Python/C extensions. But as literally all of my users were using Python 2.x, I wasn't going to drop support for at least Python 2.7. (I learned a few months ago that one potential customer was still using Python 2.6 on their compute cluster. They have since upgraded to Python 2.7. I warned them about 2020.)

sametmax 9y ago
You took my "port 20K LOC python code" to "take 70K LOC of mixed of Python and C and make it compatible with 2 and 3". That's an entirely different story, and is definitely the average project.
- eesmith 9y ago
  Agreed, but are you saying the scaleup is non-linear in LOC? Because simple linear scaleup suggests 6 days of work, not 30-40 days of work. I also pointed out that just the test suite, which not much bigger than 20KLOC, required hours of work to tweak all of the b"" strings correctly. One of the other tedious difficulties is that this code base started during Python 2.5, so there were many submodules with print statements which needed to turned into print functions. (Annoyingly, some "print >> f, s" debug lines, enabled with an undocumented environment variable, made it to release, because that is still valid syntax under Python 3 ("unsupported operand type(s) for >>: 'builtin_function_or_method'") and it wasn't part of the test suite.) I looked at another code base of mine, which is 18.5 KLOC (9.5KLOC my source, 4.5KLOC for a third-party submodule which already supported Python 3, and 4.5KLOC of a machine generated table which needed no porting). This one started under Python 2.7, and made sure to use __future__ compatibility even when it was Python 2.7-only. That helped reduce the porting cost. There are 23 commits which specifically mention fixing compatibility problems. However, I did the Python 3 update during a major refactor, and included Python 3 changes as part of the refactoring. I then re-tested under Python 2 to check for compatibility. This means I can't identify all of the changes which are 2->3 specific. These are mostly the usual iteritems()->items(), zip()->list(zip()), itertools.izip()->zip(), a compatibility shim for isinstance(obj, basestring), open(name, "U") vs. open(name, "r", newline=None), etc. Some of the trickier ones were: how to read bytes from stdin (sys.stdin.buffer); explicit flushes on sys.stderr (not needed for Python 2.x); a StringIO-like wrapper around BytesIO so I could write() both bytes and Unicode strings containing only ASCII (which is what cStringIO.StringIO did); and a few workarounds for changes in argparse behavior. The trickiest is that I allow the user to pass in an eval-able mathematical expression on the command-line. My code first compiles the expression and checks that all of the global variables are known, in order to generate a more detailed error message than Python itself would. In Python 3, True and False are keywords, while in Python 2 they are built-ins. I didn't originally allow "True"/"False" in the expression, so there could be simple expressions which were valid under the Python 3 version which weren't valid under the Python 2. While the change was easy, figuring out that I needed the change was harder. I don't know how long it took for this code base. Based on the bunching of the commit messages, it looks like I spent at least 16 hours (= 2 days) on it, figuring from first commit of each bunch to the last. % hg history | grep --before 1 -i 'python.*[23]' | grep date date: Fri Jun 02 05:04:30 2017 +0200 date: Fri Jun 02 03:51:28 2017 +0200 date: Thu Jun 01 03:37:37 2017 +0200 date: Thu Jun 01 01:16:33 2017 +0200 date: Mon May 29 17:07:00 2017 +0200 date: Sun May 28 04:48:14 2017 +0200 date: Sun May 28 03:38:34 2017 +0200 date: Sun May 28 03:18:56 2017 +0200 date: Sat May 27 00:28:08 2017 +0200 date: Fri May 26 16:29:03 2017 +0200 date: Fri May 26 16:28:43 2017 +0200 date: Fri May 26 15:37:40 2017 +0200 date: Fri May 26 14:26:47 2017 +0200 date: Fri May 26 13:56:42 2017 +0200 date: Thu May 25 04:38:45 2017 +0200 date: Thu May 25 03:52:04 2017 +0200 date: Thu May 25 03:51:29 2017 +0200 date: Thu May 25 03:51:01 2017 +0200 date: Thu May 25 03:36:34 2017 +0200 date: Thu May 25 01:33:26 2017 +0200 date: Wed May 24 18:44:42 2017 +0200 date: Wed May 24 16:52:40 2017 +0200 date: Wed May 24 16:51:47 2017 +0200 This is about twice as long as your estimate would suggest. (I'm excluding the auto-generated data table and the third-party package from my LOC count.)