3 ms·

You took my "port 20K LOC python code" to "take 70K LOC of mixed of Python and C and make it compatible with 2 and 3". That's an entirely different story, and i

You took my "port 20K LOC python code" to "take 70K LOC of mixed of Python and C and make it compatible with 2 and 3". That's an entirely different story, and is definitely the average project.

eesmith 9y ago
Agreed, but are you saying the scaleup is non-linear in LOC? Because simple linear scaleup suggests 6 days of work, not 30-40 days of work. I also pointed out that just the test suite, which not much bigger than 20KLOC, required hours of work to tweak all of the b"" strings correctly. One of the other tedious difficulties is that this code base started during Python 2.5, so there were many submodules with print statements which needed to turned into print functions. (Annoyingly, some "print >> f, s" debug lines, enabled with an undocumented environment variable, made it to release, because that is still valid syntax under Python 3 ("unsupported operand type(s) for >>: 'builtin_function_or_method'") and it wasn't part of the test suite.) I looked at another code base of mine, which is 18.5 KLOC (9.5KLOC my source, 4.5KLOC for a third-party submodule which already supported Python 3, and 4.5KLOC of a machine generated table which needed no porting). This one started under Python 2.7, and made sure to use __future__ compatibility even when it was Python 2.7-only. That helped reduce the porting cost. There are 23 commits which specifically mention fixing compatibility problems. However, I did the Python 3 update during a major refactor, and included Python 3 changes as part of the refactoring. I then re-tested under Python 2 to check for compatibility. This means I can't identify all of the changes which are 2->3 specific. These are mostly the usual iteritems()->items(), zip()->list(zip()), itertools.izip()->zip(), a compatibility shim for isinstance(obj, basestring), open(name, "U") vs. open(name, "r", newline=None), etc. Some of the trickier ones were: how to read bytes from stdin (sys.stdin.buffer); explicit flushes on sys.stderr (not needed for Python 2.x); a StringIO-like wrapper around BytesIO so I could write() both bytes and Unicode strings containing only ASCII (which is what cStringIO.StringIO did); and a few workarounds for changes in argparse behavior. The trickiest is that I allow the user to pass in an eval-able mathematical expression on the command-line. My code first compiles the expression and checks that all of the global variables are known, in order to generate a more detailed error message than Python itself would. In Python 3, True and False are keywords, while in Python 2 they are built-ins. I didn't originally allow "True"/"False" in the expression, so there could be simple expressions which were valid under the Python 3 version which weren't valid under the Python 2. While the change was easy, figuring out that I needed the change was harder. I don't know how long it took for this code base. Based on the bunching of the commit messages, it looks like I spent at least 16 hours (= 2 days) on it, figuring from first commit of each bunch to the last. % hg history | grep --before 1 -i 'python.*[23]' | grep date date: Fri Jun 02 05:04:30 2017 +0200 date: Fri Jun 02 03:51:28 2017 +0200 date: Thu Jun 01 03:37:37 2017 +0200 date: Thu Jun 01 01:16:33 2017 +0200 date: Mon May 29 17:07:00 2017 +0200 date: Sun May 28 04:48:14 2017 +0200 date: Sun May 28 03:38:34 2017 +0200 date: Sun May 28 03:18:56 2017 +0200 date: Sat May 27 00:28:08 2017 +0200 date: Fri May 26 16:29:03 2017 +0200 date: Fri May 26 16:28:43 2017 +0200 date: Fri May 26 15:37:40 2017 +0200 date: Fri May 26 14:26:47 2017 +0200 date: Fri May 26 13:56:42 2017 +0200 date: Thu May 25 04:38:45 2017 +0200 date: Thu May 25 03:52:04 2017 +0200 date: Thu May 25 03:51:29 2017 +0200 date: Thu May 25 03:51:01 2017 +0200 date: Thu May 25 03:36:34 2017 +0200 date: Thu May 25 01:33:26 2017 +0200 date: Wed May 24 18:44:42 2017 +0200 date: Wed May 24 16:52:40 2017 +0200 date: Wed May 24 16:51:47 2017 +0200 This is about twice as long as your estimate would suggest. (I'm excluding the auto-generated data table and the third-party package from my LOC count.)
- sametmax 9y ago
  > Agreed, but are you saying the scaleup is non-linear in LOC? Yes. Complexity is always exponential. > Because simple linear scaleup suggests 6 days of work, not 30-40 days of work. Well, again your project is particularly hard: c extension + Python 2 / 3 compatible code base. That's not the common case. You can always find somebody with a particular situation. And we hear those persons more, and not the hundred ons with just a few scripts or a website. > I also pointed out that just the test suite, which not much bigger than 20KLOC, required hours of work to tweak all of the b"" strings correctly. That's Python 2 / 3 compatible code base for you. This tweak is almost a no brainer in pure python 3. > many submodules with print statements which needed to turned into print functions. 2to3 does that automatically, and many other things, that you can even cherry-pick. If you don't use the provided tooling, you make your life harder. > These are mostly the usual iteritems()->items(), zip()->list(zip()), itertools.izip()->zip(), a compatibility shim for isinstance(obj, basestring), open(name, "U") vs. open(name, "r", newline=None), etc. python-future provide all the helpers for that: normalized builtins, import aliases, wrappers, etc. It has existed since 2013. People need to spread the word about the proverbial wheel on this one. It's like using > Some of the trickier ones were: how to read bytes from stdin (sys.stdin.buffer); explicit flushes on sys.stderr (not needed for Python 2.x); a StringIO-like wrapper around BytesIO so I could write() both bytes and Unicode strings containing only ASCII (which is what cStringIO.StringIO did); and a few workarounds for changes in argparse behavior. Yeah, that's the hard part. You can't automate that. But it's what, 3% of your code ? Unless you hate DRY. It's quite fair that bumping an entire project to an non compatible version of your plateform requires you do this. > The trickiest is that I allow the user to pass in an eval-able mathematical expression on the command-line. My code first compiles the expression and checks that all of the global variables are known, in order to generate a more detailed error message than Python itself would. Quite a cool project :) > This is about twice as long as your estimate would suggest. (I'm excluding the auto-generated data table and the third-party package from my LOC count.) Granted. Still, 4 days of porting for an entire project one time in a 25 years old language feels ok in my book. Don't forget that since 2000: - Perl 6 has been late for a decade. - PHP failed its V6 and jumped to V7. - NodeJS forked and merged twice. From this point of view, what Python did is an amazing accomplishment.

You took my "port 20K LOC python code" to "take 70K LOC of mixed of Python and C and make it compatible with 2 and 3". That's an entirely different story, and is definitely the average project.

eesmith 9y ago

Agreed, but are you saying the scaleup is non-linear in LOC? Because simple linear scaleup suggests 6 days of work, not 30-40 days of work. I also pointed out that just the test suite, which not much bigger than 20KLOC, required hours of work to tweak all of the b"" strings correctly. One of the other tedious difficulties is that this code base started during Python 2.5, so there were many submodules with print statements which needed to turned into print functions. (Annoyingly, some "print >> f, s" debug lines, enabled with an undocumented environment variable, made it to release, because that is still valid syntax under Python 3 ("unsupported operand type(s) for >>: 'builtin_function_or_method'") and it wasn't part of the test suite.) I looked at another code base of mine, which is 18.5 KLOC (9.5KLOC my source, 4.5KLOC for a third-party submodule which already supported Python 3, and 4.5KLOC of a machine generated table which needed no porting). This one started under Python 2.7, and made sure to use __future__ compatibility even when it was Python 2.7-only. That helped reduce the porting cost. There are 23 commits which specifically mention fixing compatibility problems. However, I did the Python 3 update during a major refactor, and included Python 3 changes as part of the refactoring. I then re-tested under Python 2 to check for compatibility. This means I can't identify all of the changes which are 2->3 specific. These are mostly the usual iteritems()->items(), zip()->list(zip()), itertools.izip()->zip(), a compatibility shim for isinstance(obj, basestring), open(name, "U") vs. open(name, "r", newline=None), etc. Some of the trickier ones were: how to read bytes from stdin (sys.stdin.buffer); explicit flushes on sys.stderr (not needed for Python 2.x); a StringIO-like wrapper around BytesIO so I could write() both bytes and Unicode strings containing only ASCII (which is what cStringIO.StringIO did); and a few workarounds for changes in argparse behavior. The trickiest is that I allow the user to pass in an eval-able mathematical expression on the command-line. My code first compiles the expression and checks that all of the global variables are known, in order to generate a more detailed error message than Python itself would. In Python 3, True and False are keywords, while in Python 2 they are built-ins. I didn't originally allow "True"/"False" in the expression, so there could be simple expressions which were valid under the Python 3 version which weren't valid under the Python 2. While the change was easy, figuring out that I needed the change was harder. I don't know how long it took for this code base. Based on the bunching of the commit messages, it looks like I spent at least 16 hours (= 2 days) on it, figuring from first commit of each bunch to the last. % hg history | grep --before 1 -i 'python.*[23]' | grep date date: Fri Jun 02 05:04:30 2017 +0200 date: Fri Jun 02 03:51:28 2017 +0200 date: Thu Jun 01 03:37:37 2017 +0200 date: Thu Jun 01 01:16:33 2017 +0200 date: Mon May 29 17:07:00 2017 +0200 date: Sun May 28 04:48:14 2017 +0200 date: Sun May 28 03:38:34 2017 +0200 date: Sun May 28 03:18:56 2017 +0200 date: Sat May 27 00:28:08 2017 +0200 date: Fri May 26 16:29:03 2017 +0200 date: Fri May 26 16:28:43 2017 +0200 date: Fri May 26 15:37:40 2017 +0200 date: Fri May 26 14:26:47 2017 +0200 date: Fri May 26 13:56:42 2017 +0200 date: Thu May 25 04:38:45 2017 +0200 date: Thu May 25 03:52:04 2017 +0200 date: Thu May 25 03:51:29 2017 +0200 date: Thu May 25 03:51:01 2017 +0200 date: Thu May 25 03:36:34 2017 +0200 date: Thu May 25 01:33:26 2017 +0200 date: Wed May 24 18:44:42 2017 +0200 date: Wed May 24 16:52:40 2017 +0200 date: Wed May 24 16:51:47 2017 +0200 This is about twice as long as your estimate would suggest. (I'm excluding the auto-generated data table and the third-party package from my LOC count.)

sametmax 9y ago
> Agreed, but are you saying the scaleup is non-linear in LOC? Yes. Complexity is always exponential. > Because simple linear scaleup suggests 6 days of work, not 30-40 days of work. Well, again your project is particularly hard: c extension + Python 2 / 3 compatible code base. That's not the common case. You can always find somebody with a particular situation. And we hear those persons more, and not the hundred ons with just a few scripts or a website. > I also pointed out that just the test suite, which not much bigger than 20KLOC, required hours of work to tweak all of the b"" strings correctly. That's Python 2 / 3 compatible code base for you. This tweak is almost a no brainer in pure python 3. > many submodules with print statements which needed to turned into print functions. 2to3 does that automatically, and many other things, that you can even cherry-pick. If you don't use the provided tooling, you make your life harder. > These are mostly the usual iteritems()->items(), zip()->list(zip()), itertools.izip()->zip(), a compatibility shim for isinstance(obj, basestring), open(name, "U") vs. open(name, "r", newline=None), etc. python-future provide all the helpers for that: normalized builtins, import aliases, wrappers, etc. It has existed since 2013. People need to spread the word about the proverbial wheel on this one. It's like using > Some of the trickier ones were: how to read bytes from stdin (sys.stdin.buffer); explicit flushes on sys.stderr (not needed for Python 2.x); a StringIO-like wrapper around BytesIO so I could write() both bytes and Unicode strings containing only ASCII (which is what cStringIO.StringIO did); and a few workarounds for changes in argparse behavior. Yeah, that's the hard part. You can't automate that. But it's what, 3% of your code ? Unless you hate DRY. It's quite fair that bumping an entire project to an non compatible version of your plateform requires you do this. > The trickiest is that I allow the user to pass in an eval-able mathematical expression on the command-line. My code first compiles the expression and checks that all of the global variables are known, in order to generate a more detailed error message than Python itself would. Quite a cool project :) > This is about twice as long as your estimate would suggest. (I'm excluding the auto-generated data table and the third-party package from my LOC count.) Granted. Still, 4 days of porting for an entire project one time in a 25 years old language feels ok in my book. Don't forget that since 2000: - Perl 6 has been late for a decade. - PHP failed its V6 and jumped to V7. - NodeJS forked and merged twice. From this point of view, what Python did is an amazing accomplishment.