6 ms·

Speed improvements in Python 3.7

78 points by orf 9y ago

singularity2001 9y ago
did they further improve backwards compatibility? Neither trolling nor beating dead horses: I recently wanted to do some screen capture and needed tkinter, which of course has different imports for py2 and py3. adding library aliases would help. I honestly wished people would use transitional deprecation warnings more often to educate about changed apis. As long as 50% of tensorflow projects are still in Python 2 it is always a pain to encounter Python 3. ``` try: from urllib2 import urlopen from urllib import urlretrieve except ImportError: from urllib.request import urlopen, urlretrieve ```
- orf 9y ago
  Have you tried the 'six' library? It does what you describe, as well as paving over many py2/py3 differences
  - singularity2001 9y ago
    Tangentially heard of it. Thanks, will try. Does it add py2 conventions to py3 or Does it add py3 conventions to py2 ?
    - viraptor 9y ago
      Neither. It adds a wrapper in `six.moves` to various modules which moved between py2/3. It will not automatically fix existing code, but it does allow you to use one import line.
    - orf 9y ago
      Neither, it just let's you write code that runs on 2 and 3 quite easily, by hiding import differences and changes to builtins. It's worth it if you are distributing code that has to run on 3 and 2, but if your upgrading your existing 2 code base it's not that great.
- kevin_thibedeau 9y ago
  Is 15 years to prepare still not long enough to migrate your code incrementally? You can get Py2 into a nearly-ready-for-Py3 state with just a little effort stretched over whatever horizon you desire. Beating a dead horse doesn't do anything about your stagnant code base.
  - singularity2001 9y ago
    it's not about migrating old code bases, it's about encountering other peoples tensorflow/ubuntu code and often ending up in unnecessary(!) hickups.
  - orf 9y ago
    I used to be quite militant about this, but you have to get that often business goals don't align. You can often do it on the side, during other changes, but convincing your manager that it's worth it is a hard problem.
    - brylie 9y ago
      NP-hard.
  - mjevans 9y ago
    No, because they did a bunch of things that make automated porting not work well and provides a ton of developer friction. I'm willing to live with Python's space related insanity. Mandating that indenting is meaningful. Mandating a best practice of using spaces instead of tabs for indent* (which makes tab width matter). Mandating that a tab equals 4 spaces instead of 8. (Honestly, I REALLY hate this, most of the Windows developers are happy with the space usage koolaid, let us have a default of tab equals 8 spaces of indent.) Those are ALL, REALLY ANNOYING, but lint tools can save me. What will bite me for as long as Python 3 exists, and maybe Python 4, is how they handle Unicode support and encoded outputs. Pretty much the ENTIRE rest of the computing world does one of two things. 1) Doesn't care what the encoding is, and flings around non-validated 8 bit sequences of data. 2) Does care, and defaults to assuming well formed UTF-8 data-streams for text data, BUT still doesn't do anything to enforce it UNLESS ASKED. Most importantly invalid data streams just continue flowing through the processing tools until they reach a point where a human has designated they're willing to care and /do/ something about it. If Python 4 wants to 'fix' Unicode support: * Use a binary sequence base storage class with metadata tags. > Length (in bytes, in 'display length', and maybe in 'codepoint length') > encoding* (including if normalized and how so) > If the encoding has been validated. * Support automatic coercion between standard encodings * Allow user defined helpers for custom conversions * Make all input/output default to UTF-8** but permit raw output of such types without validation or re-encoding, just like every other UNIX tool that knows what an encoding is.
viraptor 9y ago
For the new 3.7 there's also gc.freeze (https://bugs.python.org/issue31558 https://bugs.python.org/issue31558) which will be great for many web projects.
- orf 9y ago
  https://docs.python.org/3.7/library/gc.html#gc.freeze https://docs.python.org/3.7/library/gc.html#gc.freeze Documentation link for anyone that is interested
  - rojoca 9y ago
    Also the source(?) of this change from instagram: https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://engineering.instagram.com/copy-on-write-friendly-pyt...
  - wyldfire 9y ago
    > Also collection before a POSIX fork() call may free pages for future allocation which can cause copy-on-write too so it’s advised to disable gc in master process and freeze before fork and enable gc in child process. I guess it would be nice to know if optimizations like this are already covered in the high-level libraries like `multiprocessing`.
    - viraptor 9y ago
      They aren't / shouldn't be. This is something you need to opt into specifically because it changes behaviour. For example if you have done resource still open, calling gc.freeze will prevent it from ever cleaning up. But web frameworks should implement it.
hartator 9y ago
Is it faster than 2.7 now?
- ubernostrum 9y ago
  Python 3.0 was slower than Python 2.7, largely because some new modules were initially implemented in pure Python in order to get the APIs out there for people to experiment with. They're since been replaced with implementations in C. Python 3 passed Python 2 on performance years ago.
  - icegreentea2 9y ago
    Except at interpreter startup.
jxub 9y ago
Python is the best (as in most productive and happy) language for me bar none. It feels great to know that speed, the only major drawback, is being seriously improved in each release. Here's to the future!
- billsix 9y ago
  If your looking for a nice gui framework for python, check out my pyNuklear! https://github.com/billsix/pyNuklear https://github.com/billsix/pyNuklear
btown 9y ago
Since one of the mentioned improvements is performance of the builtin regular expression library... for anyone curious, there's a drop-in replacement, re2, that leverages a Google library to provide linear time guarantees for arbitrarily complex or user-provided regexes. (You lose lookahead assertions, but most use cases for them can just be handled in pre- or post-processing.) It's an incredibly handy tool that lets us confidently use complex regexes in production. https://pypi.python.org/pypi/re2/ https://pypi.python.org/pypi/re2/ https://github.com/google/re2/wiki/WhyRE2 https://github.com/google/re2/wiki/WhyRE2 > RE2 was designed and implemented with an explicit goal of being able to handle regular expressions from untrusted users without risk. One of its primary guarantees is that the match time is linear in the length of the input string. It was also written with production concerns in mind: the parser, the compiler and the execution engines limit their memory usage by working within a configurable budget – failing gracefully when exhausted – and they avoid stack overflow by eschewing recursion. Really interesting articles in the second link detailing how this works!
- tyingq 9y ago
  Great observation, but re2 does have tradeoffs to gain that performance. Backreferences, for example: https://github.com/google/re2/issues/101 https://github.com/google/re2/issues/101
  - Blackthorn 9y ago
    Backreferences are one of the items that makes regexes non-linear, which defeats the point of being able to trust arbitrary regexes to have good performance.
  - btown 9y ago
    Absolutely. But for most common uses of backreferences, this is largely mitigated by having a post-regex step that compares capturing groups for equality, which induces overhead also linear in the length of the string. Of course this falls short of true backreference support, but for tasks where we do not expect potential matches to overlap, it gets the job done.

78 points by orf 9y ago

singularity2001 9y ago

did they further improve backwards compatibility? Neither trolling nor beating dead horses: I recently wanted to do some screen capture and needed tkinter, which of course has different imports for py2 and py3. adding library aliases would help. I honestly wished people would use transitional deprecation warnings more often to educate about changed apis. As long as 50% of tensorflow projects are still in Python 2 it is always a pain to encounter Python 3. ``` try: from urllib2 import urlopen from urllib import urlretrieve except ImportError: from urllib.request import urlopen, urlretrieve ```

orf 9y ago
Have you tried the 'six' library? It does what you describe, as well as paving over many py2/py3 differences
- singularity2001 9y ago
  Tangentially heard of it. Thanks, will try. Does it add py2 conventions to py3 or Does it add py3 conventions to py2 ?
  - viraptor 9y ago
    Neither. It adds a wrapper in `six.moves` to various modules which moved between py2/3. It will not automatically fix existing code, but it does allow you to use one import line.
  - orf 9y ago
    Neither, it just let's you write code that runs on 2 and 3 quite easily, by hiding import differences and changes to builtins. It's worth it if you are distributing code that has to run on 3 and 2, but if your upgrading your existing 2 code base it's not that great.
kevin_thibedeau 9y ago
Is 15 years to prepare still not long enough to migrate your code incrementally? You can get Py2 into a nearly-ready-for-Py3 state with just a little effort stretched over whatever horizon you desire. Beating a dead horse doesn't do anything about your stagnant code base.
- singularity2001 9y ago
  it's not about migrating old code bases, it's about encountering other peoples tensorflow/ubuntu code and often ending up in unnecessary(!) hickups.
- orf 9y ago
  I used to be quite militant about this, but you have to get that often business goals don't align. You can often do it on the side, during other changes, but convincing your manager that it's worth it is a hard problem.
  - brylie 9y ago
    NP-hard.
- mjevans 9y ago
  No, because they did a bunch of things that make automated porting not work well and provides a ton of developer friction. I'm willing to live with Python's space related insanity. Mandating that indenting is meaningful. Mandating a best practice of using spaces instead of tabs for indent* (which makes tab width matter). Mandating that a tab equals 4 spaces instead of 8. (Honestly, I REALLY hate this, most of the Windows developers are happy with the space usage koolaid, let us have a default of tab equals 8 spaces of indent.) Those are ALL, REALLY ANNOYING, but lint tools can save me. What will bite me for as long as Python 3 exists, and maybe Python 4, is how they handle Unicode support and encoded outputs. Pretty much the ENTIRE rest of the computing world does one of two things. 1) Doesn't care what the encoding is, and flings around non-validated 8 bit sequences of data. 2) Does care, and defaults to assuming well formed UTF-8 data-streams for text data, BUT still doesn't do anything to enforce it UNLESS ASKED. Most importantly invalid data streams just continue flowing through the processing tools until they reach a point where a human has designated they're willing to care and /do/ something about it. If Python 4 wants to 'fix' Unicode support: * Use a binary sequence base storage class with metadata tags. > Length (in bytes, in 'display length', and maybe in 'codepoint length') > encoding* (including if normalized and how so) > If the encoding has been validated. * Support automatic coercion between standard encodings * Allow user defined helpers for custom conversions * Make all input/output default to UTF-8** but permit raw output of such types without validation or re-encoding, just like every other UNIX tool that knows what an encoding is.

viraptor 9y ago

For the new 3.7 there's also gc.freeze (https://bugs.python.org/issue31558 https://bugs.python.org/issue31558) which will be great for many web projects.

orf 9y ago
https://docs.python.org/3.7/library/gc.html#gc.freeze https://docs.python.org/3.7/library/gc.html#gc.freeze Documentation link for anyone that is interested
- rojoca 9y ago
  Also the source(?) of this change from instagram: https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://engineering.instagram.com/copy-on-write-friendly-pyt...
- wyldfire 9y ago
  > Also collection before a POSIX fork() call may free pages for future allocation which can cause copy-on-write too so it’s advised to disable gc in master process and freeze before fork and enable gc in child process. I guess it would be nice to know if optimizations like this are already covered in the high-level libraries like `multiprocessing`.
  - viraptor 9y ago
    They aren't / shouldn't be. This is something you need to opt into specifically because it changes behaviour. For example if you have done resource still open, calling gc.freeze will prevent it from ever cleaning up. But web frameworks should implement it.

hartator 9y ago

Is it faster than 2.7 now?

ubernostrum 9y ago
Python 3.0 was slower than Python 2.7, largely because some new modules were initially implemented in pure Python in order to get the APIs out there for people to experiment with. They're since been replaced with implementations in C. Python 3 passed Python 2 on performance years ago.
- icegreentea2 9y ago
  Except at interpreter startup.

jxub 9y ago

Python is the best (as in most productive and happy) language for me bar none. It feels great to know that speed, the only major drawback, is being seriously improved in each release. Here's to the future!

billsix 9y ago
If your looking for a nice gui framework for python, check out my pyNuklear! https://github.com/billsix/pyNuklear https://github.com/billsix/pyNuklear

btown 9y ago

Since one of the mentioned improvements is performance of the builtin regular expression library... for anyone curious, there's a drop-in replacement, re2, that leverages a Google library to provide linear time guarantees for arbitrarily complex or user-provided regexes. (You lose lookahead assertions, but most use cases for them can just be handled in pre- or post-processing.) It's an incredibly handy tool that lets us confidently use complex regexes in production. https://pypi.python.org/pypi/re2/ https://pypi.python.org/pypi/re2/ https://github.com/google/re2/wiki/WhyRE2 https://github.com/google/re2/wiki/WhyRE2 > RE2 was designed and implemented with an explicit goal of being able to handle regular expressions from untrusted users without risk. One of its primary guarantees is that the match time is linear in the length of the input string. It was also written with production concerns in mind: the parser, the compiler and the execution engines limit their memory usage by working within a configurable budget – failing gracefully when exhausted – and they avoid stack overflow by eschewing recursion. Really interesting articles in the second link detailing how this works!

tyingq 9y ago
Great observation, but re2 does have tradeoffs to gain that performance. Backreferences, for example: https://github.com/google/re2/issues/101 https://github.com/google/re2/issues/101
- Blackthorn 9y ago
  Backreferences are one of the items that makes regexes non-linear, which defeats the point of being able to trust arbitrary regexes to have good performance.
- btown 9y ago
  Absolutely. But for most common uses of backreferences, this is largely mitigated by having a post-regex step that compares capturing groups for equality, which induces overhead also linear in the length of the string. Of course this falls short of true backreference support, but for tasks where we do not expect potential matches to overlap, it gets the job done.