4 ms·

Startup time is complex, but base startup only increased about 20ms, and that's being generous. I'll admit Python 3 is still slower at a lot of things. But th

Startup time is complex, but base startup only increased about 20ms, and that's being generous. I'll admit Python 3 is still slower at a lot of things. But that feels like saying your new dog is even worse at math than your old one. The C extension thing isn't Python's fault. It's the job of library and app authors to update. Do we complain that Vulkan has bad SunOS support? This is totally backwards. Could Hinsen (and others) not just version their deps? It's not like people are erasing Python 2 off the internet. If his main worry is reproducibility, he should be doing that anyway. --- I don't want to give the impression I like the whole Python 3 thing. I think it was a pretty big mistake and a huge missed opportunity. I'm very sympathetic to people who had to put in a lot of work for basically no good reason--Python 3 didn't really offer anything significantly better than 2 until... 3.5 (3.4 if you think the first pass at async was useful, I personally don't). But I also find the ballyhooing about it really insufferable. Yeah it was a mistake; Armin Ronacher (as usual) was right. It was also over 11 years ago. Time to forget all about this and build cool stuff, please please please.

tick_tock_tick 7y ago
It takes me 37ms to load hackernews a 20ms start time is embarrassing what is it even doing?
- eesmith 7y ago
  It does a lot of module imports. Most of those probe the filesystem. Try "python -vv -c 'pass'" - I'm only showing the first few dozen lines, and I've trimmed some of the paths for conciseness: % python -vv -c 'pass' import _frozen_importlib # frozen import _imp # builtin import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import '_warnings' # <class '_frozen_importlib.BuiltinImporter'> import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> # installing zipimport hook import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'> # installed zipimport hook import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'> import '_io' # <class '_frozen_importlib.BuiltinImporter'> import 'marshal' # <class '_frozen_importlib.BuiltinImporter'> import 'posix' # <class '_frozen_importlib.BuiltinImporter'> import _thread # previously loaded ('_thread') import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import _weakref # previously loaded ('_weakref') import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> # miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/__init__.py # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc' # trying miniconda3/lib/python3.7/codecs.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/codecs.abi3.so # trying miniconda3/lib/python3.7/codecs.so # trying miniconda3/lib/python3.7/codecs.py # miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc matches miniconda3/lib/python3.7/codecs.py # code object from 'miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc' import '_codecs' # <class '_frozen_importlib.BuiltinImporter'> import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44c90> # trying miniconda3/lib/python3.7/encodings/aliases.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/encodings/aliases.abi3.so # trying miniconda3/lib/python3.7/encodings/aliases.so # trying miniconda3/lib/python3.7/encodings/aliases.py # miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/aliases.py # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc' import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd67d10> import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd440d0> # trying miniconda3/lib/python3.7/encodings/utf_8.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/encodings/utf_8.abi3.so # trying miniconda3/lib/python3.7/encodings/utf_8.so # trying miniconda3/lib/python3.7/encodings/utf_8.py # miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/utf_8.py # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc' import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44bd0> import '_signal' # <class '_frozen_importlib.BuiltinImporter'> # trying miniconda3/lib/python3.7/encodings/latin_1.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/encodings/latin_1.abi3.so # trying miniconda3/lib/python3.7/encodings/latin_1.so # trying miniconda3/lib/python3.7/encodings/latin_1.py # miniconda3/lib/python3.7/encodings/__pycache__/latin_1.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/latin_1 ... many, many more lines omitted ... This can be sped up a lot using a zipimport of the Python standard library, https://docs.python.org/3/library/zipimport.html?highlight=zipimport#module-zipimport https://docs.python.org/3/library/zipimport.html?highlight=z... , where all of the standard library is put in a zipfile. Then there's a file access to get the zip metadata. One of the things that bugged me in Python2 was that every startup imported UserDict: # trying python2.7/UserDict.so # trying python2.7/UserDictmodule.so # trying python2.7/UserDict.py # python2.7/UserDict.pyc matches python2.7/UserDict.py import UserDict # precompiled from python2.7/UserDict.pyc This is because os.environ was an instance of UserDict: % python2.7 -c 'import os; print(os.environ.__class__.__bases__)' (<class UserDict.IterableUserDict at 0x1029b14c8>,) Under Python3 this is spelled collections.abc.MutableMapping: % python3 -c 'import os; print(os.environ.__class__.__bases__)' (<class 'collections.abc.MutableMapping'>,) which triggers its own set of imports: # trying python3.6/collections/abc.cpython-36m-darwin.so # trying python3.6/collections/abc.abi3.so # trying python3.6/collections/abc.so # trying python3.6/collections/abc.py # python3.6/collections/__pycache__/abc.cpython-36.pyc matches python3.6/collections/abc.py # code object from 'python3.6/collections/__pycache__/abc.cpython-36.pyc' import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x103c2ecf8> There's better performance using SDD than a HDD, which is in turn better than using a networked filesystem.
  - tick_tock_tick 7y ago
    I'm guessing startup time just isn't an important goal for cython then? I've know they refused to implements some optimization that would significantly increase complexity but this seem like low hanging fruit?
    - eesmith 7y ago
      Oh, there's been plenty of work to reduce the Python startup cost. It's just hard to fix. I'm not sure the os.environ example I gave is low-hanging fruit now. The collections.abc module might be imported anyway. This is neat! Python 3.7 added the `PYTHONPROFILEIMPORTTIME=1` environment variable to help track down these sorts of import overheads: % env PYTHONPROFILEIMPORTTIME=1 python -c pass import time: self [us] | cumulative | imported package import time: 523 | 523 | zipimport import time: 722 | 722 | _frozen_importlib_external import time: 156 | 156 | _codecs import time: 2254 | 2409 | codecs import time: 1293 | 1293 | encodings.aliases import time: 7192 | 10893 | encodings import time: 1108 | 1108 | encodings.utf_8 import time: 182 | 182 | _signal import time: 1069 | 1069 | encodings.latin_1 import time: 395 | 395 | _abc import time: 1486 | 1881 | abc import time: 1540 | 3420 | io import time: 100 | 100 | _stat import time: 975 | 1075 | stat import time: 1481 | 1481 | genericpath import time: 1734 | 3214 | posixpath import time: 2558 | 2558 | _collections_abc import time: 2234 | 9079 | os import time: 1407 | 1407 | _sitebuiltins import time: 3498 | 3498 | sitecustomize import time: 85 | 85 | usercustomize import time: 4129 | 18196 | site Investigating further, the "import os" which triggered the UserDict/collections.abc is a consequence of "import site". If I use "python -S" then those aren't imported.
eesmith 7y ago
I can't help but interpret your response as "Python 3 is better in every way ... for the ways I think are important." For some of my programs, Python startup time is the main overhead. I avoid NumPy and SciPy if at all possible because they have a huge startup overhead. Some of this is inherent in those packages. NumPy internally imports everything so someone can do "import numpy as np; np.package.subpackage.module.function()" without doing the intermediate imports. This means NumPy is optimized for programmers (especially novice programmers) using NumPy in long-lived processes where startup cost is a negligible overhead. Which isn't all use-cases for numeric computing. 15 years ago I supported a CGI-based web app. It was very important to pull out all the stops (delay imports until needed, use zip packages) because it was easier to do that than to re-write everything for another architecture. The dog does count pretty well after all. > It's the job of library and app authors to update. Why? Linus Torvalds doesn't agree with you, for one. As Hinsen points out, ] Unfortunately, the need for long-term stability is rather specific to scientific users, and not even all of them require it (see e.g. these two tweets by Titus Brown). So while Python 3 is probably a step forward for most Python users, it’s mostly a calamity for computational science. Some scientific code has been able to run unchanged since the 1970s, through multiple new Fortran language releases. Now, yes, I know the reasons for the changes to Python. I know the funding and organizational realities. But why not recognize that for some situations Python 3 is not better? Hinsen also comments on your proposal: ] The implication is that breaking changes in the infrastructure layers are OK and must be absorbed by the maintainers of layers 3 and 4. In view of what I just said about layer 4, it should be obvious that I don’t agree at all with this point of view. But even concerning layer 3, I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter. > Could Hinsen (and others) not just version their deps? He addresses that, I think. One of the other commenters gives a more complete reply at https://metarabbit.wordpress.com/2017/11/18/numpy-scipy-backwards-stability-debate-and-why-freezing-versions-is-not-the-solution/ https://metarabbit.wordpress.com/2017/11/18/numpy-scipy-back... ending "Freezing the versions solves some problems, but does not solve the whole issue of backwards compatibility.". > Time to forget all about this and build cool stuff, please please please. I'll quote Hinsen again "I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter." Your implicit statement is that mmtk (Hinsen's code base) isn't "cool stuff". Why? Simply because it's old, or because you don't know about it or need it? What other cool old stuff will die because it's part of a community without the resources to update? Instead, accept that that loss is part of the trade-offs, be empathetic to those who suffer, and bear those lessons in mind for future work you do.

tick_tock_tick 7y ago

It takes me 37ms to load hackernews a 20ms start time is embarrassing what is it even doing?

eesmith 7y ago
It does a lot of module imports. Most of those probe the filesystem. Try "python -vv -c 'pass'" - I'm only showing the first few dozen lines, and I've trimmed some of the paths for conciseness: % python -vv -c 'pass' import _frozen_importlib # frozen import _imp # builtin import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import '_warnings' # <class '_frozen_importlib.BuiltinImporter'> import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> # installing zipimport hook import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'> # installed zipimport hook import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'> import '_io' # <class '_frozen_importlib.BuiltinImporter'> import 'marshal' # <class '_frozen_importlib.BuiltinImporter'> import 'posix' # <class '_frozen_importlib.BuiltinImporter'> import _thread # previously loaded ('_thread') import '_thread' # <class '_frozen_importlib.BuiltinImporter'> import _weakref # previously loaded ('_weakref') import '_weakref' # <class '_frozen_importlib.BuiltinImporter'> # miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/__init__.py # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc' # trying miniconda3/lib/python3.7/codecs.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/codecs.abi3.so # trying miniconda3/lib/python3.7/codecs.so # trying miniconda3/lib/python3.7/codecs.py # miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc matches miniconda3/lib/python3.7/codecs.py # code object from 'miniconda3/lib/python3.7/__pycache__/codecs.cpython-37.pyc' import '_codecs' # <class '_frozen_importlib.BuiltinImporter'> import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44c90> # trying miniconda3/lib/python3.7/encodings/aliases.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/encodings/aliases.abi3.so # trying miniconda3/lib/python3.7/encodings/aliases.so # trying miniconda3/lib/python3.7/encodings/aliases.py # miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/aliases.py # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc' import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd67d10> import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd440d0> # trying miniconda3/lib/python3.7/encodings/utf_8.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/encodings/utf_8.abi3.so # trying miniconda3/lib/python3.7/encodings/utf_8.so # trying miniconda3/lib/python3.7/encodings/utf_8.py # miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/utf_8.py # code object from 'miniconda3/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc' import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x10dd44bd0> import '_signal' # <class '_frozen_importlib.BuiltinImporter'> # trying miniconda3/lib/python3.7/encodings/latin_1.cpython-37m-darwin.so # trying miniconda3/lib/python3.7/encodings/latin_1.abi3.so # trying miniconda3/lib/python3.7/encodings/latin_1.so # trying miniconda3/lib/python3.7/encodings/latin_1.py # miniconda3/lib/python3.7/encodings/__pycache__/latin_1.cpython-37.pyc matches miniconda3/lib/python3.7/encodings/latin_1 ... many, many more lines omitted ... This can be sped up a lot using a zipimport of the Python standard library, https://docs.python.org/3/library/zipimport.html?highlight=zipimport#module-zipimport https://docs.python.org/3/library/zipimport.html?highlight=z... , where all of the standard library is put in a zipfile. Then there's a file access to get the zip metadata. One of the things that bugged me in Python2 was that every startup imported UserDict: # trying python2.7/UserDict.so # trying python2.7/UserDictmodule.so # trying python2.7/UserDict.py # python2.7/UserDict.pyc matches python2.7/UserDict.py import UserDict # precompiled from python2.7/UserDict.pyc This is because os.environ was an instance of UserDict: % python2.7 -c 'import os; print(os.environ.__class__.__bases__)' (<class UserDict.IterableUserDict at 0x1029b14c8>,) Under Python3 this is spelled collections.abc.MutableMapping: % python3 -c 'import os; print(os.environ.__class__.__bases__)' (<class 'collections.abc.MutableMapping'>,) which triggers its own set of imports: # trying python3.6/collections/abc.cpython-36m-darwin.so # trying python3.6/collections/abc.abi3.so # trying python3.6/collections/abc.so # trying python3.6/collections/abc.py # python3.6/collections/__pycache__/abc.cpython-36.pyc matches python3.6/collections/abc.py # code object from 'python3.6/collections/__pycache__/abc.cpython-36.pyc' import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x103c2ecf8> There's better performance using SDD than a HDD, which is in turn better than using a networked filesystem.
- tick_tock_tick 7y ago
  I'm guessing startup time just isn't an important goal for cython then? I've know they refused to implements some optimization that would significantly increase complexity but this seem like low hanging fruit?
  - eesmith 7y ago
    Oh, there's been plenty of work to reduce the Python startup cost. It's just hard to fix. I'm not sure the os.environ example I gave is low-hanging fruit now. The collections.abc module might be imported anyway. This is neat! Python 3.7 added the `PYTHONPROFILEIMPORTTIME=1` environment variable to help track down these sorts of import overheads: % env PYTHONPROFILEIMPORTTIME=1 python -c pass import time: self [us] | cumulative | imported package import time: 523 | 523 | zipimport import time: 722 | 722 | _frozen_importlib_external import time: 156 | 156 | _codecs import time: 2254 | 2409 | codecs import time: 1293 | 1293 | encodings.aliases import time: 7192 | 10893 | encodings import time: 1108 | 1108 | encodings.utf_8 import time: 182 | 182 | _signal import time: 1069 | 1069 | encodings.latin_1 import time: 395 | 395 | _abc import time: 1486 | 1881 | abc import time: 1540 | 3420 | io import time: 100 | 100 | _stat import time: 975 | 1075 | stat import time: 1481 | 1481 | genericpath import time: 1734 | 3214 | posixpath import time: 2558 | 2558 | _collections_abc import time: 2234 | 9079 | os import time: 1407 | 1407 | _sitebuiltins import time: 3498 | 3498 | sitecustomize import time: 85 | 85 | usercustomize import time: 4129 | 18196 | site Investigating further, the "import os" which triggered the UserDict/collections.abc is a consequence of "import site". If I use "python -S" then those aren't imported.

eesmith 7y ago

I can't help but interpret your response as "Python 3 is better in every way ... for the ways I think are important." For some of my programs, Python startup time is the main overhead. I avoid NumPy and SciPy if at all possible because they have a huge startup overhead. Some of this is inherent in those packages. NumPy internally imports everything so someone can do "import numpy as np; np.package.subpackage.module.function()" without doing the intermediate imports. This means NumPy is optimized for programmers (especially novice programmers) using NumPy in long-lived processes where startup cost is a negligible overhead. Which isn't all use-cases for numeric computing. 15 years ago I supported a CGI-based web app. It was very important to pull out all the stops (delay imports until needed, use zip packages) because it was easier to do that than to re-write everything for another architecture. The dog does count pretty well after all. > It's the job of library and app authors to update. Why? Linus Torvalds doesn't agree with you, for one. As Hinsen points out, ] Unfortunately, the need for long-term stability is rather specific to scientific users, and not even all of them require it (see e.g. these two tweets by Titus Brown). So while Python 3 is probably a step forward for most Python users, it’s mostly a calamity for computational science. Some scientific code has been able to run unchanged since the 1970s, through multiple new Fortran language releases. Now, yes, I know the reasons for the changes to Python. I know the funding and organizational realities. But why not recognize that for some situations Python 3 is not better? Hinsen also comments on your proposal: ] The implication is that breaking changes in the infrastructure layers are OK and must be absorbed by the maintainers of layers 3 and 4. In view of what I just said about layer 4, it should be obvious that I don’t agree at all with this point of view. But even concerning layer 3, I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter. > Could Hinsen (and others) not just version their deps? He addresses that, I think. One of the other commenters gives a more complete reply at https://metarabbit.wordpress.com/2017/11/18/numpy-scipy-backwards-stability-debate-and-why-freezing-versions-is-not-the-solution/ https://metarabbit.wordpress.com/2017/11/18/numpy-scipy-back... ending "Freezing the versions solves some problems, but does not solve the whole issue of backwards compatibility.". > Time to forget all about this and build cool stuff, please please please. I'll quote Hinsen again "I find it a bit arrogant. The message to research communities with weaker code development traditions, and thus fewer resources, is that their work doesn’t matter." Your implicit statement is that mmtk (Hinsen's code base) isn't "cool stuff". Why? Simply because it's old, or because you don't know about it or need it? What other cool old stuff will die because it's part of a community without the resources to update? Instead, accept that that loss is part of the trade-offs, be empathetic to those who suffer, and bear those lessons in mind for future work you do.