26 ms·

Reducing Python's startup time

219 points by vanni 9y ago

mikepurvis 9y ago
One of the comments below the article mentions this, but in my experience on Linux and Mac, by far the biggest culprit is pkg_resources and its nasty habit of spidering the filesystem: https://github.com/pypa/setuptools/issues/926 https://github.com/pypa/setuptools/issues/926 There are hacks to get around it, but it's a deep hole.
- orf 9y ago
  People often complain that using `entry_points` in setup.py is slow. I did some profiling and it came down to it importing some `pip` submodule (pip.vendor.something), which imported basically the whole of pip. :(
sillysaurus3 9y ago
$ time python -c 'print(1)' 1 real 0m0.067s user 0m0.015s sys 0m0.025s Is it slow? That was Python 2.7.10, and for me Python 3.6 is similarly fast.
- JasonSage 9y ago
  If you had read the article, you would see that the topic of discussion is reducing startup time when many modules are loaded due to the slow nature of `namedtuple`.
- agumonkey 9y ago
  Maybe only slow when importing. With mainstream pip it's often that you leverage imports a lot.
  - deleted 9y ago
    [deleted]
- benhoyt 9y ago
  On my system (macOS) I see a minimum of 0m0.034 for Python 3.6.1 and a minimum of 0m0.022 for Python 2.7.13 -- so Python 3.6 is 50% slower. This is a somewhat contrived example, however. As the article notes, it's worse when you have a lot of imports (and I don't know how that'd differ between versions 2 and 3). I remember talking to some folks at Google who work on the Google Cloud CLI, and they said that Python startup time is a constant problem for their (large) codebase. Especially when it's used for tab completion, because it might run every time you press tab, and the user wants instant results.
  - sillysaurus3 9y ago
    Aha, tab completion makes sense. Thanks!
  - mikepurvis 9y ago
    We're dealing with this on a CLI build tools project I hack on (catkin_tools). It uses plugin discovery based on entry_points, and short of caching plugins in a homedir file, it's basically impossible to make it fast enough to use with something like argcomplete. Relevant ticket: https://github.com/catkin/catkin_tools/issues/297 https://github.com/catkin/catkin_tools/issues/297
  - smcl 9y ago
    Why on earth would they think invoking a new interpreter each time someone hits "tab" would do anything except perform suboptimally?
- sreque 9y ago
  I spent a lot of time analyzing the startup time of Java, Python, and Node.js. For any real-world program, startup time is entirely dominated by time taken to load code into memory, which in python is import statements. Also, as slow as Python 2.7 is, Node.js, at least when I measured it, was twice as slow.
- draegtun 9y ago
  Here's some benchmarks i've come across - http://onetom.rebol.info/2013/08/18/startup-time-of-interpreters/ http://onetom.rebol.info/2013/08/18/startup-time-of-interpre...
- paulsutter 9y ago
  The discussion is about import speed, which is a real problem: $ time python3 -c 'import tensorflow' real 0m4.522s user 0m1.712s sys 0m0.466s
fractalb 9y ago
If the namedtuple is used so heavily, why not make it a part of built-in functions?
Walkman 9y ago
At my company we changed the interaction between two daemons by calling a command line script instead. It caused a 300ms delay in every single AJAX call and broke a dozens of end-to-end tests. I just tried with Python 2.7 and 3.6 and the hello world startup time is 100ms which is awful lot.
sevensor 9y ago
I'd been wondering why _source was in there. I'm a little surprised that Hettinger thinks it's so important to retain it. It's a neat demonstration of how to make a new data type, and I like that it's so straightforward, but I don't buy the argument that slowing down the interpreter is important for didactic reasons. Edit: I'm talking about namedtuple, the implementation of which is slow because namedtuple() has to construct _source for every new namedtuple type. Which was clear from the article but not from my comment.
- dr_zoidberg 9y ago
  Actually your edit made it clearer from both. I had the feeling that namedtuply was constructing _source, but it wasn't obvious in the article either. So thanks for explaining, both your comment and the article.
- mnarayan01 9y ago
  Yea I'm not getting the need to keep _source in collections.namedtuple at all. For teaching purposes, you can just present the existing implementation...I don't see what purpose having that teaching code be in the core distribution serves?
  - xapata 9y ago
    Not everyone is familiar with looking at the core source, or even reading the docs.
  - takeda 9y ago
    In case you overlooked, Raymond H explained it in another thread[1]. [1] https://news.ycombinator.com/item?id=15135569 https://news.ycombinator.com/item?id=15135569
- raymondh 9y ago
  One purpose of _source is to make the namedtuple self-documenting -- that has been a considerable benefit over the years and has resulted in many fewer questions and issues than we usually get about our other tools. Another purpose of _source is so that you can write it out to a module. This lets you pre-compute the named tuple for speed and it makes it possible to easily Cythonize it or to customize it (in particular, people like to take out the error checking). Since the namedtuple() factory function was already creating the source string as part of its operation, just storing it and exposing the attribute had nearly zero additional cost. That said, don't worry about it, next week I plan to make this a lazily generated attribute so you all can sleep soundly at night :-)
eesmith 9y ago
I have grown to seriously dislike 'namedtuple'. It seems great, and it's so easy to use, but there's so few places it can be used without dragging in future backwards-compatibility baggage. Let's say you have a light-weight object, like a "full name" which can be represented with a "forename" and an "surname". (Yes, there are many names which don't fit this category. This is a toy example.) So you write: from collections import namedtuple FullName = namedtuple("FullName", "forename surname") name = FullName("Edward", "Smith") So simple! So clean! No "self.forename = forename" or need to make your own __repr__. Except you also inherit the tuple behavior, because this isn't really light-weight object but a ... let's call it a heavy-weight tuple. >>> name[0] 'Edward' >>> name[1] 'Smith' This mean people may (okay, are likely) do things like: forename, surname = name or even for forename, surname in names: ... Even if that's not what you want in your API. If you care about backwards compatibility, then if you replace the namedtuple with its own class then you'll have to re-implement __getitem__, _replace, __len__, and count - methods which make no sense for a simple name class. Or, if you decided to change the API, to make it be: FullName = namedtuple("FullName", "forename middlename surname suffix") then you'll again break code which used the full indexing API that you originally implicitly promised, simply by saying this is a namedtuple. 'namedtuple' has one good use - migration from a tuple object to a real object while allowing backwards compatability to the tuple API. For example, the old tuples once returned from os.stat() and time.gmtime(). It's also fine if it will only be used by code where you have full control over its use and can either prevent inappropriate API use like destructuring assignment, or can update all uses if you want to change the API. For example, a light-weight object used in a test-suite or internal helper function. Otherwise, don't use it. Spend the extra few minutes so in the future you don't have to worry about backwards compatibility issues.
- sevensor 9y ago
  I disagree -- namedtuple has a crucial property that I've found incredibly useful. Namedtuples are immutable. This makes it impossible (well, not really, but hard) for a namedtuple to change its state without telling me about it, because if it changes state it's not the same namedtuple any more. I agree that namedtuple being a sequence type is problematic, but deciding not to use a lightweight immutable data structure because your colleagues might use it badly is throwing out the baby with the bathwater.
jxcl 9y ago
Raymond Hettinger, if I remember correctly, is the person who originally wrote NamedTuple, and he's not shy about reminding people during his talks at conferences. I wonder if his pride or other personal feelings about that particular module could be clouding his judgment over whether or not to optimize it?
- mjburgess 9y ago
  I think it's more to do with him primarily teaching python (iirc) rather than working on large systems with it. It's the same strain of rant Zed Shaw had of Py3, ie., that its harder to teach than Py2 and thef. bad. Crazy. EDIT: OK, if not primarily teaching, spending a lot of his time teaching it. That's a big part of his motivation + worth knowing that when assessing his arguments.
  - WoodenChair 9y ago
    > It's the same strain of rant Zed Shaw had of Py3, ie., that its harder to teach than Py2 and thef. bad. Crazy. Why is wanting something to be easy to teach crazy?
    - mjburgess 9y ago
      Programming languages are tools for developers. They are taught in the service of software development. Optimizing for teaching has it backwards. It can be a factor: but to place it higher than startup time is prioritising the one-off teaching experience of a few over the repeated development experience of many. For a tool for development. Crazy, no?
      - hueving 9y ago
        Ability to teach is going to be directly correlated with ability to maintain as a business. It's harder to hire people when the bar for learning the skill is higher. The vast majority of software development is not hitting any kind of scaling or performance bottlenecks. So optimizing for simplicity isn't as crazy as you think.
NelsonMinar 9y ago
namedtuple is particularly weird. When you create a new type of namedtuple, Python creates the source code for a new class from a string template and passes it to exec() to build the class. It's a clever and relatively straightforward bit of metaprogramming, but it's not a suprise it's not fast. https://hg.python.org/cpython/file/b14308524cff/Lib/collections/__init__.py#l236 https://hg.python.org/cpython/file/b14308524cff/Lib/collecti... (BTW if you need a quick and dirty way to make Python start up faster, see if you can use the -S flag for your application. It skips the site-specific initialization and has a significant impact on startup times.)
- cjbillington 9y ago
  Really? Huh. I would have expected a metaclass.
  - rajathagasthya 9y ago
    While using a metaclass is indeed cleaner, it's slower than generating code on the fly and exec-ing it. This video talks about it: https://youtu.be/sPiWg5jSoZI?t=1h37m18s https://youtu.be/sPiWg5jSoZI?t=1h37m18s
  - Xophmeister 9y ago
    I guess that's how Python 3.6's typing.NamedTuple works, which is frankly a much nicer syntax for named tuples.
    - JelleZijlstra 9y ago
      It's not actually; typing.NamedTuple is just a wrapper that calls the original collections.namedtuple. It does provide some additional features, like default values and methods.
- fulafel 9y ago
  Cool use of eval. I wonder if there are metaprogramming aimed libraries for generating eval'd code.
wyldfire 9y ago
BDFL says: > Concluding, I think we should move on from the original implementation and optimize the heck out of namedtuple. I remember reading this on the mailing list and thinking "yes, namedtuple deserves optimization because it's an excellent resource". For my use cases, I don't think I notice Python's startup time. When I write Python code it's usually not performance critical. When I do write performance-critical Python code, I usually care about total runtime and PyPy is usually a win here. Aside: if you use PyPy, you should probably be using namedtuples. IIRC it models those much better than it does dicts. And IMO namedtuples model many common data structures better than dicts do. Just because I don't feel the pain of Python's startup time doesn't mean we shouldn't try to optimize it. I think I find my way into some esoteric Python/CPython corners but FWIW I've never needed namedtuple's "_source" attr.
- mikepurvis 9y ago
  Ruby's idea of a hash where the keys are colon symbols seems to really strike a sweet spot here— it's fast and storage-efficient like a tuple, but has the flexibility of a dict.
  - sevensor 9y ago
    One big advantage of using namedtuples, aside from the immutability I mentioned elsewhere, is that you can rely on a namedtuple having particular fields. So if you're using dictionaries, you have to worry about KeyError wherever the dict is used, but if you're using namedtuples, you can't create them without assigning values to all of the fields. Dictionaries are still great for their intended purpose -- to represent a mapping -- but namedtuples are nicer if what you really want is a lightweight object.
    - mikepurvis 9y ago
      Setting default values for the namedtuple requires subclassing it, though, which is its own little bag of fun. See all the wailing and gnashing of teeth on SO: https://stackoverflow.com/search?q=namedtuple+subclass https://stackoverflow.com/search?q=namedtuple+subclass
epberry 9y ago
Ah, slow startups on unittests is a big headache. This is especially apparent on big projects like Django where you might as well leave your desk and grab coffee when the unittests start, even for small projects. That said, I can't help but think that this is just an inevitable tradeoff of Python. You want great expressive power? You have to sacrifice speed. Awesome constructs like namedtuple make this worth it imo.
- mfukar 9y ago
  I can't help but think if a one-time initialisation cost is bogging a process down, there are bigger problems than that initialisation cost.
  - a_t48 9y ago
    If you're running hundreds of very small tests, that one time initialization could take up a large portion of time.
    - mfukar 9y ago
      So...maybe...consolidate the process of executing hundreds of little things? How tiny do the steps in this thought process need to be? We can go on all night.
      - a_t48 9y ago
        It's cleanest if each test runs in a new execution context, so that tests can't effect one another.
        mfukar 9y ago
        That's a naive solution to a tiny problem.
  - coldtea 9y ago
    There are processes that you might need to run frequently. I gave an example above: showing the output of a Python script on your shell prompt (e.g. mercurial status).
jordigh 9y ago
Mercurial did a lot of work to reduce startup time via its demandimport module. https://www.mercurial-scm.org/repo/hg/file/tip/hgdemandimport/demandimportpy2.py#l1 https://www.mercurial-scm.org/repo/hg/file/tip/hgdemandimpor... Basically, it's a lazy loading of all imports. You can write `import foo` but it won't actually be imported until you do `foo.whatever()`. It's a crutch, and it's true that hg still pays the overall startup cost of Python. However, even with "45 times slower than git", the situation is not so dire: jordi@chloe:~$ time hg --version Mercurial Distributed SCM (version 4.2.2) (see https://mercurial-scm.org for more information) Copyright (C) 2005-2017 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. real 0m0.096s user 0m0.084s sys 0m0.008s jordi@chloe:~$ time git --version git version 2.1.4 real 0m0.001s user 0m0.000s sys 0m0.000s I really can barely perceive a difference between 0.096s and 0.001s. Since it really is a one-time startup cost, it's not like we can even say that this difference accumulates and that hg is overall 45 times slower than git. Pierre-Yves also has an interesting talk about all of the tricks that have to go into hg in order to make it fast with Python. There's stuff like doing `method = object.method; method()` instead of doing `object.method()` over and over again to avoid paying the cost of method lookup and so forth: https://2015.pycon.ca/en/schedule/53/ https://2015.pycon.ca/en/schedule/53/
- coldtea 9y ago
  >I really can barely perceive a difference between 0.096s and 0.001s. Since it really is a one-time startup cost, it's not like we can even say that this difference accumulates and that hg is overall 45 times slower than git. It's bad for many things -- like showing VCS status on your shell prompt or IDE status line, where you repeatedly start hg. 100ms+ delay every time you hit enter in your shell is noticeable (and that's just the version, now doing an actual "dirty" check takes more).
  - Too 9y ago
    Mercurial has something called command server[1], it's essentially a daemon listening on sockets to receive commands, designed just to avoid the cost of startup. I had to use it once in a CI system where we did invoke hg thousands of times for some reason i can't even remember. That can also be used for frequent polling like shell status. [1] https://www.mercurial-scm.org/wiki/CommandServer https://www.mercurial-scm.org/wiki/CommandServer
chubot 9y ago
I agree with optimizing Python's startup time, and I agree that namedtuple is weird and should be changed. But I doubt that the namedtuple change will noticeably decrease startup time for most applications (after having experience with this problem for 10+ years). From a comment over 3 years ago: [Python interpreter startup does] random access I/O (the slowest thing your computer can do) proportional to (large constant factor) x (num imports in program) x (length of PYTHONPATH). https://news.ycombinator.com/item?id=7842873 https://news.ycombinator.com/item?id=7842873 I don't think that instantiating a Python parser 100 times for exec() is going to compare to that. I guess the difference is that I'm thinking about the cold cache case -- namedtuple might be noticeable in the warm cache case. And there are many many command line tools that start slow because they're written in Python, not just Mercurial. Mercurial is actually one of the best because they care about it (demandimport) and they don't have too many dependencies. IIRC some of the Google Cloud tools take 500-1000+ ms to start because they are crawling tons of dependencies with long PYTHONPATH.
- sametmax 9y ago
  Does this mean the import mechanism caches loaded module, but not where to find non imported things ? I would expect that it crawl dirs from the PYTHON PATH once and then get data from a cache. I know you can extend the PYTHON PATH dynamically, but you could intercept that in dunder methods to update said cache. Also: for the same interpreter, the same "*.pyc" files and the same PYTHON PATH, we could dump a cache file and share it across interpreter starts if no dirs in PYTHON PATH have been touched. If it's not done, then it might be interesting to bring that up to the Python ideas mailing list.
  - chubot 9y ago
    I'm not sure exactly what you mean, but I think one problem is that the dirs in PYTHONPATH are only the roots. You have to consider changes to all the subdirectories of entries in PYTHONPATH too. The import mechanism is quite complicated, so while I think you could do some more aggressive .pyc-like caching, it would also be pretty hard to get right. Personally what I do is avoid Python packaging tools like setuptools, pip, virtualenv, because they tend to create long PYTHONPATH (they also are hacks piled on top of hacks IMO.) I just write a shell script that downloads dependencies and installs them with distutils / setup.py, and then I wrap the main program in a shell script that sets PYTHONPATH. Yes I know this is unorthodox but it works well for me :)
camus2 9y ago
My question is, is it worse or better with Ruby,NodeJS, Perl,Java or Go? I remember NodeJS scripts being extremely slow to start in general, way slower than Python.
- majewsky 9y ago
  Sorted from slowest to fastest: (all single measurements on my Arch Linux notebook) $ time java &> /dev/null java &> /dev/null 0,09s user 0,03s system 120% cpu 0,102 total $ time ruby -e 42 ruby -e 42 0,04s user 0,01s system 99% cpu 0,053 total $ time python -c 42 python -c 42 0,03s user 0,00s system 98% cpu 0,030 total $ time perl -e 42 perl -e 42 0,00s user 0,00s system 92% cpu 0,005 total I don't have NodeJS at hand to test with. Note that Java produces a help message (which I sent to /dev/null), but that (hopefully) shouldn't affect the timing that much. Go produces compiled binaries and, as such, exhibits no such startup delay. Startup times for Go programs that I tested are in the same ballpark as Perl above, i.e. not reasonably measureable with a millisecond-precision tool like time(1).
Redoubts 9y ago
> The approach of generating source code and exec()ing it, is a cool demonstration of Python's expressive power Man, I don't see many positive opinions of exec out there. > I think we should move on from the original implementation and optimize the heck out of namedtuple. The original has served us well. The world is constantly changing. Python should adapt to the (happy) fact that it's being used for systems larger than any of us could imagine 15 years ago. I really wish the BDFL could say the same for CPython itself. This continued insistence of treating python as a teaching language with a teaching implementation is really weird.
wiradikusuma 9y ago
heh, when I read it, I thought it was about reducing Java startup time to be on par with Python's.
first_amendment 9y ago
It's unfortunate that the new version still creates __new__ using exec(). Doesn't seem necessary at all. Instead of generating the method as a string with the argument names filled in, why not use use a combination of * n and * * kw?
evmar 9y ago
I suspect one factor for why Go is so popular within Google is that (due to complex path hacks as mentioned here as well as a network file system) Python startup is so slow. On an app I worked on the unit tests took >10 seconds to boot on each run. Among the languages available to Google developers, Go ends up being the lightest weight, despite there being plenty of other lighter languages available outside. (This is not the place to start an argument about language plurality.)
makecheck 9y ago
Moving "import" statements into dynamically-controlled blocks goes a long way in my experience, despite being flagged by tools like "pylint". Buried imports free the interpreter from doing something until it is actually required; really nice if you just want to run "--help" and not wait 4 seconds for utterly unrelated modules to be found. It also creates this interesting situation where a script can technically be broken (e.g. module not found) but you don't care as long as the part you're using is OK. Grouped imports are undoubtedly nice for purity and easily seeing dependencies but they may not be smart in a dynamic language. It is still pretty easy to "grep" to find imports if you're trying to track dependencies.
- nerdponx 9y ago
  Would be nice if CPython had an option to enable "lazy imports".
jorams 9y ago
> The approach of generating source code and exec()ing it, is a cool demonstration of Python's expressive power This sentence is very surprising to me. If the best way to implement something is to literally build a string of source code and pass it to the interpreter, to me it means the user is unable to really express what they want in the language. As a (terrible) example, imagine a language whose number addition operator only works on literals. 1 + 2 => 3 a = 1 b = 2 a + b => ERROR! It does, however, have a string concatenation operator that also works on variables and a way to read user-provided numbers into a variable, as well as some kind of eval function. What would you do to add two user-provided numbers? Something like the following: foo = read number! bar = read number! baz = concatenate x, ' + ', y quux = eval baz The same thing could be stated about this language... > The approach of generating source code and eval-ing it, is a cool demonstration of <this language>'s expressive power. ...but it clearly wouldn't make sense. Functions like exec and eval are an escape hatch for when there is no sensible way of expressing something in the language.
- rajathagasthya 9y ago
  No one is claiming exec-ing generated source code is the best way to implement something (namedtuple in this case). The correct way is to use metaclass, but as I noted in another comment exec-ing source code provides a significant speed up to using metaclasses. I imagine that was the rationale behind this particular namedtuple implementation.
- comex 9y ago
  I’d say that the existence of an escape hatch for things that can’t be expressed natively is itself a form of expressitivity. It’s one that’s near ubiquitous among dynamically typed languages, so it doesn’t say much that Python has it, but it still provides a favorable comparison to some static languages. That said, this particular use case for eval is basically macros lite: the code being evaluated doesn’t generally depend on input to the resulting program, so it could have been expressed as a transformation to the module source code, i.e. a macro. And a true macro would be able to provide more natural syntax, whereas namedtuple requires stuffing the field names into a string (or using a wrapper based on some metaclass thing, but the syntax for that isn’t great either). Thus, languages that have macro support, both dynamically typed (Lisp) and statically (e.g. Rust), could be called more expressive in this respect.
ericfrederich 9y ago
It's funny. Go on #python on freenode and ask a question about named tuple. Consensus there is that they should be avoided in favor of real classes or attrs decorated classes.
NewEntryHN 9y ago
$ time python -c '' real 0m0.071s user 0m0.064s sys 0m0.000s $ time python -c 'import functools' real 0m0.073s user 0m0.052s sys 0m0.016s I mean...
- d33 9y ago
  It's not exactly a reliable benchmark for many reasons...
raymondh 9y ago
To be clear. I never said that start-up time wasn't important. Instead, I pointed out that in the previously studied example, the measured impact on start-up time of a single named tuple was a fraction of a millisecond and less than 0.1% of start-up time. At Guido's behest, I agreed to treat this miniscule component of start-up time as being important and will review and implement some optimizations in the sprints next week. I've agreed to do this even if it comes at the expense of code clarity, complexity, or maintainability. I expect that when I'm done, _source will be a lazily generated attribute and that much of the code currently in the template will be pre-computed. This seems reasonable to me. FWIW, I've devoted 16 years of my life to optimizing Python and care a great deal about start-up time. I'm not happy about all the Raymond bashing going on in this thread.
- photonios 9y ago
  Did the author contact you before publishing? The article does not exactly provide an unbiased version of the story. It tries to assign blame.
  - raymondh 9y ago
    The author did not contact me.
- vanni 9y ago
  Thank you Raymond for all your great work. Regarding this post and related comments, I would like to say that, as an external observer who does not know the details of CPython development decision-making process, phrases such as "Okay, then Nick and I are overruled" and "At Guido's behest" sound not so good to my ears. Maybe you and Guido are best friends, and you are half-joking with that tone, or maybe you are responsible for that part of code, so it is your duty to change it according to BDFL last words even if you do not agree with them... I do not know. Anyway I think it would be better, in the open source world, if a code change is developed by someone who thinks it is the right thing to do, or I could not see so much difference from commercial companies and proprietary software. Hey, just my two cents. Keep rocking! :)
hathawsh 9y ago
I feel like the namedtuple issue could be solved cleanly by adding a feature I've wanted for a long time in Python: some simple way to cache high-level data and code in generated .pyc files. Today, there's no way to do that; Python compiles the code but evaluates nothing until execution time. I'd like my .pyc files to contain some precomputed expressions and code, to reduce startup time. I probably ought to discuss this on python-ideas.
Boxxed 9y ago
One side benefit to removing the use of "eval" is that namedtuples will then be pickle-able. I'm surprised this hasn't come up: every now and then I hit some weird data I want to quick-and-dirty serialize but can't because there's a namedtuple buried deep inside it.
- nerdponx 9y ago
  This IMO is a bigger reason to change how namedtuples are implemented than their initialization performance.
- xapata 9y ago
  Namedtuples are pickle-able when created in the global scope, just as any dynamically created class. The main section of the collections module demonstrates that feature (read the source, Luke). Removing eval from tuple would not solve your issue. You'd need to instead remove eval from pickle, of sorts. The trouble is that pickle tries to look up your namedtuple by name and can't find it in your module.
takeda 9y ago
Seems like this article is inflammatory and totally unnecessary. It is written in a way to pin a blame on someone (Raymond H) and also looking at python-dev (and even on the github issue) the decision was made month earlier (July) to go with the optimization[1], and Raymond complied[2]. [1] https://mail.python.org/pipermail/python-dev/2017-July/148606.html https://mail.python.org/pipermail/python-dev/2017-July/14860... [2] https://mail.python.org/pipermail/python-dev/2017-July/148611.html https://mail.python.org/pipermail/python-dev/2017-July/14861...
est 9y ago
IMHO this namedtuple is non-issue compared to buildout. buildout tends to append millions of path to os.path Annnnd your python project start time requires 1 minute or more because each import requires scan millions of directories.

219 points by vanni 9y ago

One of the comments below the article mentions this, but in my experience on Linux and Mac, by far the biggest culprit is pkg_resources and its nasty habit of spidering the filesystem: https://github.com/pypa/setuptools/issues/926 https://github.com/pypa/setuptools/issues/926 There are hacks to get around it, but it's a deep hole.

orf 9y ago
People often complain that using `entry_points` in setup.py is slow. I did some profiling and it came down to it importing some `pip` submodule (pip.vendor.something), which imported basically the whole of pip. :(

sillysaurus3 9y ago

$ time python -c 'print(1)' 1 real 0m0.067s user 0m0.015s sys 0m0.025s Is it slow? That was Python 2.7.10, and for me Python 3.6 is similarly fast.

JasonSage 9y ago
If you had read the article, you would see that the topic of discussion is reducing startup time when many modules are loaded due to the slow nature of `namedtuple`.
agumonkey 9y ago
Maybe only slow when importing. With mainstream pip it's often that you leverage imports a lot.
- deleted 9y ago
  [deleted]
benhoyt 9y ago
On my system (macOS) I see a minimum of 0m0.034 for Python 3.6.1 and a minimum of 0m0.022 for Python 2.7.13 -- so Python 3.6 is 50% slower. This is a somewhat contrived example, however. As the article notes, it's worse when you have a lot of imports (and I don't know how that'd differ between versions 2 and 3). I remember talking to some folks at Google who work on the Google Cloud CLI, and they said that Python startup time is a constant problem for their (large) codebase. Especially when it's used for tab completion, because it might run every time you press tab, and the user wants instant results.
- sillysaurus3 9y ago
  Aha, tab completion makes sense. Thanks!
- mikepurvis 9y ago
  We're dealing with this on a CLI build tools project I hack on (catkin_tools). It uses plugin discovery based on entry_points, and short of caching plugins in a homedir file, it's basically impossible to make it fast enough to use with something like argcomplete. Relevant ticket: https://github.com/catkin/catkin_tools/issues/297 https://github.com/catkin/catkin_tools/issues/297
- smcl 9y ago
  Why on earth would they think invoking a new interpreter each time someone hits "tab" would do anything except perform suboptimally?
sreque 9y ago
I spent a lot of time analyzing the startup time of Java, Python, and Node.js. For any real-world program, startup time is entirely dominated by time taken to load code into memory, which in python is import statements. Also, as slow as Python 2.7 is, Node.js, at least when I measured it, was twice as slow.
draegtun 9y ago
Here's some benchmarks i've come across - http://onetom.rebol.info/2013/08/18/startup-time-of-interpreters/ http://onetom.rebol.info/2013/08/18/startup-time-of-interpre...
paulsutter 9y ago
The discussion is about import speed, which is a real problem: $ time python3 -c 'import tensorflow' real 0m4.522s user 0m1.712s sys 0m0.466s

fractalb 9y ago

If the namedtuple is used so heavily, why not make it a part of built-in functions?

Walkman 9y ago

At my company we changed the interaction between two daemons by calling a command line script instead. It caused a 300ms delay in every single AJAX call and broke a dozens of end-to-end tests. I just tried with Python 2.7 and 3.6 and the hello world startup time is 100ms which is awful lot.

sevensor 9y ago

I'd been wondering why _source was in there. I'm a little surprised that Hettinger thinks it's so important to retain it. It's a neat demonstration of how to make a new data type, and I like that it's so straightforward, but I don't buy the argument that slowing down the interpreter is important for didactic reasons. Edit: I'm talking about namedtuple, the implementation of which is slow because namedtuple() has to construct _source for every new namedtuple type. Which was clear from the article but not from my comment.

dr_zoidberg 9y ago
Actually your edit made it clearer from both. I had the feeling that namedtuply was constructing _source, but it wasn't obvious in the article either. So thanks for explaining, both your comment and the article.
mnarayan01 9y ago
Yea I'm not getting the need to keep _source in collections.namedtuple at all. For teaching purposes, you can just present the existing implementation...I don't see what purpose having that teaching code be in the core distribution serves?
- xapata 9y ago
  Not everyone is familiar with looking at the core source, or even reading the docs.
- takeda 9y ago
  In case you overlooked, Raymond H explained it in another thread[1]. [1] https://news.ycombinator.com/item?id=15135569 https://news.ycombinator.com/item?id=15135569
raymondh 9y ago
One purpose of _source is to make the namedtuple self-documenting -- that has been a considerable benefit over the years and has resulted in many fewer questions and issues than we usually get about our other tools. Another purpose of _source is so that you can write it out to a module. This lets you pre-compute the named tuple for speed and it makes it possible to easily Cythonize it or to customize it (in particular, people like to take out the error checking). Since the namedtuple() factory function was already creating the source string as part of its operation, just storing it and exposing the attribute had nearly zero additional cost. That said, don't worry about it, next week I plan to make this a lazily generated attribute so you all can sleep soundly at night :-)

eesmith 9y ago

I have grown to seriously dislike 'namedtuple'. It seems great, and it's so easy to use, but there's so few places it can be used without dragging in future backwards-compatibility baggage. Let's say you have a light-weight object, like a "full name" which can be represented with a "forename" and an "surname". (Yes, there are many names which don't fit this category. This is a toy example.) So you write: from collections import namedtuple FullName = namedtuple("FullName", "forename surname") name = FullName("Edward", "Smith") So simple! So clean! No "self.forename = forename" or need to make your own __repr__. Except you also inherit the tuple behavior, because this isn't really light-weight object but a ... let's call it a heavy-weight tuple. >>> name[0] 'Edward' >>> name[1] 'Smith' This mean people may (okay, are likely) do things like: forename, surname = name or even for forename, surname in names: ... Even if that's not what you want in your API. If you care about backwards compatibility, then if you replace the namedtuple with its own class then you'll have to re-implement __getitem__, _replace, __len__, and count - methods which make no sense for a simple name class. Or, if you decided to change the API, to make it be: FullName = namedtuple("FullName", "forename middlename surname suffix") then you'll again break code which used the full indexing API that you originally implicitly promised, simply by saying this is a namedtuple. 'namedtuple' has one good use - migration from a tuple object to a real object while allowing backwards compatability to the tuple API. For example, the old tuples once returned from os.stat() and time.gmtime(). It's also fine if it will only be used by code where you have full control over its use and can either prevent inappropriate API use like destructuring assignment, or can update all uses if you want to change the API. For example, a light-weight object used in a test-suite or internal helper function. Otherwise, don't use it. Spend the extra few minutes so in the future you don't have to worry about backwards compatibility issues.

sevensor 9y ago
I disagree -- namedtuple has a crucial property that I've found incredibly useful. Namedtuples are immutable. This makes it impossible (well, not really, but hard) for a namedtuple to change its state without telling me about it, because if it changes state it's not the same namedtuple any more. I agree that namedtuple being a sequence type is problematic, but deciding not to use a lightweight immutable data structure because your colleagues might use it badly is throwing out the baby with the bathwater.

jxcl 9y ago

Raymond Hettinger, if I remember correctly, is the person who originally wrote NamedTuple, and he's not shy about reminding people during his talks at conferences. I wonder if his pride or other personal feelings about that particular module could be clouding his judgment over whether or not to optimize it?

mjburgess 9y ago
I think it's more to do with him primarily teaching python (iirc) rather than working on large systems with it. It's the same strain of rant Zed Shaw had of Py3, ie., that its harder to teach than Py2 and thef. bad. Crazy. EDIT: OK, if not primarily teaching, spending a lot of his time teaching it. That's a big part of his motivation + worth knowing that when assessing his arguments.
- WoodenChair 9y ago
  > It's the same strain of rant Zed Shaw had of Py3, ie., that its harder to teach than Py2 and thef. bad. Crazy. Why is wanting something to be easy to teach crazy?
  - mjburgess 9y ago
    Programming languages are tools for developers. They are taught in the service of software development. Optimizing for teaching has it backwards. It can be a factor: but to place it higher than startup time is prioritising the one-off teaching experience of a few over the repeated development experience of many. For a tool for development. Crazy, no?
    - hueving 9y ago
      Ability to teach is going to be directly correlated with ability to maintain as a business. It's harder to hire people when the bar for learning the skill is higher. The vast majority of software development is not hitting any kind of scaling or performance bottlenecks. So optimizing for simplicity isn't as crazy as you think.

NelsonMinar 9y ago

namedtuple is particularly weird. When you create a new type of namedtuple, Python creates the source code for a new class from a string template and passes it to exec() to build the class. It's a clever and relatively straightforward bit of metaprogramming, but it's not a suprise it's not fast. https://hg.python.org/cpython/file/b14308524cff/Lib/collections/__init__.py#l236 https://hg.python.org/cpython/file/b14308524cff/Lib/collecti... (BTW if you need a quick and dirty way to make Python start up faster, see if you can use the -S flag for your application. It skips the site-specific initialization and has a significant impact on startup times.)

cjbillington 9y ago
Really? Huh. I would have expected a metaclass.
- rajathagasthya 9y ago
  While using a metaclass is indeed cleaner, it's slower than generating code on the fly and exec-ing it. This video talks about it: https://youtu.be/sPiWg5jSoZI?t=1h37m18s https://youtu.be/sPiWg5jSoZI?t=1h37m18s
- Xophmeister 9y ago
  I guess that's how Python 3.6's typing.NamedTuple works, which is frankly a much nicer syntax for named tuples.
  - JelleZijlstra 9y ago
    It's not actually; typing.NamedTuple is just a wrapper that calls the original collections.namedtuple. It does provide some additional features, like default values and methods.
fulafel 9y ago
Cool use of eval. I wonder if there are metaprogramming aimed libraries for generating eval'd code.

wyldfire 9y ago

BDFL says: > Concluding, I think we should move on from the original implementation and optimize the heck out of namedtuple. I remember reading this on the mailing list and thinking "yes, namedtuple deserves optimization because it's an excellent resource". For my use cases, I don't think I notice Python's startup time. When I write Python code it's usually not performance critical. When I do write performance-critical Python code, I usually care about total runtime and PyPy is usually a win here. Aside: if you use PyPy, you should probably be using namedtuples. IIRC it models those much better than it does dicts. And IMO namedtuples model many common data structures better than dicts do. Just because I don't feel the pain of Python's startup time doesn't mean we shouldn't try to optimize it. I think I find my way into some esoteric Python/CPython corners but FWIW I've never needed namedtuple's "_source" attr.

mikepurvis 9y ago
Ruby's idea of a hash where the keys are colon symbols seems to really strike a sweet spot here— it's fast and storage-efficient like a tuple, but has the flexibility of a dict.
- sevensor 9y ago
  One big advantage of using namedtuples, aside from the immutability I mentioned elsewhere, is that you can rely on a namedtuple having particular fields. So if you're using dictionaries, you have to worry about KeyError wherever the dict is used, but if you're using namedtuples, you can't create them without assigning values to all of the fields. Dictionaries are still great for their intended purpose -- to represent a mapping -- but namedtuples are nicer if what you really want is a lightweight object.
  - mikepurvis 9y ago
    Setting default values for the namedtuple requires subclassing it, though, which is its own little bag of fun. See all the wailing and gnashing of teeth on SO: https://stackoverflow.com/search?q=namedtuple+subclass https://stackoverflow.com/search?q=namedtuple+subclass

epberry 9y ago

Ah, slow startups on unittests is a big headache. This is especially apparent on big projects like Django where you might as well leave your desk and grab coffee when the unittests start, even for small projects. That said, I can't help but think that this is just an inevitable tradeoff of Python. You want great expressive power? You have to sacrifice speed. Awesome constructs like namedtuple make this worth it imo.

mfukar 9y ago
I can't help but think if a one-time initialisation cost is bogging a process down, there are bigger problems than that initialisation cost.
- a_t48 9y ago
  If you're running hundreds of very small tests, that one time initialization could take up a large portion of time.
  - mfukar 9y ago
    So...maybe...consolidate the process of executing hundreds of little things? How tiny do the steps in this thought process need to be? We can go on all night.
    - a_t48 9y ago
      It's cleanest if each test runs in a new execution context, so that tests can't effect one another.
      - mfukar 9y ago
        That's a naive solution to a tiny problem.
- coldtea 9y ago
  There are processes that you might need to run frequently. I gave an example above: showing the output of a Python script on your shell prompt (e.g. mercurial status).

jordigh 9y ago

Mercurial did a lot of work to reduce startup time via its demandimport module. https://www.mercurial-scm.org/repo/hg/file/tip/hgdemandimport/demandimportpy2.py#l1 https://www.mercurial-scm.org/repo/hg/file/tip/hgdemandimpor... Basically, it's a lazy loading of all imports. You can write `import foo` but it won't actually be imported until you do `foo.whatever()`. It's a crutch, and it's true that hg still pays the overall startup cost of Python. However, even with "45 times slower than git", the situation is not so dire: jordi@chloe:~$ time hg --version Mercurial Distributed SCM (version 4.2.2) (see https://mercurial-scm.org for more information) Copyright (C) 2005-2017 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. real 0m0.096s user 0m0.084s sys 0m0.008s jordi@chloe:~$ time git --version git version 2.1.4 real 0m0.001s user 0m0.000s sys 0m0.000s I really can barely perceive a difference between 0.096s and 0.001s. Since it really is a one-time startup cost, it's not like we can even say that this difference accumulates and that hg is overall 45 times slower than git. Pierre-Yves also has an interesting talk about all of the tricks that have to go into hg in order to make it fast with Python. There's stuff like doing `method = object.method; method()` instead of doing `object.method()` over and over again to avoid paying the cost of method lookup and so forth: https://2015.pycon.ca/en/schedule/53/ https://2015.pycon.ca/en/schedule/53/

coldtea 9y ago
>I really can barely perceive a difference between 0.096s and 0.001s. Since it really is a one-time startup cost, it's not like we can even say that this difference accumulates and that hg is overall 45 times slower than git. It's bad for many things -- like showing VCS status on your shell prompt or IDE status line, where you repeatedly start hg. 100ms+ delay every time you hit enter in your shell is noticeable (and that's just the version, now doing an actual "dirty" check takes more).
- Too 9y ago
  Mercurial has something called command server[1], it's essentially a daemon listening on sockets to receive commands, designed just to avoid the cost of startup. I had to use it once in a CI system where we did invoke hg thousands of times for some reason i can't even remember. That can also be used for frequent polling like shell status. [1] https://www.mercurial-scm.org/wiki/CommandServer https://www.mercurial-scm.org/wiki/CommandServer

chubot 9y ago

I agree with optimizing Python's startup time, and I agree that namedtuple is weird and should be changed. But I doubt that the namedtuple change will noticeably decrease startup time for most applications (after having experience with this problem for 10+ years). From a comment over 3 years ago: [Python interpreter startup does] random access I/O (the slowest thing your computer can do) proportional to (large constant factor) x (num imports in program) x (length of PYTHONPATH). https://news.ycombinator.com/item?id=7842873 https://news.ycombinator.com/item?id=7842873 I don't think that instantiating a Python parser 100 times for exec() is going to compare to that. I guess the difference is that I'm thinking about the cold cache case -- namedtuple might be noticeable in the warm cache case. And there are many many command line tools that start slow because they're written in Python, not just Mercurial. Mercurial is actually one of the best because they care about it (demandimport) and they don't have too many dependencies. IIRC some of the Google Cloud tools take 500-1000+ ms to start because they are crawling tons of dependencies with long PYTHONPATH.

sametmax 9y ago
Does this mean the import mechanism caches loaded module, but not where to find non imported things ? I would expect that it crawl dirs from the PYTHON PATH once and then get data from a cache. I know you can extend the PYTHON PATH dynamically, but you could intercept that in dunder methods to update said cache. Also: for the same interpreter, the same "*.pyc" files and the same PYTHON PATH, we could dump a cache file and share it across interpreter starts if no dirs in PYTHON PATH have been touched. If it's not done, then it might be interesting to bring that up to the Python ideas mailing list.
- chubot 9y ago
  I'm not sure exactly what you mean, but I think one problem is that the dirs in PYTHONPATH are only the roots. You have to consider changes to all the subdirectories of entries in PYTHONPATH too. The import mechanism is quite complicated, so while I think you could do some more aggressive .pyc-like caching, it would also be pretty hard to get right. Personally what I do is avoid Python packaging tools like setuptools, pip, virtualenv, because they tend to create long PYTHONPATH (they also are hacks piled on top of hacks IMO.) I just write a shell script that downloads dependencies and installs them with distutils / setup.py, and then I wrap the main program in a shell script that sets PYTHONPATH. Yes I know this is unorthodox but it works well for me :)

camus2 9y ago

My question is, is it worse or better with Ruby,NodeJS, Perl,Java or Go? I remember NodeJS scripts being extremely slow to start in general, way slower than Python.

majewsky 9y ago
Sorted from slowest to fastest: (all single measurements on my Arch Linux notebook) $ time java &> /dev/null java &> /dev/null 0,09s user 0,03s system 120% cpu 0,102 total $ time ruby -e 42 ruby -e 42 0,04s user 0,01s system 99% cpu 0,053 total $ time python -c 42 python -c 42 0,03s user 0,00s system 98% cpu 0,030 total $ time perl -e 42 perl -e 42 0,00s user 0,00s system 92% cpu 0,005 total I don't have NodeJS at hand to test with. Note that Java produces a help message (which I sent to /dev/null), but that (hopefully) shouldn't affect the timing that much. Go produces compiled binaries and, as such, exhibits no such startup delay. Startup times for Go programs that I tested are in the same ballpark as Perl above, i.e. not reasonably measureable with a millisecond-precision tool like time(1).

Redoubts 9y ago

> The approach of generating source code and exec()ing it, is a cool demonstration of Python's expressive power Man, I don't see many positive opinions of exec out there. > I think we should move on from the original implementation and optimize the heck out of namedtuple. The original has served us well. The world is constantly changing. Python should adapt to the (happy) fact that it's being used for systems larger than any of us could imagine 15 years ago. I really wish the BDFL could say the same for CPython itself. This continued insistence of treating python as a teaching language with a teaching implementation is really weird.

wiradikusuma 9y ago

heh, when I read it, I thought it was about reducing Java startup time to be on par with Python's.

first_amendment 9y ago

It's unfortunate that the new version still creates __new__ using exec(). Doesn't seem necessary at all. Instead of generating the method as a string with the argument names filled in, why not use use a combination of * n and * * kw?

evmar 9y ago

I suspect one factor for why Go is so popular within Google is that (due to complex path hacks as mentioned here as well as a network file system) Python startup is so slow. On an app I worked on the unit tests took >10 seconds to boot on each run. Among the languages available to Google developers, Go ends up being the lightest weight, despite there being plenty of other lighter languages available outside. (This is not the place to start an argument about language plurality.)

makecheck 9y ago

Moving "import" statements into dynamically-controlled blocks goes a long way in my experience, despite being flagged by tools like "pylint". Buried imports free the interpreter from doing something until it is actually required; really nice if you just want to run "--help" and not wait 4 seconds for utterly unrelated modules to be found. It also creates this interesting situation where a script can technically be broken (e.g. module not found) but you don't care as long as the part you're using is OK. Grouped imports are undoubtedly nice for purity and easily seeing dependencies but they may not be smart in a dynamic language. It is still pretty easy to "grep" to find imports if you're trying to track dependencies.

nerdponx 9y ago
Would be nice if CPython had an option to enable "lazy imports".

jorams 9y ago

> The approach of generating source code and exec()ing it, is a cool demonstration of Python's expressive power This sentence is very surprising to me. If the best way to implement something is to literally build a string of source code and pass it to the interpreter, to me it means the user is unable to really express what they want in the language. As a (terrible) example, imagine a language whose number addition operator only works on literals. 1 + 2 => 3 a = 1 b = 2 a + b => ERROR! It does, however, have a string concatenation operator that also works on variables and a way to read user-provided numbers into a variable, as well as some kind of eval function. What would you do to add two user-provided numbers? Something like the following: foo = read number! bar = read number! baz = concatenate x, ' + ', y quux = eval baz The same thing could be stated about this language... > The approach of generating source code and eval-ing it, is a cool demonstration of <this language>'s expressive power. ...but it clearly wouldn't make sense. Functions like exec and eval are an escape hatch for when there is no sensible way of expressing something in the language.

rajathagasthya 9y ago
No one is claiming exec-ing generated source code is the best way to implement something (namedtuple in this case). The correct way is to use metaclass, but as I noted in another comment exec-ing source code provides a significant speed up to using metaclasses. I imagine that was the rationale behind this particular namedtuple implementation.
comex 9y ago
I’d say that the existence of an escape hatch for things that can’t be expressed natively is itself a form of expressitivity. It’s one that’s near ubiquitous among dynamically typed languages, so it doesn’t say much that Python has it, but it still provides a favorable comparison to some static languages. That said, this particular use case for eval is basically macros lite: the code being evaluated doesn’t generally depend on input to the resulting program, so it could have been expressed as a transformation to the module source code, i.e. a macro. And a true macro would be able to provide more natural syntax, whereas namedtuple requires stuffing the field names into a string (or using a wrapper based on some metaclass thing, but the syntax for that isn’t great either). Thus, languages that have macro support, both dynamically typed (Lisp) and statically (e.g. Rust), could be called more expressive in this respect.

ericfrederich 9y ago

It's funny. Go on #python on freenode and ask a question about named tuple. Consensus there is that they should be avoided in favor of real classes or attrs decorated classes.

NewEntryHN 9y ago

$ time python -c '' real 0m0.071s user 0m0.064s sys 0m0.000s $ time python -c 'import functools' real 0m0.073s user 0m0.052s sys 0m0.016s I mean...

d33 9y ago
It's not exactly a reliable benchmark for many reasons...

raymondh 9y ago

To be clear. I never said that start-up time wasn't important. Instead, I pointed out that in the previously studied example, the measured impact on start-up time of a single named tuple was a fraction of a millisecond and less than 0.1% of start-up time. At Guido's behest, I agreed to treat this miniscule component of start-up time as being important and will review and implement some optimizations in the sprints next week. I've agreed to do this even if it comes at the expense of code clarity, complexity, or maintainability. I expect that when I'm done, _source will be a lazily generated attribute and that much of the code currently in the template will be pre-computed. This seems reasonable to me. FWIW, I've devoted 16 years of my life to optimizing Python and care a great deal about start-up time. I'm not happy about all the Raymond bashing going on in this thread.

photonios 9y ago
Did the author contact you before publishing? The article does not exactly provide an unbiased version of the story. It tries to assign blame.
- raymondh 9y ago
  The author did not contact me.
vanni 9y ago
Thank you Raymond for all your great work. Regarding this post and related comments, I would like to say that, as an external observer who does not know the details of CPython development decision-making process, phrases such as "Okay, then Nick and I are overruled" and "At Guido's behest" sound not so good to my ears. Maybe you and Guido are best friends, and you are half-joking with that tone, or maybe you are responsible for that part of code, so it is your duty to change it according to BDFL last words even if you do not agree with them... I do not know. Anyway I think it would be better, in the open source world, if a code change is developed by someone who thinks it is the right thing to do, or I could not see so much difference from commercial companies and proprietary software. Hey, just my two cents. Keep rocking! :)

hathawsh 9y ago

I feel like the namedtuple issue could be solved cleanly by adding a feature I've wanted for a long time in Python: some simple way to cache high-level data and code in generated .pyc files. Today, there's no way to do that; Python compiles the code but evaluates nothing until execution time. I'd like my .pyc files to contain some precomputed expressions and code, to reduce startup time. I probably ought to discuss this on python-ideas.

Boxxed 9y ago

One side benefit to removing the use of "eval" is that namedtuples will then be pickle-able. I'm surprised this hasn't come up: every now and then I hit some weird data I want to quick-and-dirty serialize but can't because there's a namedtuple buried deep inside it.

nerdponx 9y ago
This IMO is a bigger reason to change how namedtuples are implemented than their initialization performance.
xapata 9y ago
Namedtuples are pickle-able when created in the global scope, just as any dynamically created class. The main section of the collections module demonstrates that feature (read the source, Luke). Removing eval from tuple would not solve your issue. You'd need to instead remove eval from pickle, of sorts. The trouble is that pickle tries to look up your namedtuple by name and can't find it in your module.

takeda 9y ago

Seems like this article is inflammatory and totally unnecessary. It is written in a way to pin a blame on someone (Raymond H) and also looking at python-dev (and even on the github issue) the decision was made month earlier (July) to go with the optimization[1], and Raymond complied[2]. [1] https://mail.python.org/pipermail/python-dev/2017-July/148606.html https://mail.python.org/pipermail/python-dev/2017-July/14860... [2] https://mail.python.org/pipermail/python-dev/2017-July/148611.html https://mail.python.org/pipermail/python-dev/2017-July/14861...

est 9y ago

IMHO this namedtuple is non-issue compared to buildout. buildout tends to append millions of path to os.path Annnnd your python project start time requires 1 minute or more because each import requires scan millions of directories.