5 ms·

namedtuple is particularly weird. When you create a new type of namedtuple, Python creates the source code for a new class from a string template and passes it

namedtuple is particularly weird. When you create a new type of namedtuple, Python creates the source code for a new class from a string template and passes it to exec() to build the class. It's a clever and relatively straightforward bit of metaprogramming, but it's not a suprise it's not fast. https://hg.python.org/cpython/file/b14308524cff/Lib/collections/__init__.py#l236 https://hg.python.org/cpython/file/b14308524cff/Lib/collecti... (BTW if you need a quick and dirty way to make Python start up faster, see if you can use the -S flag for your application. It skips the site-specific initialization and has a significant impact on startup times.)

cjbillington 9y ago
Really? Huh. I would have expected a metaclass.
- rajathagasthya 9y ago
  While using a metaclass is indeed cleaner, it's slower than generating code on the fly and exec-ing it. This video talks about it: https://youtu.be/sPiWg5jSoZI?t=1h37m18s https://youtu.be/sPiWg5jSoZI?t=1h37m18s
- Xophmeister 9y ago
  I guess that's how Python 3.6's typing.NamedTuple works, which is frankly a much nicer syntax for named tuples.
  - JelleZijlstra 9y ago
    It's not actually; typing.NamedTuple is just a wrapper that calls the original collections.namedtuple. It does provide some additional features, like default values and methods.
fulafel 9y ago
Cool use of eval. I wonder if there are metaprogramming aimed libraries for generating eval'd code.
- nerdponx 9y ago
  Why would you want that when you have actual metaprogramming on code-as-data in the standard library?
  - fulafel 9y ago
    Is this a joke on the namedtuple trick or is there actually such a module in the standard library?
    - nerdponx 9y ago
      Not a joke, but "code as data" was hyperbolic. You can do some pretty amazing metaprogramming in Python with metaclasses, and the inspect and ast modules give you access to the guts of the language.
makecheck 9y ago
I'm curious, given the presence of a ".pyc", shouldn't these kinds of costs be paid primarily on first invocation "ever"?
- chronial 9y ago
  No, because the string given to eval will not be stored in the pyc – it has to be compiled on every import.
  - tankenmate 9y ago
    It can't be cached? Is not caching a dependency issue?
    - pwdisswordfish 9y ago
      It can't, because lol dynamic dispatch. Exercise: what does this code do? import collections collections.namedtuple = lambda _0, _1, _2=False, _3=False: "Where is your god now?" import b b.py: import collections print(collections.namedtuple('Point', ['x', 'y'])) And this is not even particularly 'evil' code, in the sense of relying on internal implementation details; that's just your bread-and-butter Python language semantics. It prevents (or at least complicates) many other potential optimisations as well.
- adambyrtek 9y ago
  Not in a general case, because eval/exec accepts an arbitrary string which could change at runtime.
xapata 9y ago
You should link to the GitHub repo instead of Hg. They put effort into the move.
- jwilk 9y ago
  https://github.com/python/cpython/blob/v3.6.2/Lib/collections/__init__.py#L307 https://github.com/python/cpython/blob/v3.6.2/Lib/collection...
smegel 9y ago
> It's a clever and relatively straightforward bit of metaprogramming Wat? It's neither of those things.
- xapata 9y ago
  Should we take an informal survey?

cjbillington 9y ago

Really? Huh. I would have expected a metaclass.

rajathagasthya 9y ago
While using a metaclass is indeed cleaner, it's slower than generating code on the fly and exec-ing it. This video talks about it: https://youtu.be/sPiWg5jSoZI?t=1h37m18s https://youtu.be/sPiWg5jSoZI?t=1h37m18s
Xophmeister 9y ago
I guess that's how Python 3.6's typing.NamedTuple works, which is frankly a much nicer syntax for named tuples.
- JelleZijlstra 9y ago
  It's not actually; typing.NamedTuple is just a wrapper that calls the original collections.namedtuple. It does provide some additional features, like default values and methods.

fulafel 9y ago

Cool use of eval. I wonder if there are metaprogramming aimed libraries for generating eval'd code.

nerdponx 9y ago
Why would you want that when you have actual metaprogramming on code-as-data in the standard library?
- fulafel 9y ago
  Is this a joke on the namedtuple trick or is there actually such a module in the standard library?
  - nerdponx 9y ago
    Not a joke, but "code as data" was hyperbolic. You can do some pretty amazing metaprogramming in Python with metaclasses, and the inspect and ast modules give you access to the guts of the language.

makecheck 9y ago

I'm curious, given the presence of a ".pyc", shouldn't these kinds of costs be paid primarily on first invocation "ever"?

chronial 9y ago
No, because the string given to eval will not be stored in the pyc – it has to be compiled on every import.
- tankenmate 9y ago
  It can't be cached? Is not caching a dependency issue?
  - pwdisswordfish 9y ago
    It can't, because lol dynamic dispatch. Exercise: what does this code do? import collections collections.namedtuple = lambda _0, _1, _2=False, _3=False: "Where is your god now?" import b b.py: import collections print(collections.namedtuple('Point', ['x', 'y'])) And this is not even particularly 'evil' code, in the sense of relying on internal implementation details; that's just your bread-and-butter Python language semantics. It prevents (or at least complicates) many other potential optimisations as well.
adambyrtek 9y ago
Not in a general case, because eval/exec accepts an arbitrary string which could change at runtime.

xapata 9y ago

You should link to the GitHub repo instead of Hg. They put effort into the move.

jwilk 9y ago
https://github.com/python/cpython/blob/v3.6.2/Lib/collections/__init__.py#L307 https://github.com/python/cpython/blob/v3.6.2/Lib/collection...

smegel 9y ago

> It's a clever and relatively straightforward bit of metaprogramming Wat? It's neither of those things.

xapata 9y ago
Should we take an informal survey?