4 ms·
Citation? This doesn't appear to be true: https://stackoverflow.com/questions/14118564/how-does-slots-avoid-a-dictionary-lookup https://stackoverflow.com/ques
by chubot 9y ago
Citation? This doesn't appear to be true:
https://stackoverflow.com/questions/14118564/how-does-slots-avoid-a-dictionary-lookup https://stackoverflow.com/questions/14118564/how-does-slots-...
The goal of using __slots__ is to save memory; instead of using a .__dict__ mapping on the instance, the class has descriptors objects for each and every attribute
So python still has to look at the class for each attribute access on an instance of Foo (to find the descriptor). Any unknown attribute (say, Foo.ham) will still result in Python looking through the class MRO to search for that attribute, and that includes dictionary searches.
If I have time I'll write an example program to test this. Since I've been curious for awhile.
I've actually forked the Python interpreter to make my highly compatible bash shell faster and smaller, while not completely rewriting it [1].
-----
Even if __slots__ did what you think it does, it doesn't completely solve the problem.
A lookup like "return os.path.exists(foo)" can't use slots, because it's a module and not a class.
In principle, local variable lookups are also slower in Python than say Dalvik (a JVM without a JIT). LOAD_FAST still involves a dict lookup (as opposed to the even slower LOAD_ATTR). It just happens at the beginning of the function once, not on say every iteration of a loop inside the function.
In contrast, a Java compiler will resolve the attribute access at compile time.
PEP 509 might help with avoiding repeated dictionary lookups in this case, but it's very new: https://www.python.org/dev/peps/pep-0509/ https://www.python.org/dev/peps/pep-0509/
Also, if slots did what you think it does, it would be 10x faster than normal attribute lookup, but it's not (according to the benchmark above, and at least one other one I saw.)
[1] http://www.oilshell.org/blog/2017/04/25.html http://www.oilshell.org/blog/2017/04/25.html
EDIT: My last two points might seem like a tangent, since they're not really related to structs. But my overall point is that Python doesn't do very much at compile time, so some of that work has to be done at runtime. slots could easily be faster than it is; it's mainly a memory optimization.
- icebraining 9y agoSo python still has to look at the class for each attribute access on an instance of Foo (to find the descriptor). Any unknown attribute (say, Foo.ham) will still result in Python looking through the class MRO to search for that attribute, and that includes dictionary searches. Yes, it has to lookup, but the lookup is cached (see typeobject.c).
- chubot 9y agoWhere in typeobject.c? I looked, and there's a lot of stuff related to slots, but I don't see any caching. Also I did a speed test on my machine. My results are the same as the ones I've found through Google: __slots__ is not much faster. It's maybe 20% faster, whereas if it was really a dict lookup vs. index lookup I would expect it to be an order magnitude faster, or at least 5 times faster. To be clear, I'm claiming that every time you do "p.x" with a class with __slots__, at least one dictionary lookup occurs. It's not like p.x in C or Java. I could be wrong, but I've never heard the claim you're making, and I don't see any evidence it's true.
- chubot 9y agoHm I did a test with an coverage-instrumented Python binary, and you are right. I counted the number of times lookdict_string in dictobject.c is called -- call that number D. And then I made N attribute accesses like 'p.x'. Without slots, if N goes up, then D goes up. With slots, as N goes up, D stays constant. Thanks for the information -- I'll look into this more. (However, as mentioned, slots are not more than 20-30% faster on my machine, in this limited benchmark. I feel like this is a double mystery. Or maybe it's just that interpreter loop overhead or some other overhead is the bottleneck, and not dict access.)
- chubot 9y agoI'm talking to myself at this point, but I think the problem with the caching is that every attribute lookup p.x has to consult the type, not just the object. That's not true in Java. So you avoid the dictionary lookup, but you're still following extra pointers.