8 ms·
Let me introduce: __slots__
- coconutrandom 9y agoA few years ago I had read about Python performance gains and some built in attributes, but couldn't find the reference when I looked. Well it was `__slots__`, thanks for posting!
- trjordan 9y agoSince I don't think it was explicitly mentioned, this is because __slots__ is effectively a named tuple. The size reduction comes from dropping the keys in the dictionary (they're stored once on the class, not every time on the instances), and the speed reduction come from it being an array lookup instead of a dict. It's also a great way to introduce some static typing. No more setting mistyped class attributes without errors!
- deleted 9y ago[deleted]
- gipp 9y agoFor simple classes, you can also just use `NamedTuple` like: from typing import NamedTuple class Foo(NamedTuple): bar: int = 0 baz: str = 'default' foo = Foo(1, baz='not default') This provides the same performance benefits, and is very similar in a lot of ways to a Scala case class -- aside from removing the boilerplate of stating each attribute twice in __init__ and again in __slots__, it also provides nice default implementations for __repr__, __str__ (something like `Foo(bar=1, baz='quux')`), __hash__, and __eq__ (using member equality rather than identity). The downsides are A, you can't inherit from any other class (it's syntactically valid, but silently ignored (!) though it will be caught by linters), and B, `isinstance(foo, tuple)` returns `True`, which messes with a lot of reflection code (e.g. the `json` module will not serialize them like you expect)
- nerdponx 9y agoUnless you really need to iterate over your class attributes I see no reason to do this. If you really want to save yourself the extra typing, could just as easily write a "make_slotted_class" factory function or a "SlottedType" metaclass that generates these classes with the same syntax as namedtuple/NamedTuple. Edit: I stand corrected. There's no easy way (that I'm aware of) to generate an __init__ method in Python without resorting to string manipulation. Perhaps you could do it at the C level, but that defeats the purpose IMO.
- shakna 9y agoUse "type".[0] The last argument should be the dict that __INIT__ is usually responsible for creating. [0] https://jeffknupp.com/blog/2013/12/28/improve-your-python-metaclasses-and-dynamic-classes-with-type/ https://jeffknupp.com/blog/2013/12/28/improve-your-python-me...
- nerdponx 9y agoNot quite, AFAIU. The last argument is actually a dict of class attributes, one of which is the __init__ method. You can hack around it a few different ways (e.g. creating __init__ as a closure over the slot names and then inserting it into that 3rd argument), but none of them generate a proper init method. I'm planning on trying again tonight and I will probably blog my results on Friday. I'll read over that post carefully in case there's something I missed about type() and metaclasses.
- shakna 9y agoGood luck! Last time I approached this was generating classes for Pyramid, an it was hell, because it was incredibly poorly documented, and the community view was that metaprogramming was the devil's work. Community has started to shift, so there's a few more resources, but things don't behave exactly as expected so you will run into issues here and there. Hopefully you can add to the knowledge out there!
- gshulegaard 9y ago
- ntrepid8 9y agoIf you want to create a large number of Objects (like for rows in a data set) using __slots__ saves a lot of memory over just using the standard __dict__. I've always used named tuples for this in the past, but this is a nice way to do it.
- insertnickname 9y agoPlease excuse this off-topic comment, but the webpage is severely dysfunctional. Unless JavaScript is enabled in the browser, the page will just display a loading screen. All the content is right there in the HTML and it would be perfectly readable if it wasn't deliberately obscured by the "loading" screen.[1] I'm sure the idea is that the page will load perfectly all at once, there won't be flashes of unstyled text and so on, but for me it just means that the real content won't load at all. Curiously, mobile users (or other users with small screens) are spared the loading page.[2] That's nice for mobile users, I guess, but as a desktop user it's just salt in the wound. If the plain page is good enough for mobile users, why isn't it good enough for me? Please, web developers, stop doing this. It's not just the minority of people who browse with JavaScript disabled who are bothered by this, think of all the people on slow Internet connections who have to wait for your megabyte+ JavaScript program to download and execute before they can read the content of the 50 KB HTML page. I understand that it is not feasible to accommodate no-JS browsers for Single Page Applications because JavaScript is essential to their functionality, but this page is not an application and JS is obviously not essential to it. Use JavaScript to enhance webpages, not degrade them. - [1] Yes, Reader View in Firefox (and similar) are able to render the page properly. It's a wonderful utility, but I shouldn't have to strip away your broken web design to read your content. - [2] https://streamable.com/b390d https://streamable.com/b390d (no JS mirror: https://cgt.name/files/fortheloveofgod.ogv https://cgt.name/files/fortheloveofgod.ogv)
- Karunamon 9y agoYe gods! uBlock Origin had a whopping 29 items blocked here here. I know mainstream corporate news sites with less garbage in them!
- aaronbrethorst 9y agoI felt the same way. I was expecting an article about Python, not a loading indicator on my broadband connection.
- drivingmenuts 9y agoWhat's amusing, and by amusing, I mean not amusing at all, is that there isn't any heavy content in the page - it's all text until you get to the comments, which aren't going to be seen until you get to the end of the article. When all you have is a hammer and so on and so forth ...
- Animats 9y agoGuido used to be hostile to __slots__, but apparently he's lightened up. If you're not using "setattr" to dynamically add fields to your object, CPython's underlying machinery is overkill. Everything is a dict, and all accesses take a lookup. With __slots__, a Python object is more like a struct in other languages. Slotted objects should be the default. You should have to inherit from "DynamicObject" or something like it to enable setattr on an object. Actually, dynamic objects stopped being that important once Python let you inherit from "dict". Then, if you wanted "foo[ix]", it was easy to do it. There are some packages, such as BeautifulSoup, which use dynamic attributes heavily. BeautifulSoup uses it to map HTML tags to attributes. This is more trouble than it's worth, because when an HTML tag name clashes with a Python builtin name, workarounds are necessary.
- make3 9y ago> "should" this would make python a different language completely
- Spivak 9y agoI mean it would certainly be a breaking change to existing code, and we know how well that goes over in the Python community, but it wouldn't be a completely different language. The argument is that these objects are a sane default that would prevent a lot of accidental programming mistakes and I agree.
- vidarh 9y agoSee my comment elsewhere about a "split" approach to this for a Ruby implementation. I get that there's a semantic difference to change from __dict__ to the way __slots__ currently works, but it might be feasible to change the way __dict__ works to inline instance variables in the instance by default in cases where they're readily statically inferred and get much of the same effect without changing the semantics of __dict__.
- pfranz 9y agoI see __slots__ come up quite often (I think more often than it should). It's great to be aware of, but shouldn't be used unless necessary. Like any optimization, it can make code maintenance more difficult. Obviously, you have to update slots if you add more attributes to the class, but you have to be aware of slots when dealing with class inheritance.
- vidarh 9y agoI've opted for a similar approach "behind the scenes" for my (wildly incomplete, though now close to compiling itself) Ruby compiler: While you can usually statically determine most likely instance variables for Ruby classes, and optimize that by creating the equivalent of slots, you also need to be able to dynamically create more, so I allocate "slots" in the instances for any variables that I can see, and fall back on the equivalent of __dict__ for anything that's dynamically added. I've always found it quite curious that Python made this explicit rather than an implementation detail (I get that there are slight semantic differences here in Python too) because it seems like such a useful optimization. And a similar approach can be used for method lookup too. I wonder (and I don't have any measurements on this myself) what the relative benefits are to explicitly picking one over the other vs. automating it are. The downside of doing it automatically is that some types of objects might very well be very "sparse" in that it's not a given that all instance variables are used for all objects (this is more applicable in for vtables, especially in languages with single-rooted object hierarchies). My intuition is that the overhead of a map/dict/hash table to store them is likely to usually outweigh the cost of quite a lot of unused instance variables, and so that inferring instance variables is generally likely to be an improvement.
- pmontra 9y agoDo you have a link to your Ruby compiler? EDIT found it in your profile http://www.hokstad.com/compiler http://www.hokstad.com/compiler and submitted to HN. Great series of posts. About Python, it's a little older than Ruby (its design started in 1989) and Guido worked on the ABC language at the beginning of the 80s. IMHO that difference shows in many places, for example in having to explicitly pass self as argument in method definitions. Newer languages do without it and programmers don't get confused at all. All those double underscore methods and the general explicitness of the language smell of 80s and of C. Not necessarily a bad thing but reading some Python code is like getting on a time machine.
- vidarh 9y agoThanks. It's long overdue some updates as I've done quite a lot of work since (though still moving slowly) and it's pretty close to being able to fully compile itself now. What you say makes sense, and I understand that it takes a lot of effort to clean those type of things up without hurting backwards compatibility. This specific distinction though seems like one where it'd still be possible to get most of the benefit by changing the implementation of __dict__ to make it do something similar to __slot__ under the hood, but fall back to a dict for dynamic properties.
- amelius 9y agoThis is all nice and all, but of course such language hacks (as I would call them) make for a less elegant language with a higher barrier to entry. At some point it would make sense from a software-engineering standpoint to switch to a cleaner lower level language. By the way, I like the approach taken in JavaScript engines such as V8, which determine the "slots" dynamically.
- spraak 9y agoI just started learning Python, what does this syntax mean? **json.loads(my_json)) from with_slots = [get_size(MyUserWithSlots(**json.loads(my_json))) for _ in range(NUM_INSTANCES)] Edit: I mean, I get what it's doing, but specifically I don't understand the double * (somehow it's not rendering in this comment)
- vtbassmatt 9y agoThis uses the key-value pairs as keyword arguments.
- jedberg 9y agoUnpacks the object into a dictionary. It's syntactic sugar: https://stackoverflow.com/a/2921893/66202 https://stackoverflow.com/a/2921893/66202
- deathanatos 9y agoA function in Python takes two kinds of arguments: positional arguments, and keyword arguments. def foo(a, b, c): pass # Arguments passed positionally: foo(1, 2, 3) # or passed by keyword: foo(a=1, b=2, c=3) Now, say you want to make that second call somewhat dynamically, and you have the dict: a_dict = {'a': 1, 'b': 2, 'c': 3} You can make that second foo call: foo(**a_dict) # because of the value of a_dict, expands to: # foo(a=1, b=2, c=3) Functions themselves can take only positional arguments, only keyword arguments, or really, any combination of them. Here is a decent SO question and answer[1] that might help, a bit about it in the tutorial[2], and last, the formal documentation[3]. [1]: https://stackoverflow.com/questions/1419046/python-normal-arguments-vs-keyword-arguments https://stackoverflow.com/questions/1419046/python-normal-ar... [2]: https://docs.python.org/2/tutorial/controlflow.html#keyword-arguments https://docs.python.org/2/tutorial/controlflow.html#keyword-... [3]: https://docs.python.org/2/reference/expressions.html#grammar-token-call https://docs.python.org/2/reference/expressions.html#grammar...
- spraak 9y agoAwesome, thank you for the detailed explanation!
- aaronchall 9y agoI have updated the documentation on __slots__, and those changes have made it into the dev version of the Python docs.[0] I have also written up __slots__ in great detail on Stack Overflow.[1] I'm at a meetup right now, but I can try to answer any quick questions here tonight. [0] https://docs.python.org/3.7/reference/datamodel.html#object.__slots__ https://docs.python.org/3.7/reference/datamodel.html#object.... [1] https://stackoverflow.com/q/472000/541136 https://stackoverflow.com/q/472000/541136
- abhirag 9y agoI just discovered attrs(http://www.attrs.org/en/stable/ http://www.attrs.org/en/stable/). On paper it looks great, seems to reduce boilerplate, has support for immutability and slots too and claims to have no runtime overhead. If anybody here has experience using it, would love to hear from you. Is it as good as it looks on paper? Do you recommend using it?
- ericfrederich 9y agoI've been using it here and there in place of named tuples and data-only classes. It is very convenient. I had a project where I had to connect to a data source and serialize some data. When I needed to add an attribute all I had to do was add it to two places: the class itself, and the @classmethod constructor. So in this case I got the serialization for free, but that's all I was using. In reality I also got __repr__, __cmp__, etc, etc for free too, I just wasn't using it. Serialization was free because attr.asdict() knows which attributes are attr.ib() attributes. @attr.s() class Foo: bar = attr.ib() spam = attr.ib() eggs = attr.ib() # added this line @classmethod def from_something_else(cls, x): return cls( bar=x.bar.name, spam=x.spam, eggs=x.get_eggs().blah(), # and this line ) with open('blah.json', 'w') as fout: json.dump(fout, attr.asdict(o)) # got serialization for free
- wutbrodo 9y agoI don't think it's necessarily a bad thing, but it's kind of odd to see an article on the front page of HN that amounts to a less-concise version of the documentation for a very simple to understand language feature (the doc page has four or five sentences and is just as easy to understand).
- plainOldText 9y agoOne caveat though, object creation with `__slots__` is a bit slower than "normal" objects, if I remember my past Python benchmarks correctly.