14 ms·
Problems once solved by a metaclass can be solved by __init_subclass__
- lvass 5y agoAm I just bad at OOP hackery or does anyone else look at this and think it'd be extremely hard to debug?
- foolfoolz 5y agoi wouldn’t let this past code review. it’s magic and makes a great blog post about some feature but is not maintainable code
- pmarreck 5y agoYep. As a current functional-language guy, this merely confirms my suspicion that those crazy functional people were onto something. As it turns out, since the human mind is the real limitation here, anything that stops runaway complexity ("hiding" functionality behind "magic" is actually worse because you're just moving the complexity "away from your mental model" where it can fester and grow out of sight, even if what you see looks "simpler") results in fewer bugs, and more-debuggable bugs.
- qsort 5y agoFrankly I doubt you even need metaclasses in the first place (at least in non-library code). I get that it's a dynamic language but if I need metaclasses for something it's more likely than not to be a hidden technical debt generator.
- vosper 5y agoI think SQLAlchemy makes good use of metaclasses. It's the only thing I've seen that used them in a way that I thought made sense and was well justified (but I have been out of the Python world for a while). In another language it would be done with code-gen, which would also be fine. But SQLAlchemy is library code. I took the time to understand how to use metaclasses, and concluded that I never would, especially not in code I expected another engineer to understand - I expect 90% of people who write Python haven't heard of metaclasses, and 99% have never written one.
- nomel 5y agoThere was a time, early on in my python career, that I tried all the different flavors of magic that python provides. These days, I tend to write code for junior developers. I don’t use any magic at all, even when it could make sense. Once you start playing with magic, you have far fewer eyes and hands that are interested.
- nightski 5y agoIn my opinion it's not about dealing with junior developers or even having the ability to understand it. I've been programming for 25 years and to me it's about cognitive load. I despise tricks like this because they put a much higher strain on the programmer. Software is hard enough, let's not make it harder. If I can look at a piece of code and it's quickly obvious whether it is correct or not is my favorite type of code...
- draw_down 5y ago
- jreese 5y agoI'm not fond of the example given in the post, but I've very happily used __init_subclass__ in a few places where I previously would have needed a decorator. Eg, for "plugin-like" things where I would have otherwise needed a decorator to keep track, or iterated over subclasses, now can just "register" subclasses without any extra work on the child classes. Things that I normally wouldn't expect to change the behavior of the subclass, but where having a no-effort hook at subclass creation time can be extremely useful.
- sillysaurusx 5y agoI used to think so, but cls.__subclasses__() makes it possible to just walk the type tree at some later point. I find the lack of state to be a feature. (Normally with a hook like you're talking about, you'd add the class to some list somewhere, which can get out of sync.)
- formerly_proven 5y agoMetaclasses / __init_subclass__ are generally only used if you try to define different / new behavior for how the body of a class is interpreted. So if you'd want to create e.g. an explicit interface system in Python, you might use them. If you're creating an ORM or some other kind of declarative property system, you might use them (but not necessarily, as descriptors are a thing). I think the only use in the standard library is in ABCs and probably the typing module. Edit: Enums use them as well (makes sense).
- klodolph 5y agoIt's used very sparingly in the wild, for cases where you need a bunch of classes that are set up similarly, and ordinary subclassing still results in too much boilerplate. For example... you might make a class for each HTML tag in an HTML processor, or each drawing command in an SVG processor. My experience is that it's not hard to debug, because all that you're doing is getting rid of some boilerplate.
- Traubenfuchs 5y agoI am baffled that people make fun of Java and Spring and then use double underscore magic methods called __init_subclass__ with a straight face.
- RhysU 5y agoI once spent a week trying to use this kruft to make a custom class hierarchy accept dataclass annotations. I was successful but damn.
- pengwing 5y agoI feel jumping right into __init_subclass__ without explaining why metaclasses exist and what problems they typically solve excludes the majority of devs on HN. Thereby limiting any discussion only to the advanced Python developer echo chamber, while discussion across very different devs is usually far more interesting.
- floober 5y agoNote that this doesn't appear to have been submitted by the post's author, and HN may not have been the intended audience.
- abdusco 5y agoSimon keeps his TIL (today I learned) blog as a way to take notes mainly for himself, from what I understand. I like the way he documents everything in Github issues[0]. I learn a new thing or two with every release of Datasette, because there's so much troubleshooting full of relevant links and solutions. Same with his TIL blog. [0]: https://github.com/simonw/datasette/issues/1576 https://github.com/simonw/datasette/issues/1576
- DonaldPShimoda 5y agoI've not seen this before but it's so cool! I've lost track of all the ways I've tried to keep track of things I've learned. Maybe this would be a good way to do it. Link to the GitHub repo for those interested: https://github.com/simonw/til https://github.com/simonw/til
- godelski 5y agoCould you, or someone else, clue us in? Edit: another user linked their blog https://news.ycombinator.com/item?id=29813349 https://news.ycombinator.com/item?id=29813349
- Macha 5y agoOne use is for e.g. binding of database models for an ORM. class FooModel(metaclass=Base): x = Field(type=int, primary_key=True) y = Field(type=reference(BarModel)) The Base metaclass will set it up to implement methods like save() by inheriting from parent classes, but it would also be nice for the library to have a list of all model types without the library user having to call a method like FooORM.register_type(FooModel). So the metaclass is being used in these classes to build up a dictionary of models when the class definition is encountered. The metaclass is basically a class that itself builds classes, which means it can be syntactically convoluted. However, with __init_subclass__ you can write a thing that looks like a regular class with regular parent methods, but instead just gets a method called each time the interpreter encounters a new subclass, which lets you do things like build up that dictionary for your ORM.
- wheelerof4te 5y agoPython is slowly going in the direction of C++. Sad to see, really.
- imaurer 5y agoURL is to a "Things I Learned" by Simon W, but the title comes from tweet by David Beazley: https://twitter.com/dabeaz/status/1466731368956809219 https://twitter.com/dabeaz/status/1466731368956809219 Which is a warning that people shouldn't buy his new book if they want a deep dive on Metaprogramming: https://www.amazon.com/dp/0134173279/ https://www.amazon.com/dp/0134173279/ Simon does a nice job demonstrating how "the __init_subclass__ class method is called when the class itself is being constructed."
- 3pt14159 5y agoI'm working on a project right now that uses __init_subclass__ and it works great, but takes some tricks (like having classvars with types set) in order to get mypy to cooperate. But it's way easier to reason about than metaclasses.
- matsemann 5y agoHow does this play with typing and other tools? I hate the use of all the magic stuff in Python making it needlessly hard to use lots of libraries.
- sillysaurusx 5y agoIt's fine. https://news.ycombinator.com/item?id=29812639 https://news.ycombinator.com/item?id=29812639
- zbyforgotpass 5y agoI wish there was a 'Modern Python' tutorial that would walk me through all the new python stuff like this or the type declarations or the new libs, etc.
- diarrhea 5y agoI learned about metaclasses in "Fluent Python". An excellent book, but it predates typing, so that's not covered. Then again, I believe a second edition is on its way.
- jjtheblunt 5y agodocs.python.org has helped me track changes, as they have a section always on what's new.
- usrme 5y agoIf you're not averse to books I'm yet again recommending "Robust Python" by Patrick Viafore and for more esoteric things like metaclasses I would recommend "Expert Python Programming, 3rd Edition" by Michal Jaworski & Tarek Ziade. It's not wholly dedicated to metaclasses, so there are tons more to learn from it as well.
- echelon 5y agoI'm at a point in my career where I see "magic" and wince hard. This is so hard to reason about and fix at scale. If you let other engineers run wild with this and build features with it, the time will eventually come to decom your service and move functionality elsewhere. Having to chase down these rabbits, duplicate magic, and search large code bases without the help of an AST assisted search is slow, painful, and error prone. I spent several years undoing magic method dispatch in Ruby codebases. Tracing through nearly a thousand endpoints to detect blast radiuses of making schema changes impacted by CRUD ops on lazily dispatched "clever code". I'm sure there are valid use cases for this, but be exceedingly careful. Boring code is often all you need. It doesn't leave a mess for your maintainers you'll never meet. Python users tend not to behave this way, but seeing posts like this requires me to urge caution.
- jhgb 5y agoWasn't the idea behind this to make most code boring, and have only a small part of it exciting? Just like when structured programming was created, or even local variables introduced, etc. (Of course, the exciting part had to be moved into the compiler in those cases.)
- radicalbyte 5y ago..and there was me thinking that it was only Java which suffered from that problem. That kind of code is great if: (a) you wrote it (b) you knew what you were doing when you wrote it and (c) you also wrote an extensive set of tests to cover the entire domain of whatever you were doing.
- tentacleuno 5y ago(d) and you used comments! Comments are underestimated. I'm not saying they should be used as an excuse to use "magic" esoteric code, but sometimes such solutions are needed and it's much better to document them for future maintainers.
- zzzeek 5y agoYeah this is great, __init_subclass__ comes along to make dynamic injection of methods and attributes on class creation easier, just as the entire practice is fast becoming fully obsolete because type checkers like pylance and mypy report these attributes and methods as errors.
- sillysaurusx 5y agoThere are a few ways to deal with that. One is to declare the properties directly in the subclass: class Inject: def __init_subclass__(cls): cls.injected = 42 class Child(Inject): injected: int print(Child().injected) # 42 This technique isn't too useful in practice, though it has its place (e.g. when creating ctypes.Structures). But there's a clever way to simplify this. Remember that the subclass is a subclass -- it inherits from Inject: import typing as t class Inject: injected: t.ClassVar[int] def __init_subclass__(cls): cls.injected = 42 class Child(Inject): pass print(Child().injected) # 42 (ClassVar[int] ensures that type checkers know it's a variable attached to the class, not each instance.)
- zzzeek 5y agoyes you can do that, but if you want to do say, what dataclasses does, where it collects all the Field objects and creates a typed `__init__` method, typing can't do that without plugins. all the current typecheckers hardcode the specific behavior of "dataclasses" without there being any pep that allows other systems (like ORMs) to take advantage of the same thing without writing mypy plugins or simply not working for other type checkers that don't allow plugins.
- sillysaurusx 5y agoYeah, that’s true. Dataclass support is severely lacking. Is there a pep on the horizon for this?
- zzzeek 5y agoim not really sure (highly doubtful, much simpler ideas like being able to make your own Tuple subclass are still not ready for primetime) but this would be a pretty tough pep to allow the general "dataclass" concept to be available to other libraries that want to use the same pattern, but not use dataclasses.
- kkirsche 5y agoEven using Python regularly I haven’t run into a need for meta classes. I’m sure there is a valid reason, but what is it?
- tybug 5y agoThe most salient example I can think of is tracking all subclasses of a class. This is how django tracks `Model` subclasses to create a table for each model, for instance. I wrote a blog post on this topic: https://tybug.github.io/2021/11/06/tracking-all-subclasses-of-a-class.html https://tybug.github.io/2021/11/06/tracking-all-subclasses-o... I've also used metaclasses in the past to apply a decorator to certain methods of all subclasses, without needing to specify that decorator in the subclass declaration itself.
- dbrueck 5y agoCame here to say this. This is my most common use of metaclasses, and __init_subclass__ looks to be a much cleaner solution.
- fdomingues 5y agoIs useful to do code generation/transformation, for example in this utility[1] it uses the information provided on the declaration of the class to generate the methods required by the base class. The user of the utility write this: class Book(objects.Object): title: Optional[str] And the generated class code is: class Book(Base): __slots__ = ('title', ) title: typing.Optional[str] def __init__(self, title=None): self.title = title @classmethod def from_data(cls, data): title = data.get('title', None) return cls(title=title) def to_data(self): data = {} if self.title is not None: data['title'] = self.title return data @classmethod def from_oracle_object(cls, obj): return cls(title=obj.title) def to_oracle_object(self, connection): obj = connection.gettype('Book').newobject() obj.title = self.title return obj [1]https://github.com/domingues/oracle-object-mapping https://github.com/domingues/oracle-object-mapping
- 5y ago
- dreyfan 5y agoPython is quickly taking the crown for low-barrier to entry, slow, buggy code supported primarily by stackoverflow copy pastes from a never-ending supply of “data scientists”
- ransom1538 5y agoI can write php in any language.
- dreyfan 5y agomodern php and js/ts are both rather impressive how far they’ve come compared to the jumble of crap we had a decade ago.
- sam0x17 5y agoI miss it. On the javascript side there was just jquery and a mountain of vanilla js that everyone could at least understand (and no npm dependency issues, ES6+ weirdness, etc) and on the PHP side, all built-in functions were in the global scope (easy to CTRL+F on a single page at a time when search endpoints were quite slow), there were (virtually) no web frameworks (at least I didn't use any until after 2009), and no package management to worry about unless you were doing something weird and/or wrangling with wordpress. As a tween and young teen I had plenty of small web design clients.. local businesses, photographers, etc., and I could build sites 100% from scratch starting from a photoshop mockup to custom PHP/HTML/CSS/JS, 100% no packages or other people's code. Packages have done a ton for us, for sure, but I truly miss those days and feel like we've lost something we'll never get back. Back then it was actually quite possible to build an entire website or "app" as we call them now from first principles just banging your head against HTML/CSS/JS and whatever server side language you were using. Then the internet grew up, security issues became more prevalent, packages, other people's code, dependency hell, "futures", "promises", server state and client state management, shadow dom, reactivity, etc etc.
- robervin 5y agoQuickly seems like an overstatement; you've always been able to write low-barrier to entry, slow, buggy code. Is there a low-barrier language that doesn't end up with a lot of less-than-ideal code samples?
- chadykamar 5y ago“Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).” — Tim Peters
- tybug 5y agoI realize this quote isn't meant to be taken literally, and it's a nice one-liner encapsulation of metaclasses, but it always bothered me. The people who "actually need metaclasses" were, at some point, learning about metaclasses for the first time and were "wondering whether they need them". This quote ignores the middle ground of users who have a valid use case, but don't yet understand metaclasses. Not great advice for people who are trying to learn metaclasses.
- nas 5y agoI suspect Tim's meaning was that ordinary users don't need to know how metaclasses work in detail. It is enough that people know they exist (i.e. do something at time class is defined). If you need them, you are doing deep enough magic that you can take the time to learn how they work. Maybe he was also suggesting that people should generally not be using them. In my experience, it is exceedingly rare to need them. In 29 years of writing Python code, I think I've used them once or twice.
- jcranmer 5y agoI do sympathize with this sentiment, but there are times when "if you're wondering if you need it, you don't need it" is a very true statement, and Python metaclasses are in that boat. To see why, let me explain in more detail: There's a "basic" [1] understanding of Python's object model, one which understands that most of Python is actually syntactic sugar for calling certain special methods. For example, the expression a[b] is "really" just calling a.__getitem__(b). Except that's not actually true; the "real" object model involves yet more dispatching to get things to work. Metaclasses allow you to muck with that "yet more dispatching" step. So when do you need metaclasses? When you need to do that low-level mucking--the kind of mucking most won't know about until actually needed. If all you know about metaclasses is kind of what they are, then you very likely haven't learned enough about them to use them to actually need them. Conversely, if you've learned those details well enough to need to muck with them, then you've also learned enough to the point that you can answer the question as to whether or not you need metaclasses. [1] I suspect most Python users don't even have this level of understanding, which is why I put it in scare quotes.
- fdgsdfogijq 5y agoI once heard a quote that described David Beazely as: "The Jimi Hendrix of programming"
- scottwitcher83 5y ago
- captainmuon 5y agoI've only once used metaclasses but I think it was pretty nifty. The use case was with Python and Gtk+ (both version 2), if you had your UI in a glade file you could define a corresponding class, and it would automatically create members for all your named widgets. It made it a bit like coding in VB: class MainWindow(gtk.Window): __metaclass__ = MetaDescribedUI("MainWindow.ui") def __init__(self): # now you can use e.g. self.lvFiles or self.btnOK pass Although writing it now I could probably do it differently without metaclasses.
- sam0x17 5y agoI love that Python is unique and strange enough that merely seeing "__init_subclass__" in this headline is enough to know exactly what language it is talking about ;)
- Naac 5y agoFor those curious, here is a great explanation of what metaclasses are in python: https://stackoverflow.com/a/6581949 https://stackoverflow.com/a/6581949
- joebergeron 5y ago(Shameless self-plug incoming…) For those curious about what metaclasses actually are, I wrote up an article with some motivating examples about this a little while back. Might be worth a read if you’re interested! https://www.joe-bergeron.com/posts/Interfaces%20and%20Metaclasses%20in%20Python/ https://www.joe-bergeron.com/posts/Interfaces%20and%20Metacl...
- VWWHFSfQ 5y agoThere are bugs in this code and I'm glad that I'm not the only one that has done it! graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("_") } for key, method in cls._registry.items() } The first parameter to a bound method does not have to be called `self`. It's conventional, but not required. Is there a better way in the inspect module to filter these parameters? This comes up more often with classmethods where the naming convention `cls` is most common but I see `klass` somewhat frequently as well.
- deleted 5y ago[deleted]
- GVRV 5y agoThis got me thinking and I'm not even seeing the `self` being returned in the list of parameters on Python 3.8.5: >>> class Test: ... def test(self, a, b, c=5): ... return a + b + c ... >>> t = Test() >>> inspect.signature(t.test).parameters mappingproxy(OrderedDict([('a', <Parameter "a">), ('b', <Parameter "b">), ('c', <Parameter "c=5">)]))
- Starlevel001 5y agoThat's because ``t.test`` returns a MethodType instance rather than a FunctionType instance, which is invoked without the self.
- GVRV 5y agoAh, you're absolutely correct! In [4]: class Test: ...: def test(self, a, b, c=5): ...: return a + b + c ...: In [5]: inspect.signature(Test.test).parameters Out[5]: mappingproxy({'self': <Parameter "self">, 'a': <Parameter "a">, 'b': <Parameter "b">, 'c': <Parameter "c=5">}) Then, any idea how you would address the GP's original point? How should "self" be detected if it can be called something else?
- ajkjk 5y agoFor all of Python's magic hackery that it lets you inject it feels like there are still some obvious 'holes'. The one that bites me a lot is: I want to easily be able to write a decorator that accesses variables on a particular _instance_ of a class (such as a cached return value), and I want to take a Lock when I do it. But since decorators are defined on the class method, not the instance, it has to do a bunch of work when the function runs: look at the instance, figure out if the lock is defined, if not define it (using a global lock?), take the lock, then do whatever internal processing it wants. It feels like decorators should have a step that runs at `__init__` time to do per-instance setup but instead I have to figure it out myself.
- Spivak 5y agoI'm so confused. from functools import wraps from threading import Lock def my_decorator(f): # Do whatever you want to set up the lock. lock = Lock() @wraps(f) def wrapper(self, *args, **kwargs): # Do whatever with the lock. lock.acquire() # Do whatever you want with the instance. print(self.greeting) return f(self, *args, **kwargs) return wrapper class MyClass: def __init__(self): self.greeting = "Hello World!" @my_decorator def my_method(self): print("And all who inhabit it.") >>> MyClass().my_method() Hello World! And all who inhabit it. Edit: Wait do you want the decorator, before it ever runs, to add a property self.lock? Are you sure you don't want this to be a class decorator that messes with __init__? Because you can 100% wrap init to do that kind of set up.
- ajkjk 5y agoThe problem is that `lock = Lock()` is shared across all instances of the class, which creates contention if there are many of them. To solve this you need to save a per-instance lock, so yes to "Wait do you want the decorator, before it ever runs, to add a property self.lock". An example usecase is writing a @cached decorator that behaves well in multithreaded programs and can be used on classes that have many instances, with the caching happening per-instance.
- wheelerof4te 5y agoA recent anegdote with regards to typing in Python. Let's say you have this code: class Creature: def __init__(self, name: str, hp: int, attack: int): ... def attack(self, other: Creature): ... Guess what? Since Creature is not yet defined, your very smart type-checker can't see the "other" parameter. Happened to me in VS Code. That is the problem with features bolted on to a language where they perhaps don't belong. Now, I know that typing helps when you have simple, functional code. So, this code will work: from dataclasses import dataclass @dataclass class Creature: name: str hp: int attack: int def combat_between(attacker: Creature, defender: Creature): ... I was really surprised with this. We need better review in regards to adding additional features that may be broken on certain setups.
- orf 5y agoSimply add quotes around “Creature” and it works. > We need better review in regards to adding additional features that may be broken on certain setups. What does this even mean. The documentation explains that you need to do this, and it’s fairly obvious why you would. Type hints are one of the best features added to Python.
- wheelerof4te 5y agoYeah, "simply add quotes", indeed. Except, it is not that simple. It's not even intuitive. Typing is a great feature on paper, but in practice, it is just something we don't need in a scripting and glue language.
- 5y ago
- amelius 5y ago__This__ __looks__ __very__ __interesting__!
- cutler 5y agoEvidence, if any was needed, that OOP, or at least the Python variant, was never designed for metaprogramming. Give me Clojure macros any day of the week over these ugly contortions.
- contravariant 5y agoYou kind of have macros with the decorators. Which in most cases are a way saner alternative.
- tsujamin 5y agoI recently encountered __init_subclass__ for the first time in Home Assistant's custom integrations. Configuration handlers were registerd by "just declaring the class" like so and I couldn't grok how it was registered by the runtime class MyCustomOptionsFlow(OptionsFlow, DOMAIN="MYPLUGIN"):... A bit of digging showed that __init__subclass_ was being used effectively as a registration callback, adding the newly defined class/DOMAIN to the global CustomOptions handler at class definition. Very neat, not immediately obvious however :/
- wheelerof4te 5y agoIsn't it ironic that one of the most readable procedural languages today has almost unreadable OOP syntax? People unironically created interfaces and various overloaded operations using dunder methods. Not only do you need to keep a list of them under your pillow, but you must also keep notes on how to construct the interface for each one. And they look ugly as hell. In a language that prides itself on readability and user friendlines. My suggestion is to ditch the OOP syntax completely and make a built-in analogue to a C struct. Take inspiration from dataclasses syntax. OOP in a dynamic glue language is a silly idea anyway. Of course, you would have to bump the version to 4.0.
- Garlef 5y agoWhat's the advantage of this over class decorators? @decorator class SomeClass: ... To me, class decorators seem to be much easier to reason about.
- Garlef 5y agoAh... https://stackoverflow.com/questions/1779372/python-metaclasses-vs-class-decorators https://stackoverflow.com/questions/1779372/python-metaclass...
- abdusco 5y ago>>> import this > ... > Explicit is better than implicit. > ... > There should be one-- and preferably only one --obvious way to do it. It's one of Python's philosophies (/s). I agree that this is a more explicit way to handle metaprogramming in Python. It's analogous to method decorators, unlike metaclasses, which has no "metamethod" equivalent (AFAIK).
- jwilk 5y agoOfficial docs for that feature: https://docs.python.org/3/reference/datamodel.html#object.__init_subclass__ https://docs.python.org/3/reference/datamodel.html#object.__...