10 ms·
Crimes with Python's pattern matching
- wheaties 4y agoThis is why adding language features needs to be carefully thought out and explored to great lengths. Sadly C++ never got that memo. Here's hoping Python remains "simple."
- pizza 4y agoIs it better for Python to be simple than for Python to be simple for simple problems?
- Waterluvian 4y agoI’m not sure I understand the “this is why” part. Python wants to let you override almost everything. It’s a feature and a powerful and occasionally helpful one. It’s also a feature you never ever have to know about or touch.
- Blackthorn 4y agoIt's like python's metaclasses. You rarely need them but sometimes they really are just the best solution to the problem. Those times, you're really glad they're available.
- jesushax 4y agoThis debate could be summed up as: There's more than one way to do it *vs* there should be one, and preferably only one, obvious way to do it
- DangitBobby 4y ago> there should be one, and preferably only one, obvious way to do it Loops, list/dict comprehensions, if statements, and ternary expressions would all like a word
- jesushax 4y agoThe same debate applies to all of those things, so this is just another instance of Python battling with a philosophical principle underlying the language, which naturally arises anytime a redundant (but good) feature like this is added to the language
- jamesfinlayson 4y agoAgreed - I mostly use Python for small scripts and it makes that use case very easy. I know it has a bunch more features for more niche stuff as well but my throwaway script to download images from a webpage doesn't need pattern matching.
- kazinator 4y ago> It’s also a feature you never ever have to know about or touch. The "never have to know or touch" argument applies only to the lone hacker working on a completely new project with no inherited legacy code.
- zqlark 4y agoIndeed. Most Python code bases I have seen are maxing out all obscure features like those in the article. In practice, Python today is one of the most unreadable languages in existence.
- pansa2 4y agoIt’s quite clear from the design of Python, both of recent features and of fundamentals like `+=`, that its designers never got that memo either. I’m baffled as to how it retains its reputation of being a simple language.
- shepherdjerred 4y agoI think it’s a pretty simple language for how powerful it is. But it’s not simple compared to some other languages like Go
- defen 4y agoWait, what's the deal with `+=` ?
- CGamesPlay 4y agoThe obvious x = x + y and the also-works x += y. There is more than one way to do the operation. The conceptually more simple way would be the former as it does not require the programmer to know an extra operator.
- patrec 4y agoIt can mean two completely different things. `let x = x + 1` (rebinding) or `x += 1` (mutation). In [5]: def cap(x): ...: xs = [] ...: for i in range(3): ...: x+=1 ...: xs.append(x) ...: return xs ...: ...: In [6]: cap(numpy.array(0)) Out[6]: [array(3), array(3), array(3)] In [7]: cap(0) Out[7]: [1, 2, 3]
- pansa2 4y ago`a += b` sometimes does the same thing as `a = a + b`: >>> a = b = (1, 2) >>> a = b = (1, 2) >>> a += (3,) >>> a = a + (3,) >>> print(a, b) >>> print(a, b) (1, 2, 3) (1, 2) (1, 2, 3) (1, 2) Sometimes it does something different: >>> a = b = [1, 2] >>> a = b = [1, 2] >>> a += [3] >>> a = a + [3] >>> print(a, b) >>> print(a, b) [1, 2, 3] [1, 2, 3] [1, 2, 3] [1, 2] IMO if that behaviour had been "carefully thought out" by the language designers, it should have been obvious that it's a bad idea. Failing that, the implementation of that behaviour is convoluted - so the designers should have paid attention to the Zen of Python: "If the implementation is hard to explain, it's a bad idea". Failing that, if the behaviour had been "explored to great lengths", the designers would have understood how it interacts with other language features - in particular, nested mutable and immutable objects. Python's designers failed to do any of these things, so we've ended up with an operator with unpredictable behaviour and a long FAQ entry [0] about how its possible for an operator to both succeed and fail at the same time: >>> a = ([1, 2], 4) >>> a[0] += [3] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'tuple' object does not support item assignment >>> a ([1, 2, 3], 4) [0] https://docs.python.org/3/faq/programming.html#why-does-a-tuple-i-item-raise-an-exception-when-the-addition-works https://docs.python.org/3/faq/programming.html#why-does-a-tu...
- petters 4y agoI like Python, but it really isn’t “simple” (at least not for more than a decade).
- pjmlp 4y agoAll languages got the memo, including C++. Either you get Scheme, or languages with features. Even C isnt' as "simple" as people take it to be.
- ajkjk 4y agoWhat a delight. Still like pattern matching in python though. Seems super useful.
- fny 4y agoThis hack also seems super useful if used judiciously... perhaps one could write a spicy DSL.
- sanxiyn 4y agoPython is not the first language to have customizable pattern matching. Scala can do the same with extractor objects.
- lmm 4y agoYep. Honestly this seems fine - pattern matching is ultimately just a function call, you can immediately see where it is (and hopefully even click through to the code from where the pattern is defined), there's no "magic" action-at-a-distance.
- sanxiyn 4y agoCustomizable pattern matching is in fact unusual, although not unheard of. Traditional pattern matching is very much not a function call, because it is compiled with assumption that it can't be customized and no arbitrary user defined code is executed. In particular, usual compilation scheme guarantees that subterms are not matched more than once, but that can't be guaranteed with customizable patterns.
- nadavr 4y ago> But surely Python clamps down on this chicanery, right? > > $ py10 abc.py > 10 is not iterable > string is iterable > [1, 2, 3] is iterable > > Oh. > > Oh my. I'm sure I'm being dense and missing the obvious but ... what is the author responding to here? What's wrong or bad?
- davesque 4y agoEh, I think they're just overdramatizing things. That's also exactly what I would have expected to happen.
- atoav 4y agoThe only thing that can sometimes bite you here is that str is iterable, if you expect a list of str and you only get a str and suddenly you iterate over the chars. I am not sure if it wouldn't have been better to make the conversion explicit here.
- vitus 4y ago> The only thing that can sometimes bite you here is that str is iterable, if you expect a list of str and you only get a str and suddenly you iterate over the chars. Python isn't even unique in this regard! You can iterate over a string whether you're working in JavaScript, C++, or Go. (And that's not even getting into cases like Haskell where String is merely syntactic sugar for [Char].)
- shadycuz 4y agoHe is just demonstrating that __subclass__ hook has control over what is counted as a match. Which he explained in another article that it allows the author of the abstract class to hijack calls to isinstance for any instances created from subclasses.
- bilboa 4y agoThe point though is that the tone of his article seems to suggest that this is some scary "gotcha" of the language, whereas some of us consider this to just be the expected behavior.
- wdroz 4y agoPattern matching in Python is not perfect, but I still thinks this is a cool feature. The ergonomic doesn't match Rust pattern matching as explained in this recent discussion[0] but it's better than nothing IMO. [0] -- https://www.reddit.com/r/Python/comments/w4xoxb/anybody_else_feel_like_match_could_be_an/ https://www.reddit.com/r/Python/comments/w4xoxb/anybody_else...
- davesque 4y agoI didn't see the need to frame everything in the article as being so bad. I thought all those examples were awesome. I'll probably never do anything like them so as not to confuse people but I still think they were all interesting uses of the feature. Honestly, I think people are too hard on languages (and especially Python) for having new features that challenge the status quo. And then there's also too much drama when it turns out that a scripting language is, in fact, a scripting language! So you can do weird things with stateful ABCs and such. I mean yeah, it's strange. But it probably also has some perfect use case in a very specific circumstance. At the end of the day, if you understand how a feature really works, you can do creative things with it. I'm glad we have it!
- deleted 4y ago[deleted]
- contentboot 4y ago>I thought all those examples were awesome. They are great examples. Great examples of what not to do. The only use case I can think of is meta programming which most people don't need.
- yarg 4y agoIt made me want to learn python. I've been thinking about programming language development, and some weird things from the typing front. This could be easily used to achieve unions. But another thing that I was thinking about was the return from unsafe land. You have some class or object that you want to do some quasi-illegal fucky bullshit to - send it off and do what you will. But what comes back might not have the guarantee that it's still the 'shape' of the thing that you sent away. This could be used to validate that what comes back is sane.
- deleted 4y ago[deleted]
- pizza 4y agoHell yeah, another reason to get people to switch to Python >= 3.10. Unfortunately a lot of libraries are still gonna be behind for now: https://pyreadiness.org/3.10/ https://pyreadiness.org/3.10/ - of the top 360 most downloaded on PyPI, 212 have explicit Python 3.10 support. This plus the walrus operator, X | Y -style union types, and the speedup possibilities of 3.11 all look great, imo
- paulmd 4y ago41% of the top packages don't support the new version? And looking back, the numbers aren't even much better for previous versions, 3.8 is several versions old now and they've still got over 20% breakage. That is actually pathetic and I'm not blaming the package authors here - Python needs to stop making big breaking changes that rototill the codebase continuously. Breaking changes are pretty serious business in the Java world and there is an incredible amount of thought and research put into even something like modules let alone the JDK17 changes where reflection is being fundamentally changed. Python seems to have an absolutely carefree attitude to language breakage, and I guess why wouldn't they? It's always worked for them.
- Hamuko 4y agoExplicit support, meaning that the package author has added the "3.X" tag in the package metadata.
- wtallis 4y ago> is several versions old now and they've still got over 20% breakage. I don't think that's illustrating breakage, just the lack of an explicit declaration that the package supports a newer version of python (which may be newer than the latest release of a given package).
- missingcolours 4y agoYeah... I always develop against the latest Python available and have probably run into 1 package in my career with a version incompatibility, and that was with a C API portion. 20-40% is not correct.
- jenrzzz 4y agoWhy the heck is case Not(DistanceMetric)(): a syntax error?
- momothereal 4y agoBecause the right arm of the 'case' keyword is not actually a statement being executed, but its own syntax element to represent a pattern. It is not expecting two sets of brackets there.
- deleted 4y ago[deleted]
- chrismorgan 4y agoWhat follows the keyword “case” there is a pattern, not an expression. It’s not an imperative construct of code to be executed, but a declarative construct of an expected shape of an object, which may include name bindings. Consider things like “case DistanceMetric(distance=d):” earlier in the article: this checks “is the value an instance of DistanceMetric, and if so, take its distance attribute and bind it to the name d”. So in this case, what would it mean? If the value is an instance of Not, take and bind it to the DistanceMetric name (as is typical for a single positional subpattern), and… uh oh, more parentheses, what to do? There’s no obvious sensible meaning for it, so it’s a syntax error.
- psd1 4y agoThanks, that's helped me with pattern matching in C#
- rileyphone 4y agoThe real reason why you shouldn't use this is that Python is wrong about typing and classes. In OOP, classes are not types; interfaces are [0]. ABCs are a poor replica of this. Guido, having clearly avoided any relevant literature, states, in PEP-3119 [1]: > ABCs are not intrinsically incompatible with Interfaces, but there is considerable overlap. For now, I’ll leave it to proponents of Interfaces to explain why Interfaces are better. Let me try: interfaces are better because the protocol of an object isn't tied to its implementation, but in a properly encapsulated world an interface represents the information available about a class as a type [2]. A subclass may just be reusing an implementation without adhering to the same protocol, or two interchangeable classes might have no inheritance relationship. Python is in a lot of ways a nice language, and I've certainly enjoyed programming in it, but many points of its design seem intentionally unobservant of prior work and research in programming languages, though perhaps it's equally an indictment of that research that the most popular languages ignore it so much. Typescript handles this much better, though neither it or Java eschew using classes as types entirely. [0] https://dl.acm.org/doi/10.1145/96709.96721 https://dl.acm.org/doi/10.1145/96709.96721 [1] https://peps.python.org/pep-3119/#abcs-vs-interfaces https://peps.python.org/pep-3119/#abcs-vs-interfaces [2] https://dl.acm.org/doi/10.1145/2509578.2514738 https://dl.acm.org/doi/10.1145/2509578.2514738
- pjmlp 4y ago> In OOP, classes are not types; interfaces are [0]. Depends on which OOP language we are talking about, Smalltalk definitly doesn't have interfaces unless we are talking about later dialects like Pharo, which introduced traits into the language. The paper you linked to, makes its point exactly by moving beyond Simula and Smalltalk into their own view of OOP. So like anything else on the OOP ecosystem, it is only yet another view about what OOP should be like.
- lqatrb 4y agoYes, but ABCs and also method resolution are particularly hackish in Python. Generally, in Python one always has to understand the implementation and mentally execute the code, because everything is informally specified and nothing is declarative.
- calibas 4y agoDoes anybody have an example of when __subclasshook__ is truly useful? Something that can't be accomplish more coherently with a simple function? Personally, I'd prefer that isinstance() and issubclass() have predictable behavior.
- qbasic_forever 4y agoA lot of these dunder (double underscore) functions are useful for metaprogramming. Just kinda spitballing an idea but perhaps some code to generate a python object model based on the schema of a database would want to use this method. If your DB schema has some special way of defining subclass relationships (maybe a foreign key to another table) you might need to manually control when something is or isn't a subclass in python's object model based on the result of querying the DB schema. In general metaprogramming is the kind of thing you probably don't and shouldn't reach for first, in fact it's usually more for libraries and tools vs. your production business logic. It can get difficult to reason about and pass the maintenance of code that heavily uses metaprogramming to other people unfamiliar with it.
- uranusjr 4y agoThe most dominate use (these days, at least) is to implement structural typing (a la Protocol), i.e. conforming to a “shape” without actually inheriting anything. So yeah, it’s not particularly useful for day-to-day use, but still a hook needed to make certain nice things happen behind the scenes.
- motbus3 4y agoit allows fast unpacking and parameter retrieval align with the usual type checks. the other day, this allowed me to refactor a 300L in 50 which are actually readable
- yarg 4y agoIt could be used to define a union implementation.
- ThePadawan 4y ago
- sharmin123 4y ago
- deleted 4y ago[deleted]
- bandrami 4y agoIDK, this is roughly how CLOS works and it's largely held to be a Good Thing: classes are simply arbitrary sets of predicates a value does or doesn't meet.
- motbus3 4y agothe usages he complains like in the second example are wrong. you are expect to not send a obj but something to have the properties matched.
- IshKebab 4y ago__subclasshook__ is exactly the sort of nonsense that makes Python code a complete mess and impossible to optimise. I'm going to take a wild guess that none of the "fast" Python variants (Cython, Micropython, RPython etc.) support it.
- dagw 4y agoI'm going to take a wild guess that none of the "fast" Python variants (Cython, Micropython, RPython etc.) support it. Cython at least aims to be a superset of Python, so it will support it sooner or later. However I don't doubt that using it will make your cython code stop being "fast".
- hirebackenddev 4y agoSurely an amazing post. Keep sharing such posts.
- noelwelsh 4y agoReading this I can't help feeling that Python puts the "simplicity" in all the places that don't matter. Simplicity is the reason given for not having useful language features like pattern matches as expressions, or lambdas with multiple expressions, but I've never seen these features[1] cause problems in other languages. And then we have this... Surely semantic simplicity is the simplicity that actually matters, but the way the simplicity argument is used in Python is often to enforce arbitrary syntactic and semantic complexity. [1] It's almost ridiculous to call these features as they're just a consequence of the underlying language model. E.g. if you have expression blocks you have lambdas with multiple expressions with no extra work.
- Doxin 4y agoPython has plenty of complexity. It tends to stick it in places where new programmers don't run into it. Python is one of the few languages I know that lets you just... poke at the internal machinery like this.
- kmacdough 4y agoHave you ever messed around w/ Julia? It doesn't really have much "internal machinery" since it relies on a data-oriented/functional language structure. I find myself reading the Julia source code on a regular basis since it's very readable and succinct. I often find it more useful than the documentation itself. It can take a bit to fully understand/appreciate Julia's multiple dispatch, but once you do you pretty much understand the entirety of the machinery.
- saagarjha 4y agoOr performance! There's lots of low-hanging fruit in the Python interpreter that doesn't get improved to preserve the purity of the runtime, or whatever. (Well, at least this might see improvements now. But for a long time people would point at it and laugh.)
- colonwqbang 4y agoOr to enable the kind of extreme dynamism that is illustrated in TFA. How to optimize code properly when even core relationships like "x is of type T" may be nondeterministic.
- dvt 4y ago> That made me wonder if ABCs could “hijack” a pattern match. Something like this: I guess the word "hijack" is used loosely for rhetorical effect here because this seems to be working as intended, and it's not even remotely the most dangerous footgun in Python. The problem (if any) is with `isinstance`, and not the pattern matching. `isinstance` should probably explicitly work with ABCs (via flag or something) because I do agree it's a bit weird that it takes the ABC's `__subclasshook__` as gospel by default.
- CGamesPlay 4y agoIt seems like the main problem here is that Python does not enforce that the subclass hook method is pure and so allows you to create buggy implementations. The spec should mandate that the method is pure and causing side effects should be a runtime (or compile time) error.
- w-m 4y agoLooking in the implementation of the Python version of ABC[0], you'll see that `__subclasshook__` doesn't do any language magic, it just registers the method to be called in `__instancecheck__`. So if you just implement `__instancecheck__` directly, you get the same behavior, but without the caching around it. class OneWayMeta(type): seen_classes = set() @classmethod def __instancecheck__(cls, instance): C = instance.__class__ print(f"trying {C}") if C in cls.seen_classes: return False cls.seen_classes |= {C} return True class OneWay(metaclass=OneWayMeta): pass def f(x): match x: case OneWay(): print(f"{x} is a new class") case _: print(f"we've seen {x}'s class before") if __name__ == "__main__": f("abc") f([1, 2, 3]) f("efg") When running: trying <class 'str'> abc is a new class trying <class 'list'> [1, 2, 3] is a new class trying <class 'str'> we've seen efg's class before Am I missing a particular point the article is making, or did the author overlook this? [0]: https://github.com/python/cpython/blob/main/Lib/_py_abc.py https://github.com/python/cpython/blob/main/Lib/_py_abc.py
- Joker_vD 4y ago> if you just implement `__instancecheck__` directly The point of the article is that to override what "isinstance(obj, ClassA)" means one doesn't need to touch ClassA, or any descendants of it, at all.
- nadavr 4y agoCan you help me understand what you mean here? The author manages to make something match `case NotIterable()` by...modifying the `NotIterable(ABC)` class. That's exactly what I would have expected. What you mean by "one doesn't need to touch ClassA, or any descendants of it, at all."?
- w-m 4y agoNot quite! The point of the article is that you don’t need to change the class of obj to override this instance check. If you have a look, ClassA always uses the hook implementation in the examples. And this can be shortened to just using instancecheck.
- heavenlyblue 4y agoThe same author would say something similar about C/C++ where by obtaining the address of a function and then writing a value to that address you can change the code of the program. Yes, you are misusing the API. No, there ARE valid use-cases for it: for example a lot of testing/mocking facilities in Python are so easy to implement because of these features. No, the fact that you can't do it in statically typed languages does not imply Python must go the same way.
- wwilim 4y agoStrong duck typing - you go to extreme lengths to ensure that the object can indeed quack
- planede 4y agoSo this is mostly about __subclasshook__. How does this interact with mypy type checking? There must be some tension there too. edit: I guess I found my answer [1]. That's kind of expected, but it's still ugly. https://stackoverflow.com/questions/71811960/how-to-use-subclasshook-with-mypy https://stackoverflow.com/questions/71811960/how-to-use-subc...
- HelloNurse 4y agoEven if someone would like Python to pretend to be statically typed, __subclasshook__ is real (and not even type checking in the traditional sense) while type declarations and their checkers are only quasi-executable documentation. Personally, I think this is a very good situation: extremely dynamic languages are a worthwhile tool, and the only issue with the pattern matching "exploits" in the article is that boolean operators and non-cached evaluation for subclass checks are not built-in.
- jerjerjer 4y ago> That made me wonder if ABCs could “hijack” a pattern match What's the hijack there? I'm really not sure what's the author's point is. He writes the code and get's expected results back.
- Izkata 4y agoBased on other comments here, it seems like some people don't know the purpose of ABCs and are assuming they're more like a class definition for classes that have to be directly inherited.
- kortex 4y agoI've been programming python for years, consider myself extremely knowledgeable in it, and...uhh... yeah TIL I've been using ABCs wrong.
- BiteCode_dev 4y agoSo, you can use duck typing and metaprogramming in Python, even with modern features? How is that a crime? Next, you will be surprised that you can use first class functions and late variable binding?
- deleted 4y ago[deleted]