8 ms·

Python-patterns – A collection of design patterns/idioms in Python

262 points by gilad 4y ago

axblount 4y ago
I'm confused by the Borg pattern. It seems like a singleton that tricks the user into believing there are multiple instances (very bad imo). Is there any advantage over singleton? Maybe some kind of `__dict__` behavior I'm not aware of?
- sillysaurusx 4y ago
  I showed up 30 seconds after you posted your comment to say the exact same thing. It’s funny that the catchy name was the first one we both went and looked at. https://github.com/faif/python-patterns/blob/master/patterns/creational/borg.py https://github.com/faif/python-patterns/blob/master/patterns... I was nodding along like yup, yup, yeah… … …What? The thing is, this pattern is crucial in C++. It’s how MLIR works. All objects are actually values that have shared internal state. Meaning you can copy them around as much as you want, just like Python, and you’re not copying anything except an internal pointer. It’s like a smart pointer but without any pointer interface, and I’ve wished everything in C++ worked that way. (Memory is scoped to nearest enclosing context, and contexts manage the objects below it. Very simple.) The other case that this is useful is when you have a bunch of views into some data. Consider the concept of a global variable. Globals are great. I love them. But they require discipline. If you want to spin up a bunch of threads that each have their own view of that global, you’re hosed. Except you’re not. What you can do is have a thread local variable that initializes itself to the value of the global variable. That way new threads start with the current value of the global. Why? Because suddenly you can just pretend like you’re using globals everywhere! Whenever you want to dish out some work, set the global to foo, then spin up a thread. Set the global to bar, then spin up another thread. Instead of passing that damn value through 100 smaller functions, all the code simply refers to “the current user” or “the current webpage” or “the current widget”, which is a global concept. If you ever need to modify it, you set the global and spin up another thread. Internally it’s implemented by a pattern similar to borg. The instance returns its thread local state, unless it’s nil, in which case it initializes from the global value. The nitpick is that this needs to be true for child threads that spawn threads, otherwise the whole thing doesn’t work well anymore. Racket nailed this concept with thread cells. If a thread cell is preserved, any child thread immediately inherits the value from the spawning thread. And it’s all transparent. With this borg example, it’s just literally a global variable. Or maybe it’s not and someone will come say why that’s mistaken. But I don’t think so!
  - kortex 4y ago
    It's basically a global / hoist with a more ergonomic way of accessing the state. Instead of a get_global_state() style function, you just have the Borg constructor. Personally I'm a fan of dependency injection whenever possible. But borg pattern has gotten me out of a pinch (exactly) once.
    - sillysaurusx 4y ago
      Why not just make a global though?
      - kortex 4y ago
        My understanding is that Borgs can be subclassed, you can modify the behavior, use descriptors (setters/getters), etc. You also avoid the `global` keyword. It's really not the most compelling reason. I recently found a cleaner way that actually achieves identity-equality, which I think is a much better way of going about things. Not actually using it anywhere at the moment, but I think it's neat. class Singleton(type): def __init__(cls, name, bases, dict): super(Singleton, cls).__init__(name, bases, dict) cls.instance = None def __call__(cls, *args, **kw): if cls.instance is None: cls.instance = super(Singleton, cls).__call__(*args, **kw) return cls.instance class MyClass(metaclass=Singleton): ... class MySubclass(MyClass): ... >>> MyClass() is MyClass() True >>> MySubclass() is MySubclass() True >>> MySubclass() is MyClass() False
        sillysaurusx 4y ago
        Thank you for this. I love that you have little patterns in the back of your mind that you might use, but haven't yet. Any others like that? An opportunity to ask someone about that is pretty rare.
asicsp 4y ago
See also: * "Python Design Patterns": https://python-patterns.guide/ https://python-patterns.guide/ * "Clean Architectures in Python": https://www.thedigitalcatbooks.com/pycabook-introduction https://www.thedigitalcatbooks.com/pycabook-introduction * "Architecture Patterns with Python": https://www.cosmicpython.com/book/preface.html https://www.cosmicpython.com/book/preface.html
TekMol 4y ago
I wish there was a pattern to encapsulate data and functions from a module. So you could do: import greeter greet1 = encapsulate(greeter) greet2 = encapsulate(greeter) greet1.name = 'Joe' greet2.name = 'Sue' greet1.greet() greet2.greet() And greeter.py would look like this: name = 'nobody' def greet(): print ('Hello '+name) Output: Hello Joe Hello Sue This would be so much nicer and leaner than classes.
- akx 4y ago
  You could absolutely write `encapsulate`, but you will lose out on all of the established code completion and other IDE features.
  - TekMol 4y ago
    How would encapsulate look like? Would it have a negative performance impact? I don't care about code completion at all. When I type "a", I don't want anything to happen on my screen except for an "a" to appear. I think that all the flickering stuff that the average programmer experiences on their screen burns more mental energy than it saves.
    - LittlePeter 4y ago
      I once had a working "encapsulate" but lost the code. That code was modified version of this: https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module https://docs.python.org/3/library/importlib.html#approximati... My "encapsulate" was 4 lines of code at most IIRC. As someone mentioned, autocomplete in PyCharm did not work.
    - _ZeD_ 4y ago
      I hope you are talking about code you use internally and don't share. FWIW I care about code completion, and using a library that broke my workflow is not fun.
- tekknolagi 4y ago
  You could normally do this with copy.copy or copy.deepcopy but apparently you can't pickle a module object... A quick internet search does not immediately reveal anybody trying to do this before, and I am surprised.
visarga 4y ago
This makes Python have Perl vibes.
- deleted 4y ago
  [deleted]
scaraffe 4y ago
Are there real world exercises where I could apply these patterns?
tonnydourado 4y ago
Mostly, it's Java written with Python syntax, specially the gang-of-four patterns. I recall it being even more java-like, so, progress? The stuff that is actually idiomatic python is either too convoluted or rather obvious. I know this comes across rather mean, but this is just not a good resource, for either beginners (not very idiomatic) or experienced developers (trivial patterns with trivial examples). There's maybe 2 or 3 good ideas in there, but even those I might be hard pressed to find a real usecase for.
- htlion 4y ago
  I wholefully agree, the problem with this kind of repos is that Python already makes most patterns trivial. So you either create a trivial example that does not bring value or you introduce unnecessary abstractions for the sake of them. I am always looking into these repos though because I am still interested in learning new Python tricks. One of the repos that I found interesting recently is https://github.com/dry-python/returns https://github.com/dry-python/returns
  - tonnydourado 4y ago
    I like the Maybe part, everything else feels like hitting my head against a wall, every time I try to understand it. But that's likely a me problem, I assume that for people more familiar with monads and all that jazz, it's easier to understand.
    - d0mine 4y ago
      All these lambdas seem excessive. "Maybe"-like pattern in Python can look like (None is falsy and custom objects are true): # note: `None and f()` returns None without calling f balance = user and user.get_balance() credit = balance and balance.credit_amount() discount_program = choose_discount(credit) if credit and credit > 0 else None https://stackoverflow.com/questions/8507200/maybe-kind-of-monad-in-python https://stackoverflow.com/questions/8507200/maybe-kind-of-mo...
      - 4y ago
adammarples 4y ago
please, don't do any of this in your python, thanks
- deleted 4y ago
  [deleted]
- mariorojas 4y ago
  I really feel overwhelmed. I like coding in Python because it makes me feel free from Java stuff, but those patterns have getting me worried.
  - bussierem 4y ago
    A decade of python here. I have found that Python really can show you what other language is someone's primary language. You can write Python to look VERY Java-like, or very Ruby-like, or many others. Idiomatic Python does not look like this generally. Another commenter hit it well that this is just badly-hidden Java code written with Python. Python is what you make it to be; don't let this kind of stuff from random people on the internet scare you into thinking you have to use GoF and Java-like patterns and classes everywhere. You CAN, but only if those are actually _useful_ to you.
    - mariorojas 4y ago
      I agree! You know? When I feel burned out of Java, I usually write a quite simple script in Python and then, normally I get energy to complete my tasks. I don't imagine myself struggling against the same design patterns in Java when coding in Python. Thanks for your advice.
- kortex 4y ago
  Dependency injection and lazy_property are excellent, I use those all the time. I've used visitor on occasion. The rest? Mostly meh, some are straight up bad.
jjice 4y ago
I recently began reading GOF (a bit into the behavioral patterns at the moment). A lot of those patterns were ideas I had been familiar with, but not through a formal specification, likely due to seeing so much code inspired by the book. I just find it so fascinating that these patterns were written about in 1994 (?) and they were pretty well designed. I will say, the book is heavily focused on OOP, which has fallen out of favor a bit with newer languages (but obviously very alive), and it's almost like a time piece. Feels like OOP was seen as this answer to so many business problems and organization of code. Just interesting to imagine OOP at a time when it was new and exciting, since I grew up at a time when Java was ubiquitous and everyone was at least a little OO.
Kyragem 4y ago
GOF is quite inaccessible/boring to students, but the headfirst design patterns book has been a blast in my design patterns course.
mumblemumble 4y ago
For an alternative perspective, allow me to present some classic talks from Raymond Hettinger. First, some generic tips on how to write Python code that's pleasant to work with: https://youtu.be/OSGv2VnC0go https://youtu.be/OSGv2VnC0go And then this one digs more specifically into how not to write Python like it's Java, starting at 12:40 (with talking about why leading up to that): https://youtu.be/wf-BqAjZb8M https://youtu.be/wf-BqAjZb8M
timcavel 4y ago
mariorojas 4y ago
I'm beginner in Python/Django and to be honest I feel confused since those patterns add complexity to the code.
lordgrenville 4y ago
I've had Python code reviewed in the past by senior devs from more of a Java/C# background, who encouraged me to read the GoF and rewrite classes using the abc (abstract base class) library, staticmethod/classmethod decorators, etc. I never quite understood the value of it in Python. But the fact that those are included in the stdlib makes me think that they must have some clear use cases.
- digisign 4y ago
  Those can be useful, it all depends.
havan_agrawal 4y ago
Picking very specifically on the builder pattern: the whole point is to solve one or more of the following problems: 1. creating complex objects and having them valid at creation 2. many optional arguments for an object's construction 3. similar types for object construction For e.g. in Java you might see something like: Point p = new Point(10, 0, 5); // (x=10, z=5) Point q = new Point(10, 2); // (x=10, y=2) which may be problematic because 1. y needs to be specified explicitly to 0, because you can't expose an overloaded constructor that takes x and z (that would clash with the constructor that takes x and y 2. It _may_ not be evident that new Point(10, 0, 5); passes x, y and z in that order 3. It ties the constructor to the implementation. You can never expose a Point(double radius, double theta) constructor because that would clash with the Point(double x, double y) constructor. The builder would solve all of these problems: Point p = Point.newBuilder() .setX(10) .setZ(5) .build(); Python however doesn't necessarily need this pattern because idiomatically solves all three of these problems by providing 1. named arguments, and 2. default arguments so you could have: p = Point(x=10, z=5) q = Point(10, 2) t = Point(radius=5, theta=45) I feel the post completely glosses over this by providing a single super-class with the appropriate constructor, and overriding the methods in the sub-classes. What if I just have a single class that I want a builder for? I don't think it showcases the builder pattern at all, just method overloading in classes.

262 points by gilad 4y ago

I'm confused by the Borg pattern. It seems like a singleton that tricks the user into believing there are multiple instances (very bad imo). Is there any advantage over singleton? Maybe some kind of `__dict__` behavior I'm not aware of?

sillysaurusx 4y ago
I showed up 30 seconds after you posted your comment to say the exact same thing. It’s funny that the catchy name was the first one we both went and looked at. https://github.com/faif/python-patterns/blob/master/patterns/creational/borg.py https://github.com/faif/python-patterns/blob/master/patterns... I was nodding along like yup, yup, yeah… … …What? The thing is, this pattern is crucial in C++. It’s how MLIR works. All objects are actually values that have shared internal state. Meaning you can copy them around as much as you want, just like Python, and you’re not copying anything except an internal pointer. It’s like a smart pointer but without any pointer interface, and I’ve wished everything in C++ worked that way. (Memory is scoped to nearest enclosing context, and contexts manage the objects below it. Very simple.) The other case that this is useful is when you have a bunch of views into some data. Consider the concept of a global variable. Globals are great. I love them. But they require discipline. If you want to spin up a bunch of threads that each have their own view of that global, you’re hosed. Except you’re not. What you can do is have a thread local variable that initializes itself to the value of the global variable. That way new threads start with the current value of the global. Why? Because suddenly you can just pretend like you’re using globals everywhere! Whenever you want to dish out some work, set the global to foo, then spin up a thread. Set the global to bar, then spin up another thread. Instead of passing that damn value through 100 smaller functions, all the code simply refers to “the current user” or “the current webpage” or “the current widget”, which is a global concept. If you ever need to modify it, you set the global and spin up another thread. Internally it’s implemented by a pattern similar to borg. The instance returns its thread local state, unless it’s nil, in which case it initializes from the global value. The nitpick is that this needs to be true for child threads that spawn threads, otherwise the whole thing doesn’t work well anymore. Racket nailed this concept with thread cells. If a thread cell is preserved, any child thread immediately inherits the value from the spawning thread. And it’s all transparent. With this borg example, it’s just literally a global variable. Or maybe it’s not and someone will come say why that’s mistaken. But I don’t think so!
- kortex 4y ago
  It's basically a global / hoist with a more ergonomic way of accessing the state. Instead of a get_global_state() style function, you just have the Borg constructor. Personally I'm a fan of dependency injection whenever possible. But borg pattern has gotten me out of a pinch (exactly) once.
  - sillysaurusx 4y ago
    Why not just make a global though?
    - kortex 4y ago
      My understanding is that Borgs can be subclassed, you can modify the behavior, use descriptors (setters/getters), etc. You also avoid the `global` keyword. It's really not the most compelling reason. I recently found a cleaner way that actually achieves identity-equality, which I think is a much better way of going about things. Not actually using it anywhere at the moment, but I think it's neat. class Singleton(type): def __init__(cls, name, bases, dict): super(Singleton, cls).__init__(name, bases, dict) cls.instance = None def __call__(cls, *args, **kw): if cls.instance is None: cls.instance = super(Singleton, cls).__call__(*args, **kw) return cls.instance class MyClass(metaclass=Singleton): ... class MySubclass(MyClass): ... >>> MyClass() is MyClass() True >>> MySubclass() is MySubclass() True >>> MySubclass() is MyClass() False
      - sillysaurusx 4y ago
        Thank you for this. I love that you have little patterns in the back of your mind that you might use, but haven't yet. Any others like that? An opportunity to ask someone about that is pretty rare.

asicsp 4y ago

See also: * "Python Design Patterns": https://python-patterns.guide/ https://python-patterns.guide/ * "Clean Architectures in Python": https://www.thedigitalcatbooks.com/pycabook-introduction https://www.thedigitalcatbooks.com/pycabook-introduction * "Architecture Patterns with Python": https://www.cosmicpython.com/book/preface.html https://www.cosmicpython.com/book/preface.html

TekMol 4y ago

I wish there was a pattern to encapsulate data and functions from a module. So you could do: import greeter greet1 = encapsulate(greeter) greet2 = encapsulate(greeter) greet1.name = 'Joe' greet2.name = 'Sue' greet1.greet() greet2.greet() And greeter.py would look like this: name = 'nobody' def greet(): print ('Hello '+name) Output: Hello Joe Hello Sue This would be so much nicer and leaner than classes.

akx 4y ago
You could absolutely write `encapsulate`, but you will lose out on all of the established code completion and other IDE features.
- TekMol 4y ago
  How would encapsulate look like? Would it have a negative performance impact? I don't care about code completion at all. When I type "a", I don't want anything to happen on my screen except for an "a" to appear. I think that all the flickering stuff that the average programmer experiences on their screen burns more mental energy than it saves.
  - LittlePeter 4y ago
    I once had a working "encapsulate" but lost the code. That code was modified version of this: https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module https://docs.python.org/3/library/importlib.html#approximati... My "encapsulate" was 4 lines of code at most IIRC. As someone mentioned, autocomplete in PyCharm did not work.
  - _ZeD_ 4y ago
    I hope you are talking about code you use internally and don't share. FWIW I care about code completion, and using a library that broke my workflow is not fun.
tekknolagi 4y ago
You could normally do this with copy.copy or copy.deepcopy but apparently you can't pickle a module object... A quick internet search does not immediately reveal anybody trying to do this before, and I am surprised.

visarga 4y ago

This makes Python have Perl vibes.

deleted 4y ago
[deleted]

scaraffe 4y ago

Are there real world exercises where I could apply these patterns?

tonnydourado 4y ago

Mostly, it's Java written with Python syntax, specially the gang-of-four patterns. I recall it being even more java-like, so, progress? The stuff that is actually idiomatic python is either too convoluted or rather obvious. I know this comes across rather mean, but this is just not a good resource, for either beginners (not very idiomatic) or experienced developers (trivial patterns with trivial examples). There's maybe 2 or 3 good ideas in there, but even those I might be hard pressed to find a real usecase for.

htlion 4y ago
I wholefully agree, the problem with this kind of repos is that Python already makes most patterns trivial. So you either create a trivial example that does not bring value or you introduce unnecessary abstractions for the sake of them. I am always looking into these repos though because I am still interested in learning new Python tricks. One of the repos that I found interesting recently is https://github.com/dry-python/returns https://github.com/dry-python/returns
- tonnydourado 4y ago
  I like the Maybe part, everything else feels like hitting my head against a wall, every time I try to understand it. But that's likely a me problem, I assume that for people more familiar with monads and all that jazz, it's easier to understand.
  - d0mine 4y ago
    All these lambdas seem excessive. "Maybe"-like pattern in Python can look like (None is falsy and custom objects are true): # note: `None and f()` returns None without calling f balance = user and user.get_balance() credit = balance and balance.credit_amount() discount_program = choose_discount(credit) if credit and credit > 0 else None https://stackoverflow.com/questions/8507200/maybe-kind-of-monad-in-python https://stackoverflow.com/questions/8507200/maybe-kind-of-mo...
    - 4y ago

adammarples 4y ago

please, don't do any of this in your python, thanks

deleted 4y ago
[deleted]
mariorojas 4y ago
I really feel overwhelmed. I like coding in Python because it makes me feel free from Java stuff, but those patterns have getting me worried.
- bussierem 4y ago
  A decade of python here. I have found that Python really can show you what other language is someone's primary language. You can write Python to look VERY Java-like, or very Ruby-like, or many others. Idiomatic Python does not look like this generally. Another commenter hit it well that this is just badly-hidden Java code written with Python. Python is what you make it to be; don't let this kind of stuff from random people on the internet scare you into thinking you have to use GoF and Java-like patterns and classes everywhere. You CAN, but only if those are actually _useful_ to you.
  - mariorojas 4y ago
    I agree! You know? When I feel burned out of Java, I usually write a quite simple script in Python and then, normally I get energy to complete my tasks. I don't imagine myself struggling against the same design patterns in Java when coding in Python. Thanks for your advice.
kortex 4y ago
Dependency injection and lazy_property are excellent, I use those all the time. I've used visitor on occasion. The rest? Mostly meh, some are straight up bad.

jjice 4y ago

I recently began reading GOF (a bit into the behavioral patterns at the moment). A lot of those patterns were ideas I had been familiar with, but not through a formal specification, likely due to seeing so much code inspired by the book. I just find it so fascinating that these patterns were written about in 1994 (?) and they were pretty well designed. I will say, the book is heavily focused on OOP, which has fallen out of favor a bit with newer languages (but obviously very alive), and it's almost like a time piece. Feels like OOP was seen as this answer to so many business problems and organization of code. Just interesting to imagine OOP at a time when it was new and exciting, since I grew up at a time when Java was ubiquitous and everyone was at least a little OO.

Kyragem 4y ago

GOF is quite inaccessible/boring to students, but the headfirst design patterns book has been a blast in my design patterns course.

mumblemumble 4y ago

For an alternative perspective, allow me to present some classic talks from Raymond Hettinger. First, some generic tips on how to write Python code that's pleasant to work with: https://youtu.be/OSGv2VnC0go https://youtu.be/OSGv2VnC0go And then this one digs more specifically into how not to write Python like it's Java, starting at 12:40 (with talking about why leading up to that): https://youtu.be/wf-BqAjZb8M https://youtu.be/wf-BqAjZb8M

timcavel 4y ago

mariorojas 4y ago

I'm beginner in Python/Django and to be honest I feel confused since those patterns add complexity to the code.

lordgrenville 4y ago

I've had Python code reviewed in the past by senior devs from more of a Java/C# background, who encouraged me to read the GoF and rewrite classes using the abc (abstract base class) library, staticmethod/classmethod decorators, etc. I never quite understood the value of it in Python. But the fact that those are included in the stdlib makes me think that they must have some clear use cases.

digisign 4y ago
Those can be useful, it all depends.

havan_agrawal 4y ago

Picking very specifically on the builder pattern: the whole point is to solve one or more of the following problems: 1. creating complex objects and having them valid at creation 2. many optional arguments for an object's construction 3. similar types for object construction For e.g. in Java you might see something like: Point p = new Point(10, 0, 5); // (x=10, z=5) Point q = new Point(10, 2); // (x=10, y=2) which may be problematic because 1. y needs to be specified explicitly to 0, because you can't expose an overloaded constructor that takes x and z (that would clash with the constructor that takes x and y 2. It _may_ not be evident that new Point(10, 0, 5); passes x, y and z in that order 3. It ties the constructor to the implementation. You can never expose a Point(double radius, double theta) constructor because that would clash with the Point(double x, double y) constructor. The builder would solve all of these problems: Point p = Point.newBuilder() .setX(10) .setZ(5) .build(); Python however doesn't necessarily need this pattern because idiomatically solves all three of these problems by providing 1. named arguments, and 2. default arguments so you could have: p = Point(x=10, z=5) q = Point(10, 2) t = Point(radius=5, theta=45) I feel the post completely glosses over this by providing a single super-class with the appropriate constructor, and overriding the methods in the sub-classes. What if I just have a single class that I want a builder for? I don't think it showcases the builder pattern at all, just method overloading in classes.