4 ms·

Python Black Magic

19 points by jbail 16y ago

Estragon 16y ago
I learned these kinds of tricks when I started using python 12 years ago, but I have basically never had a need for them. Perhaps my needs are simple, though.
- jjames 16y ago
  Same here. Like many things in python, the unconventional hack is made just inconvenient enough to persuade most people to rethink their strategy.
- jbail 16y ago
  Agreed. I can't think of a time when you'd need to use it either (and I wrote the post). But, coming from working in Java for the past few years, it seems pretty cool that you can do it.
  - tsuraan 16y ago
    You also can do it in java: http://www.fordevs.com/2008/11/reflection-is-mechanism-by-which-java.html http://www.fordevs.com/2008/11/reflection-is-mechanism-by-wh... . As a python programmer who never uses "private" variables (I use a single underscore to indicate fields that shouldn't be used), I'm a bit torn as to which private variable abuse method is uglier :)
- sophacles 16y ago
  They are nice to know when debugging -- looking at dir(obj) can lead to things like "where the heck is _foo__attr defined?" if you don't know the __x -> _classname__x mangling.
- mccutchen 16y ago
  I learned this trick when I first picked up Python, too. I came from Java and, naturally, started writing Python code in the style I had learned to write Java: Heavily object-oriented, with getters and setters everywhere. Eventually, though, I stopped trying to program in Java in Python. Once you relax and stop trying to hide your precious instance variables from all of the potentially pernicious or incompetent programmers who might be using your code (usually, in my case, just me), you no longer need tricks like this. And you realize that you never really needed them to begin with.
  - jbail 16y ago
    I'm not advocating using getters/setters in Python. Just pointing out that you can access "private" members. I don't typically use getters and setters in JavaScript either. Java, on the other hand, kind of forces you into using getters and setters because the POJO is so ingrained into the ecosystem. That said, if you write Android apps in Java, you should avoid getters and setters because they aren't really necessary and they create extra overhead.
    - nivertech 16y ago
      One of the first textbooks on PL/1 also stated: "you should avoid using subroutines because they they create extra overhead" ;) 40 years later and we still there ...
  - koenigdavidmj 16y ago
    And you can turn your member access into called code when you need it: class Foo(object): @property def bar(self): print "Running getter" return self._bar @bar.setter def bar(self, value): print "Running setter" self._bar = value
lukeqsee 16y ago
Forgive my incompetence, but I have never gotten the point of private methods/variables. Could someone please explain why they exist? Edit: I know it's slightly off-topic.
- ianp 16y ago
  Private members are generally just to protect your from yourself. If you have an integer that should /always/ be 1-10, it's easier to use a private variable and a setter that ensures the variable is always valid, than to hope all your code follows the rules. (Then also if the rule changes, e.g. to 5-100, you just need to change one function). There are some other uses and examples, but this is the simplest one i can think of off the top of my head.
- sam191 16y ago
  I may not be the most qualified person to answer this, but I always thought private methods and variables were for internal use inside of a class while the public methods were what one would actually call with their objects. For example your object had a method foo() which would be better defined as smaller methods which are combined together. Those smaller functions serve no other purpose and therefore do not need to be called by the object directly.
- sophacles 16y ago
  One of the points not discussed in the other replies yet, is that public methods and variables are the interface, the well defined, relatively static way of using a class. The private members are internal implementation details, and may change frequently -- not to mention completely change in style, functionality and so on. The idea then, is that larger teams can be divided into sub-teams, and work in parallel because the interfaces between components don't change, even if the internals change drastically. Essentially it allows components to be somewhat independent. How this works in practice tho...
- epo 16y ago
  Private methods/variables etc are those which are not publicly available. Typically internal state for the object. Ideally, or at least for purity, all variables should be private and only accessible via get/set methods, thus encapsulation is enforced.
meese_ 16y ago
This is not exactly "black magic", it's fairly well-known and explained directly in the official documentation: http://docs.python.org/tutorial/classes.html#private-variables http://docs.python.org/tutorial/classes.html#private-variabl....
- jbail 16y ago
  I think developers coming to Python from other languages (like I am from a heavy Java background) find the auto-creation of "shadow variables" pretty interesting. To someone used to Java, being able to access a private member via its shadow variable is a lot like black magic. Discovering this for the first time in Python seemed pretty cool to me. I really hope this feeling of magic in working with Python never leaves me. I want to remain amazed.
  - piramida 16y ago
    Coming from C++, this is very similar to what all C++ compilers do internally (i.e. name mangling). Python just opens a possibility to access that "mangled" names directly, which should never be necessary unless you are doing some metaprogramming and need full access to class internals.
Jach 16y ago
You can access private data in C++ too quite trivially, especially since there's no compile-time encapsulation... Guido van Rossum on the lack of private members: "We're all consenting adults here." (And a discussion: http://mail.python.org/pipermail/tutor/2003-October/025932.html http://mail.python.org/pipermail/tutor/2003-October/025932.h... )
didip 16y ago
While we are on the black magic thread, are there other meta-programming tricks that people find useful/clever? I'll share one: import inspect. It's handy to use during debugging.
zefhous 16y ago
Also, Ruby lets you call private/protected methods with Object#send.

19 points by jbail 16y ago

I learned these kinds of tricks when I started using python 12 years ago, but I have basically never had a need for them. Perhaps my needs are simple, though.

jjames 16y ago
Same here. Like many things in python, the unconventional hack is made just inconvenient enough to persuade most people to rethink their strategy.
jbail 16y ago
Agreed. I can't think of a time when you'd need to use it either (and I wrote the post). But, coming from working in Java for the past few years, it seems pretty cool that you can do it.
- tsuraan 16y ago
  You also can do it in java: http://www.fordevs.com/2008/11/reflection-is-mechanism-by-which-java.html http://www.fordevs.com/2008/11/reflection-is-mechanism-by-wh... . As a python programmer who never uses "private" variables (I use a single underscore to indicate fields that shouldn't be used), I'm a bit torn as to which private variable abuse method is uglier :)
sophacles 16y ago
They are nice to know when debugging -- looking at dir(obj) can lead to things like "where the heck is _foo__attr defined?" if you don't know the __x -> _classname__x mangling.
mccutchen 16y ago
I learned this trick when I first picked up Python, too. I came from Java and, naturally, started writing Python code in the style I had learned to write Java: Heavily object-oriented, with getters and setters everywhere. Eventually, though, I stopped trying to program in Java in Python. Once you relax and stop trying to hide your precious instance variables from all of the potentially pernicious or incompetent programmers who might be using your code (usually, in my case, just me), you no longer need tricks like this. And you realize that you never really needed them to begin with.
- jbail 16y ago
  I'm not advocating using getters/setters in Python. Just pointing out that you can access "private" members. I don't typically use getters and setters in JavaScript either. Java, on the other hand, kind of forces you into using getters and setters because the POJO is so ingrained into the ecosystem. That said, if you write Android apps in Java, you should avoid getters and setters because they aren't really necessary and they create extra overhead.
  - nivertech 16y ago
    One of the first textbooks on PL/1 also stated: "you should avoid using subroutines because they they create extra overhead" ;) 40 years later and we still there ...
- koenigdavidmj 16y ago
  And you can turn your member access into called code when you need it: class Foo(object): @property def bar(self): print "Running getter" return self._bar @bar.setter def bar(self, value): print "Running setter" self._bar = value

lukeqsee 16y ago

Forgive my incompetence, but I have never gotten the point of private methods/variables. Could someone please explain why they exist? Edit: I know it's slightly off-topic.

ianp 16y ago
Private members are generally just to protect your from yourself. If you have an integer that should /always/ be 1-10, it's easier to use a private variable and a setter that ensures the variable is always valid, than to hope all your code follows the rules. (Then also if the rule changes, e.g. to 5-100, you just need to change one function). There are some other uses and examples, but this is the simplest one i can think of off the top of my head.
sam191 16y ago
I may not be the most qualified person to answer this, but I always thought private methods and variables were for internal use inside of a class while the public methods were what one would actually call with their objects. For example your object had a method foo() which would be better defined as smaller methods which are combined together. Those smaller functions serve no other purpose and therefore do not need to be called by the object directly.
sophacles 16y ago
One of the points not discussed in the other replies yet, is that public methods and variables are the interface, the well defined, relatively static way of using a class. The private members are internal implementation details, and may change frequently -- not to mention completely change in style, functionality and so on. The idea then, is that larger teams can be divided into sub-teams, and work in parallel because the interfaces between components don't change, even if the internals change drastically. Essentially it allows components to be somewhat independent. How this works in practice tho...
epo 16y ago
Private methods/variables etc are those which are not publicly available. Typically internal state for the object. Ideally, or at least for purity, all variables should be private and only accessible via get/set methods, thus encapsulation is enforced.

meese_ 16y ago

This is not exactly "black magic", it's fairly well-known and explained directly in the official documentation: http://docs.python.org/tutorial/classes.html#private-variables http://docs.python.org/tutorial/classes.html#private-variabl....

jbail 16y ago
I think developers coming to Python from other languages (like I am from a heavy Java background) find the auto-creation of "shadow variables" pretty interesting. To someone used to Java, being able to access a private member via its shadow variable is a lot like black magic. Discovering this for the first time in Python seemed pretty cool to me. I really hope this feeling of magic in working with Python never leaves me. I want to remain amazed.
- piramida 16y ago
  Coming from C++, this is very similar to what all C++ compilers do internally (i.e. name mangling). Python just opens a possibility to access that "mangled" names directly, which should never be necessary unless you are doing some metaprogramming and need full access to class internals.

Jach 16y ago

You can access private data in C++ too quite trivially, especially since there's no compile-time encapsulation... Guido van Rossum on the lack of private members: "We're all consenting adults here." (And a discussion: http://mail.python.org/pipermail/tutor/2003-October/025932.html http://mail.python.org/pipermail/tutor/2003-October/025932.h... )

didip 16y ago

While we are on the black magic thread, are there other meta-programming tricks that people find useful/clever? I'll share one: import inspect. It's handy to use during debugging.

zefhous 16y ago

Also, Ruby lets you call private/protected methods with Object#send.