6 ms·

Better Python APIs

160 points by ozkatz 14y ago

JulianWasTaken 14y ago
It's true that double underscore methods are sometimes underused and that using them can often make for a better experience, but it's also easy to be overly clever, or to "misuse" them by using them to add semantics to an object that is already a bit mis-designed. The first example in the post (using __repr__) is probably the least controversial. There are places in the official docs that explicitly recommend defining __repr__ for every object you create. This is mostly something that I do do (though there are exceptions of course), but one thing you do want to be careful with is to make sure that what appears in your repr is relevant, terse, and helpful. That means that I will and do sacrifice a repr that is eval'able (which I find to be useless) for a repr that is short and sweet. So for the example in the article, it may or may not make sense to put `self.objects` in there. For a "Container" class, it probably does, but don't extend that to "let's put all our data in the repr for everything". __iter__ is nice, and is one of the most oddly underused methods, but rather than using it there, I think the better thing is to not have the object encapsulate a search, but to have the object encapsulate a YoutubeAPIClient or whatever, and have `search` be a method that returns an iterable. This is also kinda touchy-feely though since I guess there might be examples of __iter__ where I'd feel more comfortable with something like this (a results object). The next one with __sub__ I don't like too much, because I don't like encouraging slow operations with syntax sugar (besides the typechecking). Anyways, I guess I'm nitpicking on examples, mostly. Double underscore methods are generally underused I've found, and if you have doubts on whether you're overreaching, you can always add a method with the same behavior you're using them for in case anyone doesn't like how it reads.
lysol 14y ago
I always consider these nice to have, because often-times they fall on the implicit side of things rather than explicit. But if you explicitly state "X is also a generator" in your documentation, then by all means implement these methods.
pjscott 14y ago
Nice post overall, but I take issue with a couple of the examples. First, the YoutubeSearch class has no reason to be a class. It wants to be a function: def youtube_search(term): url = 'https://gdata.youtube.com/feeds/api/videos' params = {'q': term, 'alt': 'json', 'orderby': 'relevance', 'v': '2'} r = requests.get(url, params=params) r.raise_for_status() # 4xx and 5xx replies raise exception for video in r.json.get('feed').get('entry'): yield { 'title': video.get('title').get('$t'), 'url': video.get('link')[0].get('href'), } The ExpressiveList's __sub__() method, as written in the article, requires O(n*k) time to remove k things from a list of length n. By using a set of things to remove, we can get that down to O(n+k): class ExpressiveList(list): def __sub__(self, other): if isinstance(other, list): other_set = set(other) return ExpressiveList(x for x in self if x not in other_set) else: return ExpressiveList(x for x in self if x != other)
- jspthrowaway2 14y ago
  The primary reason to make YoutubeSearch a class is, in the future, implementing searching other video sites and taking advantage of inheritance in the implementation. That isn't demonstrated here, but the plumbing of "doing a request", "providing an iterator" and such could be promoted to the base class and subclasses are only responsible for knowing their domain.
  - wahnfrieden 14y ago
    YAGNI and besides you can still provide two layers of access, one being convenience functions on top of the class implementation.
  - icebraining 14y ago
    When you actually want to implement those other sites, you can replace the function by a class with a __call__ method.
    - hcarvalhoalves 14y ago
yousuffauzan 14y ago
Good Post. I totally agree with REPL friendly code. While developing in python, most of my time is spent in the shell, introspecting the code etc. +1
ninetax 14y ago
I disagree about the operator overloading. It's kind of like a hidden function that could have ambiguous connotations. It could confuse people because of the association with the traditional operations. For simple things like list addition it might be intuitive what the result will be, but if I had a a Vector class and then you were reading some code and saw two vectors being multiplied like this a * b well is that a dot product or a cross product? Or maybe a is a scalar and it's scalar multiplication? If it's one of those what do the other operations get for symbols? isn't it easier to do a.cross(b)? If you keep it simple a little syntactic sugar is probably find. Nobody wants to be writing a.add(b.subtract(c.mult(d))) for a + b - c *d . Anyone have other thoughts on this?
- nostrademons 14y ago
  A lot of this depends upon your attitude toward abstraction. For example, there's a Python idiom for flattening a list that is just sum(list, []), taking advantage of the overloaded + for lists. (This is considered bad style for Python; use itertools.chain instead). However, if you're a Haskeller (or a mathematician), this makes perfect sense. Any type with an associative binary + and an identity element is a monoid; any additive monad (MonadPlus in Haskell) is a monoid; List is a monad; therefore it makes perfect sense that operations like "sum" that are defined on monoids should work on lists. This is just the age old abstraction vs. traceability debate that's been going on in programming languages for 40 years. Die-hard C programmers look at Java code that says a.dot(b) and say "How can I possibly know what this does? It depends upon the run-time type of a. Much safer just to do vector_add(a, b)." Then Java programmers look at Python code that says a.dot(b) and say "How can I possibly know what this does? I don't even know what interface a and b have to conform to, they could be anything." Then Python programmers look at Haskell code that says a * b and say "How can I possibly know what this does? You say it depends upon the type of a and b, and it could be either a dot or a cross product? And what's this -XOverlappingInstances flag I need to know about?" Somewhere, there's a happy middle ground between impenetrable code vs. rewriting the same thing over and over again for different types. Personally, I think you won't find it with a blanket rule like "never use operator overloading", and it depends a lot on which abstractions come up in your problem domain.
- 14y ago
krosaen 14y ago
I suggest the author and readers watch the excellent pycon talk, "Stop Writing Classes": http://www.youtube.com/watch?v=o9pEzgHorH0 http://www.youtube.com/watch?v=o9pEzgHorH0 That said, some nice factoids in the post about making your objects readable in the repl etc.

160 points by ozkatz 14y ago

It's true that double underscore methods are sometimes underused and that using them can often make for a better experience, but it's also easy to be overly clever, or to "misuse" them by using them to add semantics to an object that is already a bit mis-designed. The first example in the post (using __repr__) is probably the least controversial. There are places in the official docs that explicitly recommend defining __repr__ for every object you create. This is mostly something that I do do (though there are exceptions of course), but one thing you do want to be careful with is to make sure that what appears in your repr is relevant, terse, and helpful. That means that I will and do sacrifice a repr that is eval'able (which I find to be useless) for a repr that is short and sweet. So for the example in the article, it may or may not make sense to put `self.objects` in there. For a "Container" class, it probably does, but don't extend that to "let's put all our data in the repr for everything". __iter__ is nice, and is one of the most oddly underused methods, but rather than using it there, I think the better thing is to not have the object encapsulate a search, but to have the object encapsulate a YoutubeAPIClient or whatever, and have `search` be a method that returns an iterable. This is also kinda touchy-feely though since I guess there might be examples of __iter__ where I'd feel more comfortable with something like this (a results object). The next one with __sub__ I don't like too much, because I don't like encouraging slow operations with syntax sugar (besides the typechecking). Anyways, I guess I'm nitpicking on examples, mostly. Double underscore methods are generally underused I've found, and if you have doubts on whether you're overreaching, you can always add a method with the same behavior you're using them for in case anyone doesn't like how it reads.

lysol 14y ago

I always consider these nice to have, because often-times they fall on the implicit side of things rather than explicit. But if you explicitly state "X is also a generator" in your documentation, then by all means implement these methods.

pjscott 14y ago

Nice post overall, but I take issue with a couple of the examples. First, the YoutubeSearch class has no reason to be a class. It wants to be a function: def youtube_search(term): url = 'https://gdata.youtube.com/feeds/api/videos' params = {'q': term, 'alt': 'json', 'orderby': 'relevance', 'v': '2'} r = requests.get(url, params=params) r.raise_for_status() # 4xx and 5xx replies raise exception for video in r.json.get('feed').get('entry'): yield { 'title': video.get('title').get('$t'), 'url': video.get('link')[0].get('href'), } The ExpressiveList's __sub__() method, as written in the article, requires O(n*k) time to remove k things from a list of length n. By using a set of things to remove, we can get that down to O(n+k): class ExpressiveList(list): def __sub__(self, other): if isinstance(other, list): other_set = set(other) return ExpressiveList(x for x in self if x not in other_set) else: return ExpressiveList(x for x in self if x != other)

jspthrowaway2 14y ago
The primary reason to make YoutubeSearch a class is, in the future, implementing searching other video sites and taking advantage of inheritance in the implementation. That isn't demonstrated here, but the plumbing of "doing a request", "providing an iterator" and such could be promoted to the base class and subclasses are only responsible for knowing their domain.
- wahnfrieden 14y ago
  YAGNI and besides you can still provide two layers of access, one being convenience functions on top of the class implementation.
- icebraining 14y ago
  When you actually want to implement those other sites, you can replace the function by a class with a __call__ method.
  - hcarvalhoalves 14y ago

yousuffauzan 14y ago

Good Post. I totally agree with REPL friendly code. While developing in python, most of my time is spent in the shell, introspecting the code etc. +1

ninetax 14y ago

I disagree about the operator overloading. It's kind of like a hidden function that could have ambiguous connotations. It could confuse people because of the association with the traditional operations. For simple things like list addition it might be intuitive what the result will be, but if I had a a Vector class and then you were reading some code and saw two vectors being multiplied like this a * b well is that a dot product or a cross product? Or maybe a is a scalar and it's scalar multiplication? If it's one of those what do the other operations get for symbols? isn't it easier to do a.cross(b)? If you keep it simple a little syntactic sugar is probably find. Nobody wants to be writing a.add(b.subtract(c.mult(d))) for a + b - c *d . Anyone have other thoughts on this?

nostrademons 14y ago
A lot of this depends upon your attitude toward abstraction. For example, there's a Python idiom for flattening a list that is just sum(list, []), taking advantage of the overloaded + for lists. (This is considered bad style for Python; use itertools.chain instead). However, if you're a Haskeller (or a mathematician), this makes perfect sense. Any type with an associative binary + and an identity element is a monoid; any additive monad (MonadPlus in Haskell) is a monoid; List is a monad; therefore it makes perfect sense that operations like "sum" that are defined on monoids should work on lists. This is just the age old abstraction vs. traceability debate that's been going on in programming languages for 40 years. Die-hard C programmers look at Java code that says a.dot(b) and say "How can I possibly know what this does? It depends upon the run-time type of a. Much safer just to do vector_add(a, b)." Then Java programmers look at Python code that says a.dot(b) and say "How can I possibly know what this does? I don't even know what interface a and b have to conform to, they could be anything." Then Python programmers look at Haskell code that says a * b and say "How can I possibly know what this does? You say it depends upon the type of a and b, and it could be either a dot or a cross product? And what's this -XOverlappingInstances flag I need to know about?" Somewhere, there's a happy middle ground between impenetrable code vs. rewriting the same thing over and over again for different types. Personally, I think you won't find it with a blanket rule like "never use operator overloading", and it depends a lot on which abstractions come up in your problem domain.
14y ago

krosaen 14y ago

I suggest the author and readers watch the excellent pycon talk, "Stop Writing Classes": http://www.youtube.com/watch?v=o9pEzgHorH0 http://www.youtube.com/watch?v=o9pEzgHorH0 That said, some nice factoids in the post about making your objects readable in the repl etc.