6 ms·

Specific ways to write better Python (2017)

154 points by guptarohit 6y ago

dd82 6y ago
The author has a second edition out with 90 examples at https://effectivepython.com/ https://effectivepython.com/, and the corresponding github repo at https://github.com/bslatkin/effectivepython https://github.com/bslatkin/effectivepython Its been updated to be exclusive to 3.x, including 3.8 samples.
- Alex3917 6y ago
  Great book that every Python developer should own! Even for the recommendations you don't buy into, you'll still benefit just from reading a lucid argument in their favor.
- guiambros 6y ago
  The author also has a really nice Safari training video course[1]. Always worth reminding folks that Safari is included on a $99/yr ACM subscription, which is a great value. [1] https://learning.oreilly.com/videos/effective-python/9780134175249 https://learning.oreilly.com/videos/effective-python/9780134...
softwaredoug 6y ago
On “return generators instead of lists” I think it’s also important when you do this to return use a context manager to manage the generators lifetime. Otherwise you might be surprised when the underlying file or source is closed later in your code. I blogged about this: https://opensourceconnections.com/blog/2020/06/10/python-generators-and-context-managers/ https://opensourceconnections.com/blog/2020/06/10/python-gen...
- xapata 6y ago
  Maybe I misunderstood, but if you're suggesting the generator function open/close the resource, then I disagree. The caller has the responsibility of safely opening and closing. The generator function should not care what type of iterable it receives
  - cheez 6y ago
    I agree, this unnecessarily complicates library-type code.
  - softwaredoug 6y ago
    Then the caller must know they are receiving a generator dependent on the files lifetime. Pretty brittle IMO. Receiving a list wouldn’t depend on the files lifetime. This causes client code to be pretty error prone as you can very easily think you’re saving off a list built from a file, but in reality you hold a generator. Client code closes the file, the generator is used much later, and blammo, crash! NOT doing a context manager forces clients to work harder at guessing lifetimes. So you can’t just replace lists with generators. This break the uniform access principal. Client code IMO shouldn’t need to think about whether they got a generator or an interable container.
    - xapata 6y ago
      > Then the caller must know they are receiving a generator dependent on the files lifetime How would they not? The caller provides the file as an argument to the generator function. Further, the caller decides when to close the file, presumably via leaving a context and after the generator is consumed.
      - Groxx 6y ago
        it's not too hard to write code like: with open(...) as f: lines = process(f) # do stuff with lines ^ if `process` returns a generator, the file will be closed without having done anything.
        xapata 6y ago
        Put `do stuff with lines` inside the context. Where it belongs.
        Groxx 6y ago
- carapace 6y ago
  Long-time pythonista here. I read this the other day and, meaning no disrespect, I think you're off-base on this but I can't quite put my finger on it.
  - perrygeo 6y ago
    Care to elaborate or provide an example? Also long-time pythonista here and I find the article's recommendations spot on. There is huge difference between returning values (a concrete list) versus returning a stateful object that yields values (a generator). Yet Python's syntax allows you to treat them as equivalent and it can lead to bugs exactly as described. A context manager is the Pythonic way to deal with stateful objects and it ensures that you manage the lifecycle of the object explicitly, thus avoiding a whole class of potential bugs in your code. How would you handle the problem? Or do you not see it as a problem?
    - carapace 6y ago
      Well for one thing, I think as written the solution code is buggy: with judgments_open('judgments.txt') as judgments gather_features(judgments) train_model(judgments) # <-- bug. The context manager is still returning a single generator, not a list, so when you call train_model(judgments) it will have been exhausted, no? - - - - Second, the bug is not really solved by the proposed solution. This code is fine. with open('judgments.txt') as f: judgments = judgments_from_file(f) for j in judgments: process(j) This code is buggy. with open('judgments.txt') as f: judgments = judgments_from_file(f) for j in judgments: # <-- bug. process(j) The generator must not be used after the file is closed at the end of the context manager's scope. If you did this you would still have the bug: with judgments_open('judgments.txt') as judgments foo(judgments) # okay for j in judgments: # <-- same bug. process(j) In any event, in this case, I would just do: with open('judgments.txt') as f: judgments = list(judgments_from_file(f)) gather_features(judgments) train_model(judgments)
      - softwaredoug 6y ago
- ehsankia 6y ago
  Why not just have the with-statement/filehandler inside `judgments_from_file`? You're doing all sorts of messy stuff to make sure the file closes whenever `judgments_from_file` is done, wouldn't it make sense to just have it control the lifecycle of the file? The only reason I can think of to do it the other way is if you want to make unit testing easier, allowing you to pass a fake iterable instead of a file pointer to `judgments_from_file` for testing.
  - softwaredoug 6y ago
    I think that’s also valid... What you’re describing is more or less what judgments_open does. I just did it with a context manager. But you’re right there are other ways to safely manage the files lifetime. I also use this pattern for writing judgments, which has work to do when done judgments are all collected and the context exits.
tumidpandora 6y ago
I couldn't help but notice that the latest commit on the repo was over 3 years ago
carapace 6y ago
> Assigning to a list slice will replace that range in the original sequence with what's referenced even if their lengths are different. Not only that, but you can assign to a slice with a stride: In [1]: r = list(range(10)) In [2]: r Out[2]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] In [3]: r[1::3] = 10, 20, 30 In [4]: r Out[4]: [0, 10, 2, 3, 20, 5, 6, 30, 8, 9]
cs702 6y ago
Great suggestions for general-purpose code, but I would NOT recommend following them in 'mathematically dense' code (e.g., deep learning models), for which being able to fit as much logic as possible into a single editing screen becomes increasingly important as we increase the complexity of the code. Taken to the extreme, the "fit as much logic as possible into a single screen" approach leads naturally to code that looks a lot like something written in APL or one its descendants (J, K, etc.). I recognize this is not for everyone. Personally, I think Jeremy Howard and the folks at fast.ai have struck a pretty good balance of readability and succinctness with their coding style for ML/AI code: https://docs.fast.ai/dev/style.html https://docs.fast.ai/dev/style.html
- 0x402DF854 6y ago
  I don't quite enjoy "explicit mathematically dense" code, it leads to bad habits, see item 4: Python's syntax makes it all too easy to write single-line expressions that are overly complicated and difficult to read. If you really need to pack a lot of math, just define it in a module (e.g. tensorflow does that nicely). I've had an intern who came from MATLAB. Convincing them to stop packing numerical constants in the middle of every expression was unexpectedly difficult. "But it's shorter that way!"
- sweezyjeezy 6y ago
  I work in deep learning, and I'm very dubious of the benefit of terse code. In my experience 90% of novel deep learning models can be coded with base layer classes from tf/pytorch/keras + maybe a few hundred lines of 'new' code. For this code I feel like you will avoid mistakes and make things so much easier for the reader (probably yourself in 6 months), if you modularise, write it as clearly as possible and explain what each part is doing. I have used code from the fast.ai codebase and personally I find it horrendous to work with - I find the way they structure and name classes consfusing, they use wildcard imports everywhere, they use tiny variable names for everything - it's all extremely reader-unfriendly, which for an educational tool seems completely bizarre.
nickysielicki 6y ago
Items 7 and 8 seem to be at odds with each other to me, don't use reduce and map because it's ugly and unreadable, but if you use the alternative for reduce and map then you can't make a deeper pipe because it becomes too complicated to read? Get real! Is it even true that comprehensions are faster than reduce and map? I write a lot of python with reduce and map, and I know it's not considered a best practice for perf reasons, but whenever I go-ahead and rewrite it in the "more pythonic" way, I can't help but think that it ends up a lot less readable. I feel like this divide makes functional programming in python more of a hassle than it should be.
- dragonwriter 6y ago
  I think #7 would be clearer if it said “Use list comprehensions instead of map or filter where the latter would use lambda expressions, and in place of the combination of map and filter”. Map and filter where you are applying a function (including a function-returning expression) that doesn't require a lambda are, IMO, cleaner and avoiding an extra lambda or map/filter combo is explictly the basis of #7 (see #7.i., 7.ii.) I don't think this is bad writing, the author expects you to read the subpoints, but if you were to present the major points independently... #8 is a little unclear as to which expressions it is counting (particularly, whether it counts the return expression, which I don't think it intends to.) I think it's referring to the two total “for” and “if” clauses, so either two “for” or one of each as the preferred limit, which seems to me to be a sensible guideline. It think both #7 and #8, even with the additional clarification, need so to be considered in combination with #4: they set the rules as to what constitutes a complicated expression—either with comprehensions or map/filter/etc.—that calls for factoring part of it out into a helper function or named subexpression to avoid overly complex, code-golfy one-liners. > Is it even true that comprehensions are faster than reduce and map? While I'd love to see “accumulator comprehension” syntax* added to Python, in real current python comprehensions aren’t generally an alternative to reduce. * something like: (compute x from 0 as x+n for n in ns)
- jennasys 6y ago
  I used to have an issue with readability of list comprehensions until I started using them on a regular basis. Now they actually make more sense to me and just feel more like Python than using map/filter. I think if you are used to using map, especially if you are coming from another language, it can take a bit of effort to retrain your brain. In general I don't think there is a significant performance advantage either way except for in very specific cases.
deleted 6y ago
[deleted]

154 points by guptarohit 6y ago

dd82 6y ago

The author has a second edition out with 90 examples at https://effectivepython.com/ https://effectivepython.com/, and the corresponding github repo at https://github.com/bslatkin/effectivepython https://github.com/bslatkin/effectivepython Its been updated to be exclusive to 3.x, including 3.8 samples.

Alex3917 6y ago
Great book that every Python developer should own! Even for the recommendations you don't buy into, you'll still benefit just from reading a lucid argument in their favor.
guiambros 6y ago
The author also has a really nice Safari training video course[1]. Always worth reminding folks that Safari is included on a $99/yr ACM subscription, which is a great value. [1] https://learning.oreilly.com/videos/effective-python/9780134175249 https://learning.oreilly.com/videos/effective-python/9780134...

softwaredoug 6y ago

On “return generators instead of lists” I think it’s also important when you do this to return use a context manager to manage the generators lifetime. Otherwise you might be surprised when the underlying file or source is closed later in your code. I blogged about this: https://opensourceconnections.com/blog/2020/06/10/python-generators-and-context-managers/ https://opensourceconnections.com/blog/2020/06/10/python-gen...

xapata 6y ago
Maybe I misunderstood, but if you're suggesting the generator function open/close the resource, then I disagree. The caller has the responsibility of safely opening and closing. The generator function should not care what type of iterable it receives
- cheez 6y ago
  I agree, this unnecessarily complicates library-type code.
- softwaredoug 6y ago
  Then the caller must know they are receiving a generator dependent on the files lifetime. Pretty brittle IMO. Receiving a list wouldn’t depend on the files lifetime. This causes client code to be pretty error prone as you can very easily think you’re saving off a list built from a file, but in reality you hold a generator. Client code closes the file, the generator is used much later, and blammo, crash! NOT doing a context manager forces clients to work harder at guessing lifetimes. So you can’t just replace lists with generators. This break the uniform access principal. Client code IMO shouldn’t need to think about whether they got a generator or an interable container.
  - xapata 6y ago
    > Then the caller must know they are receiving a generator dependent on the files lifetime How would they not? The caller provides the file as an argument to the generator function. Further, the caller decides when to close the file, presumably via leaving a context and after the generator is consumed.
    - Groxx 6y ago
      it's not too hard to write code like: with open(...) as f: lines = process(f) # do stuff with lines ^ if `process` returns a generator, the file will be closed without having done anything.
      - xapata 6y ago
        Put `do stuff with lines` inside the context. Where it belongs.
        Groxx 6y ago
carapace 6y ago
Long-time pythonista here. I read this the other day and, meaning no disrespect, I think you're off-base on this but I can't quite put my finger on it.
- perrygeo 6y ago
  Care to elaborate or provide an example? Also long-time pythonista here and I find the article's recommendations spot on. There is huge difference between returning values (a concrete list) versus returning a stateful object that yields values (a generator). Yet Python's syntax allows you to treat them as equivalent and it can lead to bugs exactly as described. A context manager is the Pythonic way to deal with stateful objects and it ensures that you manage the lifecycle of the object explicitly, thus avoiding a whole class of potential bugs in your code. How would you handle the problem? Or do you not see it as a problem?
  - carapace 6y ago
    Well for one thing, I think as written the solution code is buggy: with judgments_open('judgments.txt') as judgments gather_features(judgments) train_model(judgments) # <-- bug. The context manager is still returning a single generator, not a list, so when you call train_model(judgments) it will have been exhausted, no? - - - - Second, the bug is not really solved by the proposed solution. This code is fine. with open('judgments.txt') as f: judgments = judgments_from_file(f) for j in judgments: process(j) This code is buggy. with open('judgments.txt') as f: judgments = judgments_from_file(f) for j in judgments: # <-- bug. process(j) The generator must not be used after the file is closed at the end of the context manager's scope. If you did this you would still have the bug: with judgments_open('judgments.txt') as judgments foo(judgments) # okay for j in judgments: # <-- same bug. process(j) In any event, in this case, I would just do: with open('judgments.txt') as f: judgments = list(judgments_from_file(f)) gather_features(judgments) train_model(judgments)
    - softwaredoug 6y ago
ehsankia 6y ago
Why not just have the with-statement/filehandler inside `judgments_from_file`? You're doing all sorts of messy stuff to make sure the file closes whenever `judgments_from_file` is done, wouldn't it make sense to just have it control the lifecycle of the file? The only reason I can think of to do it the other way is if you want to make unit testing easier, allowing you to pass a fake iterable instead of a file pointer to `judgments_from_file` for testing.
- softwaredoug 6y ago
  I think that’s also valid... What you’re describing is more or less what judgments_open does. I just did it with a context manager. But you’re right there are other ways to safely manage the files lifetime. I also use this pattern for writing judgments, which has work to do when done judgments are all collected and the context exits.

tumidpandora 6y ago

I couldn't help but notice that the latest commit on the repo was over 3 years ago

carapace 6y ago

> Assigning to a list slice will replace that range in the original sequence with what's referenced even if their lengths are different. Not only that, but you can assign to a slice with a stride: In [1]: r = list(range(10)) In [2]: r Out[2]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] In [3]: r[1::3] = 10, 20, 30 In [4]: r Out[4]: [0, 10, 2, 3, 20, 5, 6, 30, 8, 9]

cs702 6y ago

Great suggestions for general-purpose code, but I would NOT recommend following them in 'mathematically dense' code (e.g., deep learning models), for which being able to fit as much logic as possible into a single editing screen becomes increasingly important as we increase the complexity of the code. Taken to the extreme, the "fit as much logic as possible into a single screen" approach leads naturally to code that looks a lot like something written in APL or one its descendants (J, K, etc.). I recognize this is not for everyone. Personally, I think Jeremy Howard and the folks at fast.ai have struck a pretty good balance of readability and succinctness with their coding style for ML/AI code: https://docs.fast.ai/dev/style.html https://docs.fast.ai/dev/style.html

0x402DF854 6y ago
I don't quite enjoy "explicit mathematically dense" code, it leads to bad habits, see item 4: Python's syntax makes it all too easy to write single-line expressions that are overly complicated and difficult to read. If you really need to pack a lot of math, just define it in a module (e.g. tensorflow does that nicely). I've had an intern who came from MATLAB. Convincing them to stop packing numerical constants in the middle of every expression was unexpectedly difficult. "But it's shorter that way!"
sweezyjeezy 6y ago
I work in deep learning, and I'm very dubious of the benefit of terse code. In my experience 90% of novel deep learning models can be coded with base layer classes from tf/pytorch/keras + maybe a few hundred lines of 'new' code. For this code I feel like you will avoid mistakes and make things so much easier for the reader (probably yourself in 6 months), if you modularise, write it as clearly as possible and explain what each part is doing. I have used code from the fast.ai codebase and personally I find it horrendous to work with - I find the way they structure and name classes consfusing, they use wildcard imports everywhere, they use tiny variable names for everything - it's all extremely reader-unfriendly, which for an educational tool seems completely bizarre.

nickysielicki 6y ago

Items 7 and 8 seem to be at odds with each other to me, don't use reduce and map because it's ugly and unreadable, but if you use the alternative for reduce and map then you can't make a deeper pipe because it becomes too complicated to read? Get real! Is it even true that comprehensions are faster than reduce and map? I write a lot of python with reduce and map, and I know it's not considered a best practice for perf reasons, but whenever I go-ahead and rewrite it in the "more pythonic" way, I can't help but think that it ends up a lot less readable. I feel like this divide makes functional programming in python more of a hassle than it should be.

dragonwriter 6y ago
I think #7 would be clearer if it said “Use list comprehensions instead of map or filter where the latter would use lambda expressions, and in place of the combination of map and filter”. Map and filter where you are applying a function (including a function-returning expression) that doesn't require a lambda are, IMO, cleaner and avoiding an extra lambda or map/filter combo is explictly the basis of #7 (see #7.i., 7.ii.) I don't think this is bad writing, the author expects you to read the subpoints, but if you were to present the major points independently... #8 is a little unclear as to which expressions it is counting (particularly, whether it counts the return expression, which I don't think it intends to.) I think it's referring to the two total “for” and “if” clauses, so either two “for” or one of each as the preferred limit, which seems to me to be a sensible guideline. It think both #7 and #8, even with the additional clarification, need so to be considered in combination with #4: they set the rules as to what constitutes a complicated expression—either with comprehensions or map/filter/etc.—that calls for factoring part of it out into a helper function or named subexpression to avoid overly complex, code-golfy one-liners. > Is it even true that comprehensions are faster than reduce and map? While I'd love to see “accumulator comprehension” syntax* added to Python, in real current python comprehensions aren’t generally an alternative to reduce. * something like: (compute x from 0 as x+n for n in ns)
jennasys 6y ago
I used to have an issue with readability of list comprehensions until I started using them on a regular basis. Now they actually make more sense to me and just feel more like Python than using map/filter. I think if you are used to using map, especially if you are coming from another language, it can take a bit of effort to retrain your brain. In general I don't think there is a significant performance advantage either way except for in very specific cases.

deleted 6y ago

[deleted]