5 ms·

The Ultimate Guide to Error Handling in Python

96 points by kdamica 2y ago

InDubioProRubio 2y ago
It still has exceptions- but exceptions are normal. Errors are normal. How would one do : https://fsharpforfunandprofit.com/rop/ https://fsharpforfunandprofit.com/rop/ in python?
- benrutter 2y ago
  I think monad patterns can work pretty well in python, but my experience is nobody uses them. I wrote a small utility library for exactly that kind of "railway oriented programming" if you're curious about what it'd look like in python: https://github.com/benrutter/ufo-tools https://github.com/benrutter/ufo-tools
  - fire_lake 2y ago
    Without do-notation (computation expressions in F#) it’s hard to see how this can scale.
    - gpderetta 2y ago
      you might be able to wrestle yield to do your bidding.
- meehai 2y ago
  what about this pattern? https://www.inngest.com/blog/python-errors-as-values https://www.inngest.com/blog/python-errors-as-values I tried it once in an sqlite DB connector with some business logic and simply checking stuff like res: DBException | Result = db_handler.some_business_logic() if isinstance(res, DBException): return res # you can also log or even raise if this function isn't returning exceptions as values # guaranteed to be Result type here See here: - https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/db_handler.py?ref_type=heads#L186 https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/db_han... - https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/app.py?ref_type=heads#L69 https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/app.py...
atoav 2y ago
You can do nearly everything using Python errors. One thing that bothers me a bit, is the amount of text needed to catch them and bubble them up. Depending on the programming style traditional exceptions may also not be adequately re For this Rust style error handling can be interesting, eg using poltergeist (https://github.com/alexandermalyga/poltergeist https://github.com/alexandermalyga/poltergeist) or result (https://pypi.org/project/result/ https://pypi.org/project/result/).
- drcongo 2y ago
  Would you mind explaining what the benefits of either of those libraries is over normal exception handling? The examples at the top of the Result readme seem almost exactly as verbose as each other. I kinda like the idea of a Result object, but it seems like a lot of library imports for no gain so I'm sure I'm missing something.
  - benrutter 2y ago
    Not OP, but as someone who likes this style of exception handling, I really appreciate that it offloads everything to the type system. With normal exceptions, I have to use my meat brain to figure out if I've handled all the exceptions that might come up, whereas mypy can just tell me if I've handled the result object correctly That said, I don't use either library since, in my experience, you need them to be used all the way down the stack to actually get the benefit.
    - drcongo 2y ago
      Ahhh, I get it now, thank you!
    - BerislavLopac 2y ago
      > I have to use my meat brain to figure out if I've handled all the exceptions that might come up The point of exceptions is exactly that you don't want to handle all that might come up -- you need to handle only those that you actually expect and know how to handle. Everything else is left to bubble up and either be handled by a custom, generic top-level handler, or by the language's default mechanism (in Python's case, dumping a stack trace).
syllogism 2y ago
The examples in the article recommend doing the right things, but I think the initial discussion of the LBYL vs EAFP discussion could be better. The article says basically "LBYL is bad", but this isn't a good description of what the author does in the later examples. Following "EAFP" without exception is also really bad. The simple policy is that you should use "LBYL" when you're dealing with local state that can't change out from under you, especially when it's just properties of your local variables, but you need to use "EAFP" for anything that deals with remote state. Function calls can go either way, depending. Blanket advice against LBYL leads people to use try/except blocks to express basic conditionals. If you want to ask something about your local variables, just ask it in a conditional --- don't make the reader infer "ah, these are the situations which will raise the error, and therefore we'll go into this block if the values are this way". If you want to do something when a key is missing from a dictionary, don't write: try: value = table[key] except KeyError: return default_value Just write: if key not in table: return default_value value = table[key] (If you really need to avoid two lookups for efficiency, you would do something like value = table.get(key, MISSING_VALUE) and then check 'if value is missing_value'. But this optimisation will seldom be necessary.) The example the author gives about interacting with the file system is a good example of where indeed you really should use EAFP. The file system is remote state that your function does not own. Similarly if you're doing database operations, calling a remote API...lots of things. There's various middle ground when you're calling functions. Often you don't want to worry about whether that function is going to be interacting with remote state, and you want to treat it totally as a black box, so you just use EAFP. But if the function documents really clear pre-conditions of your variables you can just go ahead and check, it's better to do that.
- stevesimmons 2y ago
  What's really wrong with try/except here other than it's not to your personal taste? Brett Cannon, one of the Python core devs, wrote a blog post using exactly this dict KeyError example in 2016 [1]. It concludes: "The key takeaway you should have for the post is that EAFP is a legitimate coding style in Python and you should feel free to use it when it makes sense." [1] https://devblogs.microsoft.com/python/idiomatic-python-eafp-versus-lbyl/ https://devblogs.microsoft.com/python/idiomatic-python-eafp-...
alex_suzuki 2y ago
One thing that bothers me in my day-to-day Python work is that often libraries don’t document the errors that can be thrown by their functions. As the language lacks a “throws” statement, I found myself digging through library code on multiple occasions. How do people approach this?
- BerislavLopac 2y ago
  > the language lacks a “throws” statement I'm not sure what you mean by this? Python has the "raise" statement [0], which "throws" an error from within a function or other code. [0] https://docs.python.org/3/reference/simple_stmts.html#raise https://docs.python.org/3/reference/simple_stmts.html#raise
  - silvester23 2y ago
    What they mean is that in Java, for example, a method has to explicitly state which exceptions it might throw as part of its signature. Note that they said "throws", not "throw". Python does not have that.
    - BerislavLopac 2y ago
      Ah, I understand now. Well, by default Python doesn't declare a return type either yet the tools are able to infer it in many cases; I see no reason why tools like mypy couldn't similarly infer the raised exception types as well. Plus, the typing annotations could presumably be expanded to include some notification to declare raised exception types explicitly.
  - alex_suzuki 2y ago
    Yes sure, but the raise statement is part of the implementation (like throw in Java), and not part of the signature (throws), which would presumably make it into a generated API documentation.
  - valbaca 2y ago
    They're talking about Java's Checked Exceptions: https://www.baeldung.com/java-checked-unchecked-exceptions https://www.baeldung.com/java-checked-unchecked-exceptions which are part of a method's declaration `void foo() throws KaboomException` The ironic thing is that Checked Exceptions aren't particularly well-loved in Java. For example, Kotlin (which is like Java++) did away with them.
deleted 2y ago
[deleted]
valbaca 2y ago
Can someone explain the last bit: print(f"Unexpected error: {error}") sys.exit(1) How is that any different than just letting the exception throw all the way out? Is all this doing is making sure the exit code is 1 instead of some other non-zero value? I'm not seeing how this is any better than just letting the program hard-crash. (I am asking in earnest. I'm getting back into Python and wanting to know what's generally preferred)
- krysp 2y ago
  It gives you control over the output of your program (and as you mention, the exact error code - which probably should be 1 in this instance). It's often more user friendly to print a short human-readable message (or nothing) than to dump a huge stack trace. YMMV, if this is an internal dev tool then the opposite might be true!
- whinvik 2y ago
  I feel like both of them should crash the application. The only benefit is that you don't see the full stack trace in production, which depending on circumstances you don't want to be shown.

96 points by kdamica 2y ago

It still has exceptions- but exceptions are normal. Errors are normal. How would one do : https://fsharpforfunandprofit.com/rop/ https://fsharpforfunandprofit.com/rop/ in python?

benrutter 2y ago
I think monad patterns can work pretty well in python, but my experience is nobody uses them. I wrote a small utility library for exactly that kind of "railway oriented programming" if you're curious about what it'd look like in python: https://github.com/benrutter/ufo-tools https://github.com/benrutter/ufo-tools
- fire_lake 2y ago
  Without do-notation (computation expressions in F#) it’s hard to see how this can scale.
  - gpderetta 2y ago
    you might be able to wrestle yield to do your bidding.
meehai 2y ago
what about this pattern? https://www.inngest.com/blog/python-errors-as-values https://www.inngest.com/blog/python-errors-as-values I tried it once in an sqlite DB connector with some business logic and simply checking stuff like res: DBException | Result = db_handler.some_business_logic() if isinstance(res, DBException): return res # you can also log or even raise if this function isn't returning exceptions as values # guaranteed to be Result type here See here: - https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/db_handler.py?ref_type=heads#L186 https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/db_han... - https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/app.py?ref_type=heads#L69 https://gitlab.com/meehai/drpciv-flask/-/blob/main/be/app.py...

atoav 2y ago

You can do nearly everything using Python errors. One thing that bothers me a bit, is the amount of text needed to catch them and bubble them up. Depending on the programming style traditional exceptions may also not be adequately re For this Rust style error handling can be interesting, eg using poltergeist (https://github.com/alexandermalyga/poltergeist https://github.com/alexandermalyga/poltergeist) or result (https://pypi.org/project/result/ https://pypi.org/project/result/).

drcongo 2y ago
Would you mind explaining what the benefits of either of those libraries is over normal exception handling? The examples at the top of the Result readme seem almost exactly as verbose as each other. I kinda like the idea of a Result object, but it seems like a lot of library imports for no gain so I'm sure I'm missing something.
- benrutter 2y ago
  Not OP, but as someone who likes this style of exception handling, I really appreciate that it offloads everything to the type system. With normal exceptions, I have to use my meat brain to figure out if I've handled all the exceptions that might come up, whereas mypy can just tell me if I've handled the result object correctly That said, I don't use either library since, in my experience, you need them to be used all the way down the stack to actually get the benefit.
  - drcongo 2y ago
    Ahhh, I get it now, thank you!
  - BerislavLopac 2y ago
    > I have to use my meat brain to figure out if I've handled all the exceptions that might come up The point of exceptions is exactly that you don't want to handle all that might come up -- you need to handle only those that you actually expect and know how to handle. Everything else is left to bubble up and either be handled by a custom, generic top-level handler, or by the language's default mechanism (in Python's case, dumping a stack trace).

syllogism 2y ago

The examples in the article recommend doing the right things, but I think the initial discussion of the LBYL vs EAFP discussion could be better. The article says basically "LBYL is bad", but this isn't a good description of what the author does in the later examples. Following "EAFP" without exception is also really bad. The simple policy is that you should use "LBYL" when you're dealing with local state that can't change out from under you, especially when it's just properties of your local variables, but you need to use "EAFP" for anything that deals with remote state. Function calls can go either way, depending. Blanket advice against LBYL leads people to use try/except blocks to express basic conditionals. If you want to ask something about your local variables, just ask it in a conditional --- don't make the reader infer "ah, these are the situations which will raise the error, and therefore we'll go into this block if the values are this way". If you want to do something when a key is missing from a dictionary, don't write: try: value = table[key] except KeyError: return default_value Just write: if key not in table: return default_value value = table[key] (If you really need to avoid two lookups for efficiency, you would do something like value = table.get(key, MISSING_VALUE) and then check 'if value is missing_value'. But this optimisation will seldom be necessary.) The example the author gives about interacting with the file system is a good example of where indeed you really should use EAFP. The file system is remote state that your function does not own. Similarly if you're doing database operations, calling a remote API...lots of things. There's various middle ground when you're calling functions. Often you don't want to worry about whether that function is going to be interacting with remote state, and you want to treat it totally as a black box, so you just use EAFP. But if the function documents really clear pre-conditions of your variables you can just go ahead and check, it's better to do that.

stevesimmons 2y ago
What's really wrong with try/except here other than it's not to your personal taste? Brett Cannon, one of the Python core devs, wrote a blog post using exactly this dict KeyError example in 2016 [1]. It concludes: "The key takeaway you should have for the post is that EAFP is a legitimate coding style in Python and you should feel free to use it when it makes sense." [1] https://devblogs.microsoft.com/python/idiomatic-python-eafp-versus-lbyl/ https://devblogs.microsoft.com/python/idiomatic-python-eafp-...

alex_suzuki 2y ago

One thing that bothers me in my day-to-day Python work is that often libraries don’t document the errors that can be thrown by their functions. As the language lacks a “throws” statement, I found myself digging through library code on multiple occasions. How do people approach this?

BerislavLopac 2y ago
> the language lacks a “throws” statement I'm not sure what you mean by this? Python has the "raise" statement [0], which "throws" an error from within a function or other code. [0] https://docs.python.org/3/reference/simple_stmts.html#raise https://docs.python.org/3/reference/simple_stmts.html#raise
- silvester23 2y ago
  What they mean is that in Java, for example, a method has to explicitly state which exceptions it might throw as part of its signature. Note that they said "throws", not "throw". Python does not have that.
  - BerislavLopac 2y ago
    Ah, I understand now. Well, by default Python doesn't declare a return type either yet the tools are able to infer it in many cases; I see no reason why tools like mypy couldn't similarly infer the raised exception types as well. Plus, the typing annotations could presumably be expanded to include some notification to declare raised exception types explicitly.
- alex_suzuki 2y ago
  Yes sure, but the raise statement is part of the implementation (like throw in Java), and not part of the signature (throws), which would presumably make it into a generated API documentation.
- valbaca 2y ago
  They're talking about Java's Checked Exceptions: https://www.baeldung.com/java-checked-unchecked-exceptions https://www.baeldung.com/java-checked-unchecked-exceptions which are part of a method's declaration `void foo() throws KaboomException` The ironic thing is that Checked Exceptions aren't particularly well-loved in Java. For example, Kotlin (which is like Java++) did away with them.

deleted 2y ago

[deleted]

valbaca 2y ago

Can someone explain the last bit: print(f"Unexpected error: {error}") sys.exit(1) How is that any different than just letting the exception throw all the way out? Is all this doing is making sure the exit code is 1 instead of some other non-zero value? I'm not seeing how this is any better than just letting the program hard-crash. (I am asking in earnest. I'm getting back into Python and wanting to know what's generally preferred)

krysp 2y ago
It gives you control over the output of your program (and as you mention, the exact error code - which probably should be 1 in this instance). It's often more user friendly to print a short human-readable message (or nothing) than to dump a huge stack trace. YMMV, if this is an internal dev tool then the opposite might be true!
whinvik 2y ago
I feel like both of them should crash the application. The only benefit is that you don't see the full stack trace in production, which depending on circumstances you don't want to be shown.