11 ms·

Raising exceptions or returning error objects in Python

44 points by EntICOnc 4y ago

tbrlpld 4y ago
Has anyone tried this: https://returns.readthedocs.io/en/latest/pages/result.html https://returns.readthedocs.io/en/latest/pages/result.html
hlpweg 4y ago
Zababa 4y ago
Interesting, I've seen the same discussion but applied to OCaml and with an option type vs an exception. For example, the module List has both List.nth that raises Failure if the list is too short, and List.nth_opt that returns None of the list is too short. One aspect that I've seen mentioned about OCaml and not with this article is performance. An option will trigger an allocation, and will generay be slower than an exception. One the other hand, the option approach is safer. I'm not sure if that's true with Python too though.
- g_delgado14 4y ago
  Performance cost is a legitimate concern but one that should be considered in relative terms. e.g. yes there are more allocations, but in what context is the code being used? Are we building gigabyte-scale real-time systems? no? ok then let's maybe not worry about performance right now and instead optimize for intuition, maintainability and readability. Obviously while at the same time monitoring CPU and memory consumption of our applications to help us determine when we might want to look to optimize our code.
  - dahfizz 4y ago
    > then let's maybe not worry about performance right now and instead optimize for intuition, maintainability and readability. I would argue that throwing exceptions is the intuitive and readable thing for Python. Having to call isinstance for all possible return types of a function is not the "pythonic" way.
- dahfizz 4y ago
  That's interesting - coming from C++, exceptions are much slower than returning -1 on error. I suppose this is because C++ doesn't need to allocate a Option.
  - int_19h 4y ago
    Throwing an exception is much slower, but that only happens on error. With option types, you still have to create and return an instance even if successful, and if that causes a heap allocation, you're worse off than exceptions if the success case is much more likely. That said, it's really an OCaml implementation deficiency - std::optional in C++ is not heap-allocated, for example (although there's still some overhead due to sizeof being larger & possibly not fitting in a single register). On the other hand, in (C)Python, exceptions behave more or less like in C++ semantically, but under the hood the VM implements it all as error return codes and thread-local "current exception" pointer, which is closer to std::option perf-wise than it is to real exceptions. Hence why performance is rarely a consideration when deciding whether to throw or return in Python, and why exceptions are so idiomatic there.
g_delgado14 4y ago
Yessss... I'm a big advocate of using error objects rather than throwing exceptions. I think the industry over-uses throwing exceptions and it's basically turned into the new GOTO statement. I wrote an article about this as well but for TypeScript [0] and I've got a relatively popular npm package that does what I mention in the article [1] --- [0] - https://gdelgado.ca/type-safe-error-handling-in-typescript.html#title https://gdelgado.ca/type-safe-error-handling-in-typescript.h... [1] - https://www.npmjs.com/package/neverthrow https://www.npmjs.com/package/neverthrow
- throwaway290 4y ago
  I'm not surprised. TS exceptions feel limiting and awkward and I don't like using them, compared with Python's. Add to that, TS's powerful typing makes it easier to precisely type return values compared to Python's annotations, but still you cannot type what function throws in TS. In Go, I tend to use the return error pattern. In Python, I raise and try. With TS, I do both (but slightly more confident with returning errors). IMO, rule of thumb: try not to surprise the developer after you, who most likely will be hired for knowing this language not this project (note: that developer may be future yourself). When in Rome...
  - g_delgado14 4y ago
    ... do as the Roman's do! Agreed :)
- int_19h 4y ago
  It really depends on the language. In Python, exceptions are idiomatic even for non-fatal failures, the runtime is optimized for that performance-wise, and it's what API clients expect.
- Supermancho 4y ago
  > I think the industry over-uses throwing exceptions and it's basically turned into the new GOTO This is a misunderstanding of what's bad about GOTO versus jumping to a label. eg Functions are labels you GOTO. The difference between a GOTO instruction and a Function call, for the purposes of differentiating the methodology of code routing, is that a Function is part of a stack that returns in a serial fashion. ie Each element of the stack returns to the parent before disappearing, which can be though of as "unwinding". An exception unwinds a Function stack without restriction (eg no consideration of signatures like args/returns), until it is caught in that stack, making them conceptually and practically different from GOTOs.
dahfizz 4y ago
> If you instead raise exceptions, you are immediately forcing the calling code into a special control flow structure, namely the try/except dance, which can be inconvenient. I fail to see how a try/catch is more inconvenient than a handful of isinstance calls. What happens when you want to add a new error case? With exceptions, the exception will always be thrown. With error objects, you're going to fall through a bunch of isinstance checks and continue on with a result object which isn't the type you think it is. Then, some time later, youre going to try and reference a field that doesn't exist. This is the kind of thing which is a nightmare to debug. I do appreciate the author's point that error objects are nice in functional programming languages. But using them in Python because you like using them in fp is not a good justification.
- dragonwriter 4y ago
  > With error objects, you're going to fall through a bunch of isinstance checks and continue on with a result object which isn't the type you think it is. If you are doing isinstance checks (or pattern matching or something else equivalent) for separating error/success returns where, as in Python, you don't have exhaustiveness checks, your default case should be treating the result as an unknown error, not a success. Also, this error can be avoided on the called-function side by using a common type (concrete or root) for errors. Using a wrapper for errors, as with a Result type (whether or not successes are also wrapped as they would be with a result) achieves this, for instance.
  - dahfizz 4y ago
    Yes, having a default case that throws an Unknown Error is the correct thing to do here. But that is still less ergonomic. For an unhandled exception type, the person creating the exception can attach a helpful error message. For an unhandled exception object, you can't do any better than "An Unexpected Error Occurred". You can do a little better by having multiple error subclasses. But at that point you're literally reinventing exceptions, but with a greater onus of the caller to do the right thing. This seems to go against the "pit of success" idea from the article. There's no way for the developer to accidentally miss an exception, or misunderstand the subtleties of the error inheritance structure.
zmmmmm 4y ago
One thing I miss when the error object pattern is used in preference to exceptions is the stack trace. With an exception, you get a detailed log of the full stack that led to the error. With an error object, I have to reverse engineer out all the places in the code that could potentially return that type of error. In a perfect world, you don't need to know that, and all the information you need is in the error object. But we don't live in a perfect world and very often the stack trace is the main lead you have to debug some crazy thing happening in production.
- akdor1154 4y ago
  Yes i have hit this too. My planned goto approach if I was on a prod app instead of a toy, is to use something like `result`, but with the `Err` variant modded to generate an internal traceback whenever it's constructed. With that said i haven't ever done the above - anyone done something similar? Or approached this a different way?
  - bb88 4y ago
    Given enough digging around in the python runtime system I'm sure you can find the current stack trace and shove that in an error class. But now it's a lesser error than an exception. Any exception while processing your error will cause the stack to unwind and drop the error variable with the stack trace inside of it. Now if you raise an exception while in the handling of an exception, python will keep around both stack traces.
kstrauser 4y ago
In some API code, we adopted and use the heck of out the pattern of making Exception subclasses that mapped to HTTP error codes, like: class NotFoundError(ClientError): """Not Found.""" status = 404 The top level error handler would catch exceptions raised like: raise NotFoundError(f"The requested object ID {object_id} doesn't exist.") and turn them into consistently formatted error messages. The end result was that you could write very readable, understandable code like: def handle(request): validate_signature(request) check_authorization(request) person = fetch_from_database(request.object_id) return {"first_name": person.first_name} The `validate_signature` method call may raise a 400 if the request is invalid. `check_authorization` could raise a 403 exception. `fetch_from_database` would raise a 404 if the row couldn't be fetched. In effect, each line of that function has an implicit early return: if it fails, processing stops there. Compare and contrast with: def handle(request): if validate_signature(request): if check_authorization(request): person = fetch_from_database(request.object_id) if person: return {"first_name": person.first_name} else: return 404 else: return 403 else: return 400 Python's exceptions are idiomatic. Embrace them.
- eddd-ddde 4y ago
  What about actually using early returns ? I feel like the implicitness of exceptions is more of a downside than an advantage def handle(request): if !validate_signature(request): return 400 if !check_authorization(request): return 403 person = fetch_from_database(request.object_id) if !person: return 404 return { "first_name":person.first_name }
  - kstrauser 4y ago
    That's twice the code for the same result. Also, `validate_signature` itself might have lots of similar code, like def validate_signature(request): signature = sig_from_request(request) payload = payload_from_request(request) check_signature(payload, signature) where each of those nested calls follows the same pattern. Yeah, you can do all of that with passing around error objects. A lot of people writing Go do that every day, so obviously it works and it's not overwhelmingly tedious. I highly prefer the Python style of EAFP though.
bb88 4y ago
I personally like requests api. Requests gives you a response, but let's you raise an exception if an error case was caught (404, 500, etc) And it's only one type (Response) and doesn't force the user into using isinstance to figure out if it succeeded or not.
akdor1154 4y ago
++, i normally use the `result` lib to wrap good/error return values, but your approach without it honestly doesn't lose that much. I would still use `result` for bigger projects or if I was dubious that consumer would do the right thing though.
yccs27 4y ago
One place where you are pretty much forced to use error objects is queue-based concurrency. In my case, a seperate thread listened for network events, parsed them and put the results on a queue for the main thread to consume. For parsing errors, I needed to put error objects onto the queue instead of terminating the whole thread with an exception. In this sense, error objects generalize much better to cases where you don't have an usual function-call structure.

44 points by EntICOnc 4y ago

Has anyone tried this: https://returns.readthedocs.io/en/latest/pages/result.html https://returns.readthedocs.io/en/latest/pages/result.html

hlpweg 4y ago

Zababa 4y ago

Interesting, I've seen the same discussion but applied to OCaml and with an option type vs an exception. For example, the module List has both List.nth that raises Failure if the list is too short, and List.nth_opt that returns None of the list is too short. One aspect that I've seen mentioned about OCaml and not with this article is performance. An option will trigger an allocation, and will generay be slower than an exception. One the other hand, the option approach is safer. I'm not sure if that's true with Python too though.

g_delgado14 4y ago
Performance cost is a legitimate concern but one that should be considered in relative terms. e.g. yes there are more allocations, but in what context is the code being used? Are we building gigabyte-scale real-time systems? no? ok then let's maybe not worry about performance right now and instead optimize for intuition, maintainability and readability. Obviously while at the same time monitoring CPU and memory consumption of our applications to help us determine when we might want to look to optimize our code.
- dahfizz 4y ago
  > then let's maybe not worry about performance right now and instead optimize for intuition, maintainability and readability. I would argue that throwing exceptions is the intuitive and readable thing for Python. Having to call isinstance for all possible return types of a function is not the "pythonic" way.
dahfizz 4y ago
That's interesting - coming from C++, exceptions are much slower than returning -1 on error. I suppose this is because C++ doesn't need to allocate a Option.
- int_19h 4y ago
  Throwing an exception is much slower, but that only happens on error. With option types, you still have to create and return an instance even if successful, and if that causes a heap allocation, you're worse off than exceptions if the success case is much more likely. That said, it's really an OCaml implementation deficiency - std::optional in C++ is not heap-allocated, for example (although there's still some overhead due to sizeof being larger & possibly not fitting in a single register). On the other hand, in (C)Python, exceptions behave more or less like in C++ semantically, but under the hood the VM implements it all as error return codes and thread-local "current exception" pointer, which is closer to std::option perf-wise than it is to real exceptions. Hence why performance is rarely a consideration when deciding whether to throw or return in Python, and why exceptions are so idiomatic there.

g_delgado14 4y ago

Yessss... I'm a big advocate of using error objects rather than throwing exceptions. I think the industry over-uses throwing exceptions and it's basically turned into the new GOTO statement. I wrote an article about this as well but for TypeScript [0] and I've got a relatively popular npm package that does what I mention in the article [1] --- [0] - https://gdelgado.ca/type-safe-error-handling-in-typescript.html#title https://gdelgado.ca/type-safe-error-handling-in-typescript.h... [1] - https://www.npmjs.com/package/neverthrow https://www.npmjs.com/package/neverthrow

throwaway290 4y ago
I'm not surprised. TS exceptions feel limiting and awkward and I don't like using them, compared with Python's. Add to that, TS's powerful typing makes it easier to precisely type return values compared to Python's annotations, but still you cannot type what function throws in TS. In Go, I tend to use the return error pattern. In Python, I raise and try. With TS, I do both (but slightly more confident with returning errors). IMO, rule of thumb: try not to surprise the developer after you, who most likely will be hired for knowing this language not this project (note: that developer may be future yourself). When in Rome...
- g_delgado14 4y ago
  ... do as the Roman's do! Agreed :)
int_19h 4y ago
It really depends on the language. In Python, exceptions are idiomatic even for non-fatal failures, the runtime is optimized for that performance-wise, and it's what API clients expect.
Supermancho 4y ago
> I think the industry over-uses throwing exceptions and it's basically turned into the new GOTO This is a misunderstanding of what's bad about GOTO versus jumping to a label. eg Functions are labels you GOTO. The difference between a GOTO instruction and a Function call, for the purposes of differentiating the methodology of code routing, is that a Function is part of a stack that returns in a serial fashion. ie Each element of the stack returns to the parent before disappearing, which can be though of as "unwinding". An exception unwinds a Function stack without restriction (eg no consideration of signatures like args/returns), until it is caught in that stack, making them conceptually and practically different from GOTOs.

dahfizz 4y ago

> If you instead raise exceptions, you are immediately forcing the calling code into a special control flow structure, namely the try/except dance, which can be inconvenient. I fail to see how a try/catch is more inconvenient than a handful of isinstance calls. What happens when you want to add a new error case? With exceptions, the exception will always be thrown. With error objects, you're going to fall through a bunch of isinstance checks and continue on with a result object which isn't the type you think it is. Then, some time later, youre going to try and reference a field that doesn't exist. This is the kind of thing which is a nightmare to debug. I do appreciate the author's point that error objects are nice in functional programming languages. But using them in Python because you like using them in fp is not a good justification.

dragonwriter 4y ago
> With error objects, you're going to fall through a bunch of isinstance checks and continue on with a result object which isn't the type you think it is. If you are doing isinstance checks (or pattern matching or something else equivalent) for separating error/success returns where, as in Python, you don't have exhaustiveness checks, your default case should be treating the result as an unknown error, not a success. Also, this error can be avoided on the called-function side by using a common type (concrete or root) for errors. Using a wrapper for errors, as with a Result type (whether or not successes are also wrapped as they would be with a result) achieves this, for instance.
- dahfizz 4y ago
  Yes, having a default case that throws an Unknown Error is the correct thing to do here. But that is still less ergonomic. For an unhandled exception type, the person creating the exception can attach a helpful error message. For an unhandled exception object, you can't do any better than "An Unexpected Error Occurred". You can do a little better by having multiple error subclasses. But at that point you're literally reinventing exceptions, but with a greater onus of the caller to do the right thing. This seems to go against the "pit of success" idea from the article. There's no way for the developer to accidentally miss an exception, or misunderstand the subtleties of the error inheritance structure.

zmmmmm 4y ago

One thing I miss when the error object pattern is used in preference to exceptions is the stack trace. With an exception, you get a detailed log of the full stack that led to the error. With an error object, I have to reverse engineer out all the places in the code that could potentially return that type of error. In a perfect world, you don't need to know that, and all the information you need is in the error object. But we don't live in a perfect world and very often the stack trace is the main lead you have to debug some crazy thing happening in production.

akdor1154 4y ago
Yes i have hit this too. My planned goto approach if I was on a prod app instead of a toy, is to use something like `result`, but with the `Err` variant modded to generate an internal traceback whenever it's constructed. With that said i haven't ever done the above - anyone done something similar? Or approached this a different way?
- bb88 4y ago
  Given enough digging around in the python runtime system I'm sure you can find the current stack trace and shove that in an error class. But now it's a lesser error than an exception. Any exception while processing your error will cause the stack to unwind and drop the error variable with the stack trace inside of it. Now if you raise an exception while in the handling of an exception, python will keep around both stack traces.

kstrauser 4y ago

In some API code, we adopted and use the heck of out the pattern of making Exception subclasses that mapped to HTTP error codes, like: class NotFoundError(ClientError): """Not Found.""" status = 404 The top level error handler would catch exceptions raised like: raise NotFoundError(f"The requested object ID {object_id} doesn't exist.") and turn them into consistently formatted error messages. The end result was that you could write very readable, understandable code like: def handle(request): validate_signature(request) check_authorization(request) person = fetch_from_database(request.object_id) return {"first_name": person.first_name} The `validate_signature` method call may raise a 400 if the request is invalid. `check_authorization` could raise a 403 exception. `fetch_from_database` would raise a 404 if the row couldn't be fetched. In effect, each line of that function has an implicit early return: if it fails, processing stops there. Compare and contrast with: def handle(request): if validate_signature(request): if check_authorization(request): person = fetch_from_database(request.object_id) if person: return {"first_name": person.first_name} else: return 404 else: return 403 else: return 400 Python's exceptions are idiomatic. Embrace them.

eddd-ddde 4y ago
What about actually using early returns ? I feel like the implicitness of exceptions is more of a downside than an advantage def handle(request): if !validate_signature(request): return 400 if !check_authorization(request): return 403 person = fetch_from_database(request.object_id) if !person: return 404 return { "first_name":person.first_name }
- kstrauser 4y ago
  That's twice the code for the same result. Also, `validate_signature` itself might have lots of similar code, like def validate_signature(request): signature = sig_from_request(request) payload = payload_from_request(request) check_signature(payload, signature) where each of those nested calls follows the same pattern. Yeah, you can do all of that with passing around error objects. A lot of people writing Go do that every day, so obviously it works and it's not overwhelmingly tedious. I highly prefer the Python style of EAFP though.

bb88 4y ago

I personally like requests api. Requests gives you a response, but let's you raise an exception if an error case was caught (404, 500, etc) And it's only one type (Response) and doesn't force the user into using isinstance to figure out if it succeeded or not.

akdor1154 4y ago

++, i normally use the `result` lib to wrap good/error return values, but your approach without it honestly doesn't lose that much. I would still use `result` for bigger projects or if I was dubious that consumer would do the right thing though.

yccs27 4y ago

One place where you are pretty much forced to use error objects is queue-based concurrency. In my case, a seperate thread listened for network events, parsed them and put the results on a queue for the main thread to consume. For parsing errors, I needed to put error objects onto the queue instead of terminating the whole thread with an exception. In this sense, error objects generalize much better to cases where you don't have an usual function-call structure.