8 ms·

Python errors as values: Comparing useful patterns from Rust and Go

• oDot 3y ago
  In FileMonger[0], which uses Tauri, I have implemented a `Result` logic for errors in TS, similar to what's available in Rust. It's clunkier, but still much preferable to the mess of throwing. Much easier to handle errors and debug. [0] https://filemonger.app/ https://filemonger.app/
  • tonyhb 3y ago
    I don't know Python too well but I think this is covered in TFA. There's a comparison between a Result type using https://github.com/rustedpy/result https://github.com/rustedpy/result and an ADT/sum type — and the sum type looks like it's way more ergonomic with Python's current typechecker. While I like Rust — and Result works really well with `?` — it doesn't actually look like that's the best pattern for Python?
  • ewuhic 3y ago
    Wanna opensource that part as a library, or at least write a blogpost?
    • oDot 3y ago
      I have written a rudimentary implementation to avoid external dependencies. If you don't mind, a quick google comes up with a couple: https://github.com/vultix/ts-results https://github.com/vultix/ts-results https://github.com/badrap/result https://github.com/badrap/result
• klyrs 3y ago
  Oh, good, heavyweight error handling just in time for py3.11's zero-cost exception happy path. But, more generously: why not simply return an error, and use isinstance(val, Error) for error handling? Making objects and calling functions is quite costly, and that can largely be avoided.
  • all2 3y ago
    This is what they ended the article on. Return a union type and then error check using isinstance.
    • klyrs 3y ago
      Oh, so they did. I guess I got bored after reading so many bad approaches trying to write go/rust in python.
  • avgcorrection 3y ago
    > Oh, good, heavyweight error handling just in time for py3.11's zero-cost exception happy path. I don’t get it. Languages that use exceptions for all kinds of errors will also use exceptions for routine errors that happen as a matter of course—the happy path is not the overwhelmingly most common branch, and errors are not exceptional. In turn not zero-cost for all but the exceptional case.
    • klyrs 3y ago
      The solution to this is to not use exceptions except for actual errors, unless it's going to be amortized away. For example, objects which have a heavy parameter, which gets memoized. With the zero-cost happy path, you only raise the exception once, when you're doing the costly thing, and subsequent accesses are free.
  • ed_blackburn 3y ago
    Sure, it depends on what is expensive. For us, in a primarily asynchronous domain, saving cycles is helpful for lower computational costs and the environment, but in the main, we optimise for reading the code, and one way to lower the cognitive load is to avoid using errors to control flow.
• TrustPatches 3y ago
  One problem I’ve experienced doing something like this is you end up with both exceptions and error values since the standard library and 3rd party libraries are still primarily exception based. You either have to live with it or create wrappers that catch errors and return them as values.
  • goodoldneon 3y ago
    We write wrappers for standard and third-party libraries that throw errors. For example, we have our own dump_json function that catches-and-returns errors thrown by json.dumps. We didn't need many wrappers given the nature of our SDK, but some programs will need many wrappers and that could get unwieldy
    • dmurray 3y ago
      But people are going to import your SDK, so your customers will be stuck with a mix of thrown exceptions and errors-as-values in their code. Based on your decision to write wrappers internally, presumably you'd argue that the sensible thing for them to do is to write wrappers either for your SDK or for every other library to get back to a consistent style and less mental overhead.
      • goodoldneon 3y ago
        Author here! We don't return errors in our public methods so our SDK still feels Pythonic. We're debating whether to add `_safe` suffix methods that return errors to give people the option, but for now our library only throws errors to consumers. You might ask "well what's the point if you still throw errors to consumers?" We feel that forcing our engineers to deal with errors where they can happen is worthwhile. Our team writes a lot of Go and we love how it forces you to think about every spot that can error, so we wanted the same experience in Python
        • dmurray 3y ago
          OK, in that case I think it's a very reasonable choice! Perhaps it's worth clarifying that in the article. I might express a slight preference that your internal code is idiomatic if I'm going to be poking around in there at some point. But I wouldn't be bikeshedding about your C coding style if you'd provided a Python module written in C, so I don't feel I should have a strong opinion as a user here.
• ed_blackburn 3y ago
  I'm six months into my Python journey. We aren't building a library, so everything runs on 3.11. Having spent most of my career in statically typed and sometimes functional languages, I've found the result package approach and pattern-matching suggestion work well. There's been a suggestion it's not very Pythonic, but I'm willing to continue using a result monad because the trade-off is one-sided; it comfortably pays for itself.
  • all2 3y ago
    What is a result monad?
    • avgcorrection 3y ago
      It is linguistically related to the (plus, int) monoid; the fact that it conforms to some algebraic/categorical interface is irrelevant in most contexts and just a distraction. (A “Result” is Rust-speak for a sum type which is either A or B, where (say) B is conventionally an error)
    • ed_blackburn 3y ago
      The result package in the article: https://github.com/rustedpy/result https://github.com/rustedpy/result There's a pretty decent explanation in the readme. If you're more interested in Monads, I am not sure I'd cover that in a HN comment very well, but I would encourage you to take a look.
    • itishappy 3y ago
      It just lets you combine results in an intuitive way: Ok(2) + Ok(3) = Ok(5) Ok(2) + Err() = Err() Err() + Ok(3) = Err()
• guitarbill 3y ago
  > Rust returns returns errors using a "wrapper" type called Result. A Result contains both a non-error value (Ok) and an error value (Err) A `Result` can contain either a non-error value (Result::Ok) or and error value (Result::Err), never both.
  • goodoldneon 3y ago
    Oops! I'll make that correction later today. You're right: they're mutually exclusive
    • guitarbill 3y ago
      That's what makes it good :) Well, that and the fact you have to unpack/destructure the result somehow to get at the value (or the error), so you're forced to handle it.
• jmholla 3y ago
  Why does the author want to swallow these exceptions? Let them propagate and it's obvious where the issue lies. If you can't handle an exception, don't catch it. > It's impossible to know which line might throw an error without reading the functions themselves... and the functions those functions call... and the functions those functions call. Some thorough engineers may document thrown errors but documentation is untested and therefore untrustworthy. Java is a little better because it forces you to declare uncaught errors in method signatures. The author's proposal doesn't change this as much as they think it does. You still don't know what type of errors a function will throw without inspecting the code and thus how to resolve them. Unless, you have a blanket switch for every possible error anything could return which is the very thing they are complaining about.
  • goodoldneon 3y ago
    Author here! > Why does the author want to swallow these exceptions? Sometimes you want to swallow exceptions and sometimes you don't. The examples in the article may be a little contrived, but there are situations where logging an error and continuing is better because it prevents data loss. > Let them propagate and it's obvious where the issue lies. If they propagate down the stack then you lose context. You also may not have all the data you need to properly recover (e.g. still write to a table but make the errored field null). > You still don't know what type of errors a function will throw without inspecting the code and thus how to resolve them. You're right that we don't know which error occurred with the approach in the article, but you at least know that there could be an error. This is better than try/catch because that doesn't tell you whether an error could happen
    • mickeyp 3y ago
      > Sometimes you want to swallow exceptions and sometimes you don't. You capture `Exception' in your examples which is a serious anti-pattern in nearly all cases. One of the rare places you'd want to do that is at the top level of a service that has to clean up safely before bailing out. None of the contrivances in your example demonstrate that requirement. If you need to capture an exception, you capture the narrowest one that matches the business/technical requirement you have. The rest should bubble up and get handled by another part of your code or, indeed, cause your app to crash if it's unhandled. Exceptions don't mean "error" either; they're just exceptions. > If they propagate down the stack then you lose context I cannot think of any instances where you 'lose context' by propagating them. You should raise Foo from e if you're re-raising, though, which you clearly did not bother doing. That does preserve context. Passing an instance of an Exception object around? What is this madness... are you building a debugger? No? A complex framework or tool like pytest where you may want to manipulate the stack frames to benefit the end user? No? Then don't pass exception objects around and raise them somewhere else.
• js2 3y ago
  The most important thing about writing code is that you write it idiomatically for the language it is in. With Python, idiomatic code is known as Pythonic. def rename_user(user_id: str, name: str) -> User | Exception: # Consume the function user = get_user(user_id) if isinstance(user, Exception): return user user.name = name return user This is not Pythonic. Don't do it. Like it or not, Python uses exceptions. How do I know that some other code that get_user calls isn't going to raise an exception? What if the program receives an interrupt signal during the get_user call? When I'm writing Python code, I have to think about exception handling, and now when I use this library, I have to add a bunch of isinstance calls too? Perspective: I've written primarily in Python for over 20 years, as well as coded extensively in many other languages (C, Objective-C, JavaScript, Java, Bash) and have familiarity with a bunch more (Go, Ruby, Gradle, TCL, Lua, Kotlin, C++). In practice, exception handling, declared exceptions or not, just isn't that big of a problem. When I'm writing code, I have to reason about it, and exceptions and return/error values are just one aspect of that reasoning.
  • avgcorrection 3y ago
    This topic is explicitly about considering alternatives to the current Pythonic way.
    • ska 3y ago
      And GP is pointing out (correctly) that most of the time being idiomatic has more net gain than any putative advantage of the non-idiomatic proposal. Exceptions to this are extremely rare.
      • deleted 3y ago
        [deleted]
    • dmurray 3y ago
      The topic is explicitly about writing an SDK for the company's product, which is presumably intended to be imported in customers' code and used alongside hundreds of other libraries. As such, writing idiomatic code for the language of the SDK really should be a concern, rather than implementing some idiosyncratic half-baked version of Go or Rust error handling in Python.
• TigerTeamX 3y ago
  I love it. Recently been rewriting Goleko.com backend in Go from Python. The errors as value paradigm is so nice for 90% of the time. It is so much more robust code out of the bat.
  • ARandomerDude 3y ago
    > Goleko.com I won’t claim to know what your intent was but to me the brand placement changed your post from a comment to an ad.
• sethammons 3y ago
  python is where I learned to hate exceptions as control flow - Twisted is twisted. Go was such a breath of fresh air. Now I'm back in python primarily and I am constantly wondering what my functions actually take and actually return. Exceptions are just spooky GOTO and a distance. Our logs are littered with them and have to use Sentry to tell us "oops, you introduced a new error path." Our builds are full of warnings and versioning issues. I've yet to see a decent sized python project that is not a mess.
  • goodoldneon 3y ago
    > Exceptions are just spooky GOTO and a distance I couldn't agree more! When you throw exceptions it's unclear where the control flow will go
    • Karellen 3y ago
      I just don't see how multiple levels of if err: return nil, err in the call stack makes things any clearer? Or how having many instances of that snippet scattered everywhere makes the code easier to read?
      • mmnfrdmcx 3y ago
        Agreed. It's also objectively worse because you lose the stack trace in the final exception you will display to your user. Enjoy debugging that when a customer sends you an error.
        • demi56 3y ago
          Or you could use a 3rd party error package that’s not a good excuse because the standard don’t have it as a priority
      • goodoldneon 3y ago
        It makes the happy path harder to read but the unhappy paths much clearer. I hated this aspect of Go when I first started with the language but I love it now
• rednafi 3y ago
  This doesn’t do much. Many of the stdlib functions will still raise exceptions. Also, just because you’re returning exception from a function doesn’t guarantee that something else won’t raise an error. This isn’t an issue in Go/Rust since errors are treated as values in the core language and you’re forced to treat them so everywhere.
• anentropic 3y ago
  "How we made a non-idiomatic Python SDK for our app that Python devs will hate"
  • neuromanser 3y ago
    Probably true. Without commenting on the particulars of the SDK in question (read: I haven't read TFA): Monadic code is obviously better than the mainstream alternatives. Does this point towards eventual inviability of Python itself?
• Karellen 3y ago
  > So if we want to be really safe then we'll wrap each call with a try/catch: try: thing.set_name("Doodad") except Exception as err: raise Exception(f"failed to set name: {err}") from err > As we think about each possible error we realize that our original logic would crash the program when we didn't want it to! But while this is safe it's also extremely verbose. How is this safer than the original? If the caller wasn't catching exceptions thrown by this function before, it's not catching exceptions thrown by this function now. What is being gained by catching an exception to do nothing but throw another exception? This feels like a strawman.
• deleted 3y ago
  [deleted]
• b3orn 3y ago
  Nope, don't like it. It's also a little disingenuous to switch the example halfway through. Anyway, I fixed your code, with exceptions def get_user(user_id: str) -> User: rows = users.find(user_id=user_id) if not rows: raise Exception("user not found") return rows[0] def rename_user(user_id: str, name: str) -> User: user = get_user(user_id) user.name = name return user, None
• jflakjwoasdlf 3y ago
  > tuple[User | None, Exception | None] tuple[User, None] | tuple[None, Exception]
  • goodoldneon 3y ago
    That type won't help if you're unpacking like `user, err = get_user(user_id)`. Both values will be nullable and the type-checker won't understand that `user` is not None if `err` is None
  • progbits 3y ago
    Yup, don't allow nonsensical states (User+Exception or None+None) to even exist. The original is more like Go approach which is a big flaw with the language.
    • goodoldneon 3y ago
      Go can be `tuple[User | None, Exception | None]` or `tuple[User, Exception | None]`, depending on whether you're returning a pointer. But yea, Go's approach has its warts. Like if you aren't returning a pointer then you need to return the zero value (e.g. `User{}`) even when returning an error
      • progbits 3y ago
        Yeah but pointer + nil also have issues: https://go.dev/doc/faq#nil_error https://go.dev/doc/faq#nil_error I've also seen APIs that allow returning both error and a value ("something went wrong, but here is a fallback or something") which are of course extremely confusing given the usual style.
        • goodoldneon 3y ago
          Oh I agree, I've run into too many nil pointer panics. I wish Go had a first-class way of expressing optionality, rather than using pointers
• kitd 3y ago
  Quite apart from the Python discussion, the author captures why I prefer errors as values (a la Go) to exceptions (a la Java), and I have written both styles for many years. > Regardless of the specific approach, returning errors as values makes us consider all of the places an error could occur. The error scenarios become self-documenting and more thoroughly handled. This is so true. Most Java exception handling is a try/catch around about 20 lines of code, with superficial logging/rethrowing and no context about exactly what was being done to what when the exception occurred, just a filename/line# and a probably cryptic error message. In Go, best practice is something like: bytes, err := os.ReadFile("myfile.json") if err != nil { return nil, fmt.Errorf("reading file %s: %v", "myfile.json", err) } var data map[string]any err = json.Unmarshal(bytes, &data) if err != nil { return fmt.Errorf("unpacking json from file %s: %v", "myfile.json", err) } This gives you precisely targeted errors that tell exactly what you were doing to what. Your future self will thank you when you're desperately trying to work out what went wrong last thing on a Friday. If you are going to replicate this with exceptions, it would require much more boilerplate, as his example demonstrates, which is ironic given that is the charge levelled at Go.
  • theamk 3y ago
    I think it's kinda the opposite actually... with go, you have to go out of the way to generate those errors, while in Python, most of this is automatic. In particular, the ReadFile example in Python will raise OSError, and those already include filename and error message. So you'd get the same result with 0 extra lines. For the second example, the json unmarshalling will not auto-add filename, but its easy enough to do using exception chaining: try: data = json.loads(bytes) process_1(data) process_2(data) except: raise Exception(f"Error while parsing file {filename}") which will give stacktrace like: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "somefile.py", line 4, in somefile Exception: Failed while parsing file.json Note that's even more detailed than go, with significantly less boilerplate (only 3 lines per function) vs 3 lines per call. (an anecdote: we've had the team which converted the CLI tool from Python to Golang. The first time they ran the tool, it printed a single line: invalid character '"' after top-level value and that's it. It took a lot of debugging before they could figure out what happened. All because they got used to python doing super-rich traceback automatically, with 0 effort from programmers)
• theamk 3y ago
  I don't understand why people insist that all errors must be handled all the time. It could be my C++ background, but I feel there are two very different errors: Expected errors - like "user not found" - should use a value instead of exception. In Python, you can use sentinel objects, or tuples, or None.. lots of options really. Occasionally there is a good reason to use exceptions for flow control even for know errors (various hooks come to mind), but this should be pretty rare compared to number of places that can raise an unexpected errors. Unexpected errors should not be caught at all, except maybe at the very top level (to record them and return 500 to user). The examples in post, where you catch exception and re-raise Exception back are terrible - what's the point of them? There is no extra clarity, just verbosity. I would defect them in any code review. Coarse-grained error handling is great as long as exceptions are meaningful and stack traces are good, which is the usual case in python. All that matters for unexpected errors is that (1) user sees an error message and (2) the real cause is recorded for later analysis. A single top-level try block does both.
  • danpla 3y ago
    > Expected errors - like "user not found" - should use a value instead of exception. In Python, you can use sentinel objects, or tuples, or None.. lots of options really. That's why exceptions are called exceptions, not errors. If a routine called openFile() can't open a file, that's a pretty exceptional situation, and it's up to the caller to decide whether the exact reason is an error in their case. The exception object is already a value that can have not only a message text, but also any data members, so why reinvent the wheel with sentinel objects, tuples, etc.? > The examples in post, where you catch exception and re-raise Exception back are terrible - what's the point of them? There is no extra clarity, just verbosity. I would defect them in any code review. Typically, you re-raise an exception after adding some content to it. This may be less important in Python, which gives you a great stack trace, but in a language like C++, the lack of context information makes the exception basically useless.
• mmnfrdmcx 3y ago
  I don't agree with this post. Engineering is about tradeoffs. There is more advantage in doing the accepted python solution (exceptions) than inventing your own (which you claim to be better, but I personally think is worse). If you are developing in a team, stick to established conventions and spend your time focusing on your business problem.
• txbuck 3y ago
  Not necessarily advocating for unidiomatic python/code, but you could use a decorator to automatically wrap the function call with a try-catch and package the return value appropriately. Lot less mangling of function bodies that way, just return and raise exceptions like normal. You *could* specify the expected Exception type, but considering the rest of the ecosystem probably won’t be following along with documenting expected exceptions, I assume it wouldn’t be worth it and would be more straightforward that all the exception types in the signatures be the plain vanilla Exception. Would also be super inefficient pre-3.11 but ¯\_(ツ)_/¯
• demi56 3y ago
  Let python be python, let Go be Go, let Rust be Rust it has never worked well for any language trying to be like another language
• diarrhea 3y ago
  As for helping the type checker, try typing’s TypeGuard. They help the type checker reason about situations such as “if this is None, this other thing is that”.