12 ms·
Statically-typed error handling in Python using Mypy
- roseway4 7y agoHaving recently found myself switching between Python and Scala, I've come to value IntelliJ's type hinting and red squiggly prompts, not to mention compile-time errors. While mypy and type hinting support in PyCharm and VSCode is great, it's not the seamless experience as with typed languages and the lack of (optional) runtime typing still allows all sorts of bad things to happen. A "safe" mode for Python, where typing and perhaps some other stuff is checked at run-time, might be an interesting advancement for those in the community building larger or mission-critical systems.
- astrea 7y agoMy opinion is that python shouldn't be used for mission- critical systems anyway.
- traverseda 7y agoAside from static types, why?
- jraph 7y agoOne thing I dislike in Java is the string comparison. If you write str1 == str2, it compiles but probably does not do what you want (object identity vs string comparison). (edit: to clarify, this does not mean Java is not a good language overall, every language has its own set of shortcomings and as someone points out in the replies, this behavior is consistent with everything else in the language)
- weberc2 7y agoIt’s been a while since I used Java, but what on earth does that string comparison do if not comparing the strings? Is it an address-only comparison?
- dpratt 7y agoYes, and this is entirely consistent with the overall design of the language. The ‘==‘ does not have a special overload for String instances, which are reference types just like any other in the JVM. The equals operator does a reference comparison, just as it should. If you want to do a logical comparison of instances, you use the equals() method just as you would for anything else. Frankly, if this is the most meaningful complaint you can come up with, it might be indicative that you haven’t learned enough about it to have an informed opinion.
- CamouflagedKiwi 7y agoIt isn't at all consistent, because integer types compare correctly with == and do not have a .equals() method. And regardless of consistency, if it's a mistake people make frequently with the language and need to learn not to do the obvious thing, it's probably not an ideal bit of design.
- dpratt 7y agoIn java (and every other C derived language that I know of) there’s a hard and explicit distinction between reference and primitive types. In fact, from a certain perspective the two flavors get treated identically. Comparing two integers with ‘==‘ will be true if the contents of the variables are the same, and comparing two references will be true if the pointers they contain are the same value. It just so happens that for primitives identity and logical equality are the same thing.
- weberc2 7y agoC, C++, Rust, Go, etc have no special distinction between reference types and primitive types but rather everything is a value and some values are pointers/references. In a language with operator overloading, I don’t see why you’d ever want to check string identity by default. Interestingly, Go doesn’t have operator overloading but it treats strings as a special case including doing a string comparison for ==. Not saying this is better or worse, although no one makes mistakes with string comparison as far as I’m aware, for whatever that is worth.
- dfee 7y agoI’m on the other side; having gotten deeper into languages like c# and java; I think those technologies are vastly over used, and great languages like Python and Typescript are under used.
- weberc2 7y agoHow do you figure? If it’s mission critical, it means failure is unacceptable. Python and other dynamic languages (TS is a middle ground here) excel when failure isn’t so bad and making and deploying changes is fast/easy/cheap. This is exactly the wrong tradeoff for mission critical systems.
- igor47 7y agoWhy do dynamic languages excel when failure is not so bad? What makes non-dynamic (compiled?) languages better when failure is bad? I'm parsing your comment to mean, "it's easier to write correct code in compiled languages", but this is not obvious to me, or anyone who's, for instance, written any C at all.
- Reelin 7y agoC is the wrong comparison to make here - it's extremely low level. Java or perhaps C++ would be much closer. * Ahead of time compiled languages reduce flexibility but allow the compiler to do more reasoning about the system. * Statically typed languages allow the compiler to reason about the assignments you make and methods you call. If the language is also AOT compiled, you avoid crashing at runtime. * Dynamically typed languages significantly reduce boilerplate and avoid the mental effort of expressing an idea in a rigorously typed manner. It's the "hold my beer" approach.
- pytester 7y agoC is a good example precisely because it highlights the fact that explicit typecasts and memory safety are a LOT better at preventing bugs than static types.
- 7y ago
- 1e10 7y agoWhy? Seems that it is a well proven language. Just like php, asp.net, and other languages people like to hate on. Or rather, how would you define mission critical?
- jtdev 7y agoCare to elaborate? Which languages would you use for mission-critical systems? What specifically about Python is not conducive to such systems? You realize that many mission-critical workloads run on Python with great success, right?
- deleted 7y ago[deleted]
- senko 7y agoThat depends on the mission, wouldn't you say?
- staticassertion 7y agoPeople are gonna dogpile you, but if you'd said this about JS they wouldn't have blinked.
- pytester 7y agoCoz there's a difference between "dynamic type system" and "type system that's thrown together with bits of string and rubber bands": https://www.destroyallsoftware.com/talks/wat https://www.destroyallsoftware.com/talks/wat
- staticassertion 7y agoPython has is own 'wat' stuff, especially Python 2. https://stackoverflow.com/questions/3270680/how-does-python-2-compare-string-and-int-why-do-lists-compare-as-greater-than-n https://stackoverflow.com/questions/3270680/how-does-python-... "CPython implementation detail: Objects of different types except numbers are ordered by their type names; objects of the same types that don’t support proper comparison are ordered by their address."
- pytester 7y agoNone of those surprising behaviors exist in python 3. I wouldn't say "100" < "2" really counts, since, alphabetically it makes sense (an equivalent would be "baa" < "c").
- staticassertion 7y agoYes, Python2 was called out explicitly. Reference equality is definitely 'wat' shit IMO, though it's really mainstream too.
- detaro 7y ago> objects of different types always compare unequal, and are ordered consistently but arbitrarily Seems like a completely reasonable thing to do?
- HALtheWise 7y agoWould you rather build safety-critical systems in Python (type-unsafe but memory-safe) or C++(type-safe but memory-unsafe)? While static typing does catch code errors, in my experience they are mostly errors that would be caught somewhere else in the testing process. Memory safety issues are rarer, but also have a penchant for showing up unexpectedly and for crashing the entire system in unrecoverable ways. Obviously, the best would be to have both kinds of safety, but people program safety critical systems in C++ all the time, and deal with the consequences. At least you can catch a TypeError in Python and handle it.
- ailideex 7y agoI would rather build it in a type-safe and memory-safe language - but if that is not allowed I would go for C++ even though I have my issues with it. Catching TypeError in python and handling it is not a substitute for static typing.
- Riverheart 7y agoAppreciate how in depth this is, looking at all the possible ways of handling the issue. Makes the final result more interesting knowing the journey it took to get there.
- karlicoss 7y agoHi, I'm the author! Happy to answer your questions here!
- dbrgn 7y agoHi, thanks for featuring my result library! Regarding your criticism that `result.ok()` returns `None` if the value is not an `Ok` type: That's what `result.unwrap()` and `result.expect(msg)` are for :) (There's no unwrap_err and expect_err so far, but PRs are welcome.) The lib is strongly inspired by Rust (see https://doc.rust-lang.org/std/result/enum.Result.html https://doc.rust-lang.org/std/result/enum.Result.html), that's why `result.ok()` returns something option-ish. I like your `isinstance` approach btw! Will have to think about it a bit more. Reminds me a bit of Typescript as well. Does mypy have type guards (https://www.typescriptlang.org/docs/handbook/advanced-types.html#type-guards-and-differentiating-types https://www.typescriptlang.org/docs/handbook/advanced-types....)? Maybe that could be used to create helper functions that check for success/failure and which help mypy to derive the correct type.
- uryga 7y ago> Does mypy have type guards? not yet, maybe in the future. might be possible to emulate some use cases with Literal Types. tracking issue: https://github.com/python/mypy/issues/5206 https://github.com/python/mypy/issues/5206
- karlicoss 7y agoOh, not sure how I missed these methods, thanks! I'll update the post to reflect it.
- Agathos 7y agoThanks for writing it! I'm a Python programmer in my day job. I have sometimes remarked that the reason I play around with Haskell in my free time is because I don't want to see another run time type error unless I'm being paid to do so. But I needed to do some spreadsheet-scraping and data-munging for a personal project and I figured it would be easier in Python. So I searched and found your article and it was just what I needed. Now to sneak it in at work...
- staticassertion 7y agoHere's my opinion, having written many thousands of lines of mypy code. * Third party libraries are still not typed, which sucks * Inference is weak. Sometimes an 'if' statement narrows accurately, sometimes it doesn't. * Generics are extremely confusing, moreso than any other typed language I have used. Any Generics are even more confusing. * Fundamentally, Python is structured such that the concept of interfaces are coupled tightly to implementation of classes, which is severely limiting. As an example, a module A can declare a class, a module B can declare an interface (Protocol), but module B can not implement that interface for A (without modifying A's code). * Errors from mypy are still very very light. There are very few "Try XYZ to fix it", and it's just a single line of read text that amounts to an assertion. * Still weak support for JSON (use Dict[str, Any]) * Slowly improving but still not amazing support for Self types I could go on. Mypy is awesome but it still feels very immature, and as it should since it's pre 1.0. It does not come close to matching the experience of other languages with static types.
- karlicoss 7y ago* Third party libraries: I sometimes see them annotated, but what people don't suspect, is that you need to include 'py.typed' file with your package in order for it to be discoverable. Perhaps that's something setuptools could warn the developer about.. * Regarding errors: there are few quite recent flags: ' --pretty', '-show-error-context' and '--show-error-codes', they make it a bit more pleasant. You can see me using them in this section: https://beepb00p.xyz/mypy-error-handling.html#container https://beepb00p.xyz/mypy-error-handling.html#container
- staticassertion 7y agoThe 'py.typed' thing is extremely unintuitive, I package up python libs and I'm still not sure I've done it right. Thank you for letting me know about those flags, I had not heard of them before!
- Caligatio 7y ago* Still weak support for JSON (use Dict[str, Any]) Python 3.8 added TypedDict which finally helps this particular shortcoming.
- foxes 7y agoI think in the long run, python, ruby, js, etc are all bad ideas. You are solving high level problems with these languages, correctness should be the main priority. Realising that type checking is a good thing, then bolting it on, I feel is a crappy solution, when you should just use something designed better from the start. I completely disagree that there is a "productivity toll". If you dont know how to speak a language then obviously it seems difficult. I think you save time in the long run not dealing with all the type errors.
- jtdev 7y ago>> “ I think you save time in the long run not dealing with all the type errors.” Have you written and deployed production code in the languages you mention above? Did you encounter a substantial number of type errors?
- staticassertion 7y agoThere is no good definition of a type error. It certainly goes beyond TypeError(Exception). Is a '.close()' method being called twice a type error? It is in a language that can express states in the type system. Is SQL injection a type error? It is when you use refined types in your sql library interface. The vast, vast majority of errors I run into are errors that, with effort and the right type system, I could turn into type errors. The point being that asking "would those be type errors?" is a really big question that is, when answered simply, "yes".
- mixmastamyk 7y agoThere aren't any mainstream languages that have the ability to detect the ultra high-level errors you describe. Maybe typescript is getting better. Pyflakes and unit tests will detect the great majority of potential errors. Mypy gets you closer to zero.
- staticassertion 7y agoRust can express state machines just fine, as well as refinement types (Python can do refinement types too), but yes, I agree that there's room for languages to build more ergonomic but advanced type systems.
- macca321 7y agoI maintain a library for doing exactly this in C# (https://github.com/mcintyre321/OneOf https://github.com/mcintyre321/OneOf). I tried to replicate it in Python as I'm writing it at the moment, but couldn't quite manage it. Can you have a multi generic typed Union in python?
- Recursing 7y agoDo you mean something like this? T1 = TypeVar("T1") T2 = TypeVar("T2") GenericUnion = Union[T1, T2]
- macca321 7y agoYes I can't quite remember why I couldn't get it to work. I want to make something where this is valid, but couldn't manage it: def foo() -> OneOf[Bar, Baz]: if blah(): return Bar() return Baz() def bar_to_blah(bar:Bar) -> Blah : ... def baz_to_blah(baz:Baz) -> Blah : ... one_of_bar_or_baz = foo() blah = one_of_bar_or_baz.match(bar_to_blah, baz_to_blah) but the critical thing is to get a type error if you add another case to foo's return type, or change the generics to different types.
- sandGorgon 7y agoHas anyone evaluated libraries like Pydantic ( https://pydantic-docs.helpmanual.io/ https://pydantic-docs.helpmanual.io/) and Encode Typesystem (https://www.encode.io/typesystem/ https://www.encode.io/typesystem/) versus MyPy as a real life thing ?
- fermigier 7y agoThese are completely different objects: data validation libraries. I think naming the latter "Encode Typesystem" is misleading (and your question is an indication that I am right).
- sandGorgon 7y agowhy is it misleading ? didnt understand. i just didnt want people mistaking "typesystem" as a word rather than a project name by Encode. its a very common name.
- ghostwriter 7y agoThere's also Typeit (https://typeit.readthedocs.io/en/latest/ https://typeit.readthedocs.io/en/latest/)