22 ms·
Python 3 Types in the Wild: A Tale of Two Type Systems [pdf]
- satya71 6y agoHmm, I mostly use Pylance/Pyright because of tight integration with VS Code. Wonder why they chose PyType, which I believe is not very widely used.
- mcstafford 6y agoGoogle's pytype seems to be better deployed within the standard PyPi ecosphere. The Microsoft packages you're using aren't.
- Mehdi2277 6y agoThat’s because pyright was written in typescript so it makes sense to use npm to install it. Pyright has weak ide support beyond visual studio stuff. I’m pretty fond of pyright due to how responsive the developer is on GitHub. I’ve made multiple issues before and gotten an answer same day (sometimes same hour) and some small bugs I’ve reported got fixed in the next release (which happens like twice a week).
- IceWreck 6y agoPylance/Pyright are proprietary and don't even work on open source builds of VScode (theyre hardcoded to work only on MS builds of VSCode), let alone any other IDE.
- Spiritus 6y agoPyright is open source (MIT)
- Evan-Purkhiser 6y agoPyright works well for me in vim-coc. Of course ymmv
- nyanpasu64 6y agoFrom what I can tell, Pylance is proprietary and intended to "extend/extinguish" the open-source VSCode codebase with Microsoft's proprietary VSCode binaries, but Pyright is open-source.
- satya71 6y agoHmm, I hadn't checked the license. Pylance is indeed proprietary.
- IshKebab 6y agoI was wondering the same thing. I've never even heard of PyType, but I use Pyright every day. I expect many developers are similar. Does Pyright behave like MyPy or PyType? Or neither? Also did Python really create a type annotation system without specifying the semantics of how it is supposed to work? That's crazy.
- Mehdi2277 6y agoThe type system is specified but the main pep people point to is the base type system. There have been multiple extension peps adding new type features and different type checkers have different progress implementing them. Also different type checkers have different bonus features. Mypy comes with plug-in support to make it easier to extend the system for very dynamic custom situations. Pyright is able to generate rough guess type stubs for untyped codebases. Pyre has the best tensor related type support likely motivated by Facebook having pytorch. Pytype I know the least about but it likely has some unique features motivated by google usage. If the type checkers disagree on stuff defined in pep 484 that’s a bug to report. Beyond that they can vary but feel mostly similar.
- nonameiguess 6y agoVery good discussion from my wife's alma mater! One thing I didn't see mentioned that is a particularly annoying false positive in mypy is the enforcement of Liskov Substitution for equality checks. All types have the common bottom type of object and object itself defines the __eq__ magic method as accepting any other object, which means all custom Python classes that want to define an equality check have to either define the '==' operator as valid accepting anything as a second operand, or tell mypy to ignore the error, which is the obviously saner thing to do since you want a type error, not a return of False, when you check equality against a completely unrelated type. Unfortunately, mypy provides no way to give you that statically. I'm glad they mentioned the default parameter None to avoid the classic footgun of empty collections, though. That has to be the single most annoying false positive, when you declare a parameter as Optional and the type checker then complains it was set to None in the parameter definition but later set to an actual value, which you have to do because of the totally insane and unintuitive fact that all invocations of a Python function share the same parameter, so filling in the empty list with something else upon calling it means everything else that calls it now gets the changed default value instead of the empty list. This is probably the single greatest rookie mistake made in Python because of how unexpected it is if you don't deeply understand the object model, but doing the literally recommended by the book solution to get around it results in the official type checker endorsed by Guido himself to report a type error.
- karlicoss 6y agoIt's possible to work around without forcing wrong type on parameter def f(l: List[int]=cast(Any, None)) -> int: if l is None: l = [] ... f() Using casts is useful for all sorts of things like this when sentinel values/objects are involved. People often feel it's somehow 'wrong', but it's fine to trick the type checker as long as it's localized to neighbouring lines of code :)
- nonameiguess 6y agoYou learn something every day! I didn't even know about the cast function.
- 6y ago
- a_cool_username 6y ago>A key question arises: why are so few repositories type-correct? The authors don't seem to ever discuss the fact that mypy version changes frequently make previously-passing code fail type checks. I don't think I have ever upgraded mypy and not found new errors. Usually they're correct, sometimes incorrect, but it's a fact of being a mypy user. Between mypy itself and typeshed changes, most mypy upgrades are going to involve corresponding code changes. The larger your code base and the more complicated your types are, the worse it'll be, but it's basically an ever-present issue for any program interesting enough to really benefit from a type checker. How many of those repositories were "type-correct" but only on particular versions of mypy? I bet it's a lot!
- nerdponx 6y agoI don't have this experience. Can you give an example of a previously-valid codebase that failed typechecking unexpectedly on a recent MyPy update, that wasn't a result of a false negative bug in MyPy?
- dharmab 6y agoI obviously can't share it here, but my primary codebase at my job needs a few dozen changes every time we change mypy versions. There's a few places of false positives where I've had to comment `# shut up, mypy` and a `type: ignore`. Usually when I'm being clever with the subprocess module.
- emptysea 6y agoI wonder if that's a mypy issue or more that the typeshed types are bugged, since type shed versions also get shipped (used to?) with new type checker versions. https://github.com/python/typeshed https://github.com/python/typeshed
- a_cool_username 6y agoI can't give you any links because it's not open source code, but there was a bug fix in 0.790 named "Don't simplify away Any when joining union types" that caused me some problems with bad annotations in the existing code: The annotations implied that Any was possible, but it wasn't, but it got dropped from the final Union by the bug so we never had to handle Any. Dataclasses have had some backwards-incompatible improvements as well. But the big culprit is typeshed. Something will get new/fixed annotations and suddenly you aren't handling all possible return types in your callers, or whatever.
- zackees 6y ago“At the same time, developers still see value in writing type annotations as they serve as documentation.” Also useful for autocomplete. For me, most of the value of typing is at the function level: what does this function accept as input and what does it produce as output? Also mypy gets in the way of using loop constructs, as the iterating variables have to be predeclared with typing before the loop, breaking encapsulation.
- karlicoss 6y agoHmm, what do you mean? If you type the thing you're iterating over, the type of variable will be inferred?
- emptysea 6y agoI'm curious what you mean by "iterating variables have to be predeclared". In my experience mypy's type inference works with bog standard python constructs like a for loop that appends into a list that gets declared before the loop, aka: output = [] for x in range(1): output.append(x) reveal_type(output) # note: Revealed type is 'builtins.list[builtins.int*]'
- zackees 6y agomypy will complain that x is type Any when checking assignments. To silence the warning one has to do: x: int for x in range(1): ...
- codethief 6y agoMaybe they're referring to the fact that if you do `for x in y` and y isn't properly annotated, then you will need to write x: MyType for x in y: ... for x to be annotated properly.
- xvilka 6y agoSomeone who needs types could just write in a strongly-typed language.
- seanmclo 6y agoUnfortunately, this isn’t always an option if you need to use a library that’s only available in Python.
- xvilka 6y agoIt's possible to use it in a Julia-like fashion (like PyCall.jl[1]). For example, with OCaml one could use Pyml[2]. [1] https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl [2] https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml
- dr_zoidberg 6y agoPython is strongly typed. It's also dynamically typed. They are not contradictory between each other.
- solumos 6y agoI once had someone try to tell me that python was "untyped" - I guess people assume a lot when they come from a first-class statically-typed language.
- kstrauser 6y agoIn Python, objects have types. Variables are best thought of as void* pointers to objects.
- dr_zoidberg 6y agoI've found over the years that people have an easier time understanding how Python manages variables when you tell them that, rather than "in-memory boxes with data" as is the case in other languages, Python variables are more like "name tags that you attach to an object".
- travisjungroth 6y agoNeat stuff. I've become a huge Python types fan. Even on small personal things, my flow is to write the empty classes, add attributes, add method signatures with hints, then fill in the methods. Leads to a lot less rewriting. Writing the hints doesn't take long at all. What does take a long time is learning about it all. There's the docs, then PEP 484, then at least eight other related PEPs of various impact. It would be a good topic for a small book / website / course, except things change kinda fast. I do think it could be a hack for a new dev to establish credibility. Spend a few months type hinting code and fixing bugs in popular open source repos. You'll learn way more about real software engineering than following another tutorial. If someone wants to do that and wants some mentoring, hit me up at my last name at gmail.
- exdsq 6y agoThis is, in my opinion, the best way to design software. If you’d like to see what happens when you take it further with a more powerful type system, read type-driven development in Idris or any of the F# type-driven development blog posts.
- travisjungroth 6y agoI will upvote anyone who says my way of designing software is the best. And thanks for the suggestions.
- gandalfgeek 6y agoIf you're too lazy to read the paper, here's a short explainer video on the paper: https://youtu.be/_o9CgLjo9Kk https://youtu.be/_o9CgLjo9Kk
- matanrubin 6y agoThe article mentions that MyPy and PyType use two different type system models and therefore may report different errors. Does it make sense to use both of them on a large codebase? I've been working with MyPy for a while and it has gotten pretty good and fast. Assuming PyType is a valuable tool, I think I'll try adding it to our CI pipeline and see if we get new typing errors.
- guram11 6y agocurious what do you do when each type systems agree and disagree with the other?
- Mehdi2277 6y agoYou determine which is right and report a bug in the other. After that you add a type ignore statement to keep the wrong one happy until the issue is resolved. My current workplace I have two type checkers enabled (mypy/pyright) and that’s our way of dealing with it.
- arjunkava 6y agoThis is something informative but python is the list of bad type safe.
- arjunkava 6y agoMay be I've written something that really does not make any sense. I should have understand the blog in the first place.
- carlosf 6y agoI gave up on using mypy as I was spending more time telling it to shut up than actually using it catch mistakes ahead of the time. That said using type annotations made me realize I want to be fluent in a language with a strong static type system (huge productiveness booster imo), so I'm currently studying Typescript and nim.
- ZephyrBlu 6y agoI wish that you could use typed Python to speed up the execution of the program instead of them just being hints.
- m_mueller 6y agoafaik that was first planned for Python 4 and then dropped again :(
- ZephyrBlu 6y agoAny links on it being planned for Python 4? I didn't know they'd ever even considered it.
- m_mueller 6y agoI think that the mentions of that have been deleted from the relevant PEP since, but e.g. https://www.reddit.com/r/Python/comments/7zvyhx/pep_563_mentions_python_40_whats_going_on/ https://www.reddit.com/r/Python/comments/7zvyhx/pep_563_ment... is also asking about it.
- isoprophlex 6y agoNot exactly what you're talking about, but... For some types of functions, Numba's JIT offers some pretty nice speed ups. It can infer function types (or you supply them yourself) and compiles to optimized machine code
- chriswarbo 6y agoIt's tricky, since only the typed part can be sped up, and there will usually be lots of back-and-forth between typed and untyped code; which requires conversions and checks. For comparison, Racket has gradual typing which gives a speed boost when everything is typed, but can end up much slower if anything remains untyped, e.g. see figure 3 in http://www.ccis.northeastern.edu/home/types/publications/gradual-dead/pre-treatment.pdf http://www.ccis.northeastern.edu/home/types/publications/gra... Python would probably be even worse, since it's "more dynamic" than Racket; e.g. there are many hooks that can run arbitrary code (like dunder methods), and there's little encapsulation so values may get swapped out via monkey-patching at any time.
- slifin 6y agoI mean it seems obvious but most/all? modern type systems are not portable and don't compose with other type systems Splitting out the mistake checking and the data specification aspects of types would probably make pieces more portable across different languages
- PuercoPop 6y agoVivek Haldar did an overview of the paper this week: https://www.youtube.com/watch?v=_o9CgLjo9Kk https://www.youtube.com/watch?v=_o9CgLjo9Kk