9 ms·
A Failed Experiment with Python Type Annotations
- Deimorz 7y agoThe first issue is fixed by just using a string. Yes, you can't do Sequence[Node], but you can do Sequence["Node"]. The docs talk about this here ("class name forward references"): https://mypy.readthedocs.io/en/stable/kinds_of_types.html#class-name-forward-references https://mypy.readthedocs.io/en/stable/kinds_of_types.html#cl...
- mortoray 7y agoGreat!
- Deimorz 7y agoThe second issue is a bit of a judgment call too, I think. If the types were inferred you could end up with errors that you're not able to see because inferred types end up compatible. Personally, I think I'd rather write them manually so that I'm validating whether the method actually matches what it was intended to return. I can't find a more authoritative source right now, but I also remember seeing the same thing that this SO answer says - that it was a deliberate choice not to do this so that people could incrementally add type annotations: https://stackoverflow.com/a/38775381 https://stackoverflow.com/a/38775381 There are also some separate tools that can infer and generate the annotations for you. I haven't tried them personally, but the mypy docs recommends MonkeyType: https://mypy.readthedocs.io/en/stable/existing_code.html#automate-annotation-of-legacy-code https://mypy.readthedocs.io/en/stable/existing_code.html#aut...
- rcfox 7y agoFor Python 3.7, you can do: from __future__ import annotations This will defer the parsing of annotations until after the file has been parsed, allowing you to use the actual symbol instead of a string.
- anentropic 7y agoTIL!
- BurningFrog 7y agoOK, that works, but it does seem dumb, or even deliberately obtuse. The language could just as well recognize Node as it recognizes 'Node', but instead it given as a burden for the human programmer to handle. Is it maybe because mypy is not really part of the Python language, and can't really make changes to it?
- shakna 7y agoThe language will recognise it in a future version by default, or a from future import can be used now. mypy isn't behind (here), it's ahead.
- _ZeD_ 7y agoThis is not mypy, it's "python-the-language": the annotations are values, and, in this case, the method annotations are evaluated when encountered - which is before the end of the class. So you are trying to use a value still not defined. It's like doing something like def foo() -> Bar: ... class Bar: ... obviously the python interpreter cannot find a "Bar" definition during the parsing of the foo function
- BurningFrog 7y agoYou're saying this is how the language works. I'm saying the language could work differently. The case of referring to a class inside its definition is quite different from your "Bar" case. The thing referred to is already declared, it's just not fully defined yet. There is no logical reason I can see making it impossible for Python to honor the obvious intention of the programmer here. It seems it has just chosen not to do so. Though I'll admit I haven't thought through every weird corner case :)
- Deimorz 7y agoPEP 563 is intended to resolve it: https://www.python.org/dev/peps/pep-0563/ https://www.python.org/dev/peps/pep-0563/ As rcfox said, you can already enable this behavior, and it will be included in Python 4.0.
- j-e-k 7y agoI would have to disagree with this post (first point is actually inaccurate and the second I could go either way on). Type annotations/mypy, especially when coupled with dataclasses/pydantic has been very helpful in maintaining a rather substantial 3.7 codebase.
- trimbo 7y ago> rather substantial 3.7 codebase What size codebase is substantial?
- andrelaszlo 7y agoA rather sizable one.
- j-e-k 7y agoExcluding tests/patches to third party libs, around 65K LOC
- michaelanckaert 7y agoI agree. I'm currently working on a project containing 45k lines of Python code and we have really benefited from having type annotations.
- ggregoire 7y agoWhy would you keep Python if you want static typing? I mean, there are several modern, performant (more than Python actually) statically typed languages. Rust, Go, Kotlin, Scala, Swift… All of them are mature and have a good library ecosystem. So, except if you need some math/AI/ML packages available only for Python, why bother? Choose the right tool for the job, no?
- dragonwriter 7y ago> Seems like all of this (adding static typing on top of Python, PHP, etc) started with the success of TypeScript Mypy is a few years older than TypeScript, so, no.
- ggregoire 7y agoYep, already removed that part since I was not sure about the historic and it actually doesn't matter who started the trend.
- rat9988 7y ago> Choose the right tool for the job, no? The thing is, static typing is never the job by itself.
- ggregoire 7y ago"job" doesn't refer to static typing but the project you are working on. - script of 2,000 LOC to clean data => dynamic typing is fine => Python - web service of 1,000,000 LOC => static typing seems a better choice => Rust, Go, Kotlin, Scala, Swift, C#, Java…
- cwyers 7y agoA lot of people don't set out to write a million LOC web service, they build a smaller web service and it grows.
- 7y ago
- dragonwriter 7y ago> Or do you have any idea what type get_closure returns? It doesn't. Because of the syntax error.
- joshuamorton 7y ago> mypy has no trouble understanding this, but it’s unfortunately not valid Python code. You can’t refer to Node within the Node class. No, the workaround is to stringify "Node". class Node(Object): def add_sub(self, sub: 'Node'): ... def get_subs(self) -> Sequence['Node']: #Or maybe 'Sequence[Node]' works just fine. As of python3.7, this stringification is done by automatically under a from future import, and will eventually become the default, so the original code will be valid. > This complexity helped drive the introduction of the auto keyword to C++. There are many situations where writing the type information isn’t workable. This is especially true when dealing with parametric container classes, This is absolutely wrong! You cannot annotate a function as returning `auto` in C++. `auto` and its variants in other languages are useful for eliding redundant type declarations. Especially long ones that use generics/templates. `auto foo = MyContainer<Tuple<int, str>>();` or whatever is nice than having to double write the type declaration. But, in C++ or java, if you write a function that returns a Mycontainer<Tuple<int, str>>, you have to write that in the function. This is an intentional choice: function declarations are your apis, and explicit and clear APIs are useful for human readers of your code. That's why you should annotate the public apis[0] even when you can have them be type inferred. Speaking of which, if you want type inference, check out pytype[1], it's like MyPy, but does do type inference on unannotated code. But you should still annotate your public apis. It serves as a sanity check that you aren't accidentally returning something you don't expect. And of course, being familiar with the differences between iterables, iterators, sequences, containers, etc. is not a bad idea. [0]: https://google.github.io/styleguide/pyguide.html#3191-general-rules https://google.github.io/styleguide/pyguide.html#3191-genera... [1]: https://github.com/google/pytype https://github.com/google/pytype
- dragonwriter 7y ago> But you should still annotate your public apis. The good thing about inference (especially with a REPL) is you can write it without the annotation, and then use the inferred type (in Haskell, I usually find that when I resist the temptation to explicitly annotate types, the actual types are more general than I would have specified.)
- 7y ago
- neuland 7y agoOther people have mentioned that the self-referential case from this post is doable. But what is not doable (to my knowledge) is this kind of self-reference: class Node(typing.NamedTuple): # or dataclass child: 'Node' The error is example.py:4: error: Recursive types not fully supported yet, nested types replaced with "Any". This error has been in there for several years at least. And it always bites me when I forget. It's particularly cryptic when the self-reference is many levels deep. I've even had the error message break in these cases where it prints out a line number from the wrong file. So, then you have to hunt to see where the self-reference is hiding.
- rcfox 7y agoJust tried this myself, and it seems to only be an issue for typing.NamedTuple. Though, it still manages to do type checking correctly... from __future__ import annotations from typing import Optional, NamedTuple from dataclasses import dataclass class Foo: def __init__(self, foo: Foo = None) -> None: self.child: Optional[Foo] = foo Foo(Foo(Foo(None))) # Works Foo(Foo(Foo(1))) # error: Argument 1 to "Foo" has incompatible type "int"; expected "Optional[Foo]" @dataclass class Bar: child: Optional[Bar] Bar(Bar(Bar(None))) # Works Bar(Bar(Bar('bar'))) # error: Argument 1 to "Bar" has incompatible type "str"; expected "Optional[Bar]" class Hep(NamedTuple): # error: Recursive types not fully supported yet, nested types replaced with "Any" child: Optional[Hep] Hep(Hep(Hep(None))) # Works Hep(Hep(Hep({'a': 'b'}))) # error: Argument 1 to "Hep" has incompatible type "Dict[str, str]"; expected "Optional[Hep]"
- lozenge 7y agoIt looks like the constructor is being typechecked, but probably not attribute access. `val.child.junk` will pass because `val.child` is `Any` and you can do any-thing to an Any.
- rcfox 7y agoThat seems to work too.. a = Foo(Foo(Foo(None))) b = Bar(Bar(Bar(None))) c = Hep(Hep(Hep(None))) a.child.junk = 1 # error: Item "Foo" of "Optional[Foo]" has no attribute "junk" # error: Item "None" of "Optional[Foo]" has no attribute "junk" b.child.junk = 'a' # error: Item "Bar" of "Optional[Bar]" has no attribute "junk" # error: Item "None" of "Optional[Bar]" has no attribute "junk" c.child.junk = 1.1 # error: Item "Hep" of "Optional[Hep]" has no attribute "junk" # error: Item "None" of "Optional[Hep]" has no attribute "junk" a.child = 1 # error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Foo]") b.child = 1 # error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Bar]") c.child = 1 # error: Property "child" defined in "Hep" is read-only # error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Hep]") I'm using mypy 0.701 with Python 3.7.3.
- rcfox 7y agoI recently added type annotations to my Python 3.7 project and found it very helpful. Just the process of adding the annotations caught a handful of bugs! Some things I found useful: 1. from __future__ import annotations As I mentioned in another comment, this lets you write annotations naturally without worrying about when something gets defined. 2. typing.TYPE_CHECKING is only True while type checking. This allows you to conditionally import files that would otherwise cause a circular dependency so that you can use the symbols for annotations. 3. def foo(bar: str = None) -> str If an argument defaults to None, then mypy will recognize that its type is actually Optional[whatever]. So in the example above, bar is an Optional[str]. (I'm not sure if this is mypy-specific.) Optional[T] is equivalent to Union[T, None]. 4. You can make typedefs easily. MyType = Union[Sequence[str], Dict[str, int]] Makes writing complicated annotations easier.
- yegle 7y agoIf your code doesn't need to be py 2/3 compatible it's fine to annotate using str. But if you do, typing.Text and bytes is better.
- rcfox 7y agoGood to know, but I hope to never need this.
- quietbritishjim 7y agoAm I missing something? Annotations aren't supported by Python 2 so if you're using them then you never need to worry about Python 2/3 compatibility.
- joshuamorton 7y agoYes and no. Annotations in comments or in type stubs are supported in python2 (you can look at typeshed for typing.Text and conditional py2/3 stuff). There's also some other cases, but they're...unique.
- rcfox 7y ago
- davesque 7y agoThe first point can be rather easily circumvented by using strings as forward references and it only takes a bit of googling to figure that out. The second doesn't seem remotely like a deal breaker to me. Half the reason for having type annotations is to provide a description of what a function or method does at a glance. This is idiomatic even in hardcore functional languages like Haskell. With return type inference, you don't get that benefit.
- cjwoodall 7y agoIt seems to me that type inference would be silly in this case. Type suggestion during mypy checks might be useful though. Function x does not specify a type, but returns SomeHelpfulTypeString. We recommend you use that. Put it behind a flag and call it a day. Maybe even have a way of requesting type suggestions for functions during development
- totalperspectiv 7y agoI really like using and writing type annotated Python, but every time I do, I wonder if I should just use a language where all the annotation time I put in gets me actual runtime improvements.
- carlmr 7y agoAnd accurate compile time checks. I recommend checking F# if you want mostly inferred, strong static types and terse pythonesque Syntax (few braces, significant whitespace). Also much faster. Considering you can use all of .NET from F# you can benefit from the work on C# as well.
- Multicomp 7y agoI'll also wave the F# flag here, but there's also Scala if moving to Javaland is better for you than Dotnetland.
- whalesalad 7y agoI cannot stand typing in Python. It's never given me any saving grace in a project, only headaches.
- jimmaswell 7y agoWhat's possibly so cumbersome about defining return types? It does seem a little embarrassing that the annotations don't even support circular references.
- adrianratnapala 7y agoCircular references like that are called "recursive types" in formal type theory. They are one of those big issues that academics like to write papers about. Even if you just want to write a practical interpreter, and choose to gloss over the issues, they will still come back in some disguised form and either by requiring some sort of implementation kludge, or just by creating weird edge cases.
- heavenlyblue 7y agoAnd how is that an issue for type checking, would you elaborate?
- heavenlyblue 7y agoAuthor of the subj didn’t even bother to Google that issue, which is solved by the way.
- deepsun 7y agoTry Kotlin. Syntax is similar to Python (but better), but it's actually just Java, with all the supporting ecosystem. Kotlin is just syntactic sugar.
- amelius 7y ago> Try Kotlin. Syntax is similar to Python (but better) Well, from [1], I'd say syntax is closer to C. [1] https://kotlinlang.org/docs/reference/control-flow.html https://kotlinlang.org/docs/reference/control-flow.html
- sametmax 7y agoWhile the first complaint is just a reflect of the lack of good tutorials on mypy, the second one is very valid: I too wish for automatic type detection for obvious cases. What's more, while I found mypy useful myself in several instances, it's still cumbersome to use: - you must know to use the magic command line arguments to avoid mypy complaining about other libs and imports all the time - you need to use List[], Dict[], Set[], Iteratble[], Tuple[], etc. instead of list[], dict[], set[], iter[], tuple[], etc., which means an import in almost every file and a very unnatural workflow. - you need to use Union and instead of |. This is ridiculous. I have to do: from typing import Union, List def mask(...) -> List[Union[bool, int]] Instead of: def mask(...) -> list[bool|int] And Guido explicitly rejected the proposal for those on github despite the fact most imports are due to those. Now, mypy has improved a lot. It's way faster, shows many more things that before, way less false positive, and has support for duck typing and dunder methods (named "protocol"). But it's sad to thing TypeScript is actually easier to use. Having a JS tech easier to use than a Python tech is a good sign we can improve things.
- nickm12 7y agoThis is a really poor article with a click-bait headline. I was expecting this to be a case study where Python type annotations just couldn't work due to some interesting specifics of the projects. Instead, it's some petty gripes about the syntax of annotations not being just to the author's liking.
- Barrin92 7y agoI feel baking type annotations on top of dynamic, late binding languages is fundamentally misunderstanding what they are about. Python might not go as far as say, smalltalk, but if you're picking a dynamic language just accept the runtime dynamism and don't program against it.
- heavenlyblue 7y agoOr well, accept the dynamism and use type declarations as automated declarative teat coverage, at the very least.
- perlgeek 7y agoI think the premise of this blog post is misguided. Yes, there are two things about introducing types that the author finds annoying. So don't do those two things, and use type annotations for all the rest. The type annotation system and mypy are designed to give you benefits even if you don't fully annotate everything -- that's the starting point of nearly all existing python code bases.
- jks 7y agoWhat's the status with numpy types? I find two projects, https://github.com/machinalis/mypy-data/tree/master/numpy-mypy https://github.com/machinalis/mypy-data/tree/master/numpy-my... and https://github.com/numpy/numpy-stubs https://github.com/numpy/numpy-stubs but neither looks very complete and neither has been updated recently.