9 ms·

Honestly I haven't looked into MyPy a huge amount, and haven't used it yet. But from reading the docs on it I feel like it's taking the wrong approach. In my op

Honestly I haven't looked into MyPy a huge amount, and haven't used it yet. But from reading the docs on it I feel like it's taking the wrong approach. In my opinion, Python is all about duck typing, so having a static type checker that seems to focus on concrete types feels wrong to me. Consider the first example, def sum_and_stringify(nums: List[int]) -> str: return str(sum(nums)) The type here is way more specific than it needs to be. Not only will this work on any sequence type (anything that supports __iter__), the element type only needs to support __add__ and __radd__ (to work against the implicit seed value of sum). I really feel like in order to get the best of duck typing, you need a good syntax for describing types by usage, not by explicit name. Please let me know if I'm missing something, and this is actually more expressive than the documentation suggests.

dbcurtis 10y ago
Well, I like to put that another way -- I think Python is all about protocols (in the Python sense of the word protocol). I think that is what most people mean when they say Python is all about duck typing, and it communicates the concept more precisely. The concept of 'type' is overloaded with too many meanings these days. It started out as a what-storage-to-allocate and which-instruction-to-generate (int v float) concept (FORTRAN I, variables starting with I..N are ints, others real). Deep class hierarchies muddied the waters by overlaying the idea of whether or not a class implements a particular property or method, and how to locate the property or method that you want. Python shows us that types and protocols/interfaces are two very different thing. In Python, asking 'isinstance()' is usually a bad idea unless you are doing something very low level, low level as in next stop is a C extension. The missing predicate in Python is 'hasprotocol()'. Abstract base classes can be used to accomplish that: isinstance(foo, SomeABC), but I think the intent could be more transparent with different syntax.
- clusmore 10y ago
  Yeah I agree with you completely, I think we just use different terminology to describe the same thing. It sounds like you use "type" to mean "concrete type", i.e. the thing it actually is, and not the thing it acts like, and then "protocol/interface" to describe the behaviour, the thing we can use it as. As somebody who mostly uses statically-typed languages, I use "type" to mean "the thing my static type checker knows this as", which could be a concrete type or simply an interface, and then obviously I use the term "concrete type" when I need to be more explicit. I like the idea of a hasprotocol or supportsprotocol predicate, certainly over usage of isinstance, since one of the beauties of duck typing is that it works without requiring the type hierarchy. The tricky part will be in allowing you to define your own protocols, like "has these attributes/methods". An example would be the difference (or lack thereof) between classes and namedtuples.
  - benhoyt 10y ago
    Interestingly, I think that's what isinstance() actually does when it's called on abstract base classes like collections.abc.Iterable -- it looks to see whether the type supports __iter__ and whatnot. See the abc module and it's implementation here: * https://docs.python.org/3.4/library/collections.abc.html https://docs.python.org/3.4/library/collections.abc.html * https://hg.python.org/cpython/file/3.4/Lib/_collections_abc.py https://hg.python.org/cpython/file/3.4/Lib/_collections_abc....
    - clusmore 10y ago
      Pasting the relevant snippet from your link: def __subclasshook__(cls, C): if cls is Iterable: if any("__iter__" in B.__dict__ for B in C.__mro__): return True return NotImplemented Wow, hadn't seen this before. That's exactly what I mean. If I can define my own, e.g. Addable, which does a similar check for the presence of __add__ (and __radd__) and MyPy uses the subclasshook, then that'd be perfect.
      - dbcurtis 10y ago
        Get _Fluent_Python_ by Ramalho and read chapter 11.
- dllthomas 10y ago
  > Python shows us that types and protocols/interfaces are two very different thing. In Python, asking 'isinstance()' is usually a bad idea Interesting perspective. This would seem to be pretty well described by parametric polymorphism with constraints (Haskell-style polymorphism with typeclasses). In that context, too, we are often well served staying as generic as possible, and asking at runtime about the type of something is a bit of a code smell (and only possible where you have a Typeable constraint available).
  - clusmore 10y ago
    Yes. Haskell's type deduction is exactly what I was thinking of as I read the article. You write a function using what you need and it deducts the requirements based on usage, giving you the most general type. I think this is actually very close to compile-time duck-typing. Actually I think if you took the class declarations out of Haskell and removed the instance lines (but left the subsequent implementations), and had the compiler deduce typeclasses for you, you'd get something that felt very much like compile-time duck-typing.
    - int_19h 10y ago
      If you want an even closer example, it would be OCaml with its structurally typed object system with inferred types. For example: # let f obj x y = (obj#m x)#n y;; val f : < m : 'a -> < n : 'b -> 'c; .. >; .. > -> 'a -> 'b -> 'c = <fun> Here I defined a function taking 3 arguments, called method m on obj, passing x as argument, then called method n on the result of that, passing y as argument. Note the inferred type signature for this: the first argument is of type <m : 'a -> <n : 'b -> 'c; ..>; ..> - i.e. any object that has a method named m (and possibly some other unspecified members - that's what ".." means), with said method m having a signature that allows us to pass some 'a, and returning another object of type <n : 'b -> 'c; ..> - i.e. something with a method named n that accepts 'b and returns 'c. 'a and 'b' are then used as types of x and y, respectively, and 'c is the the result of f. So effectively, this is full-fledged duck typing in the vein of Python objects, but captured on type system level.
      - 10y ago
- brettcannon 10y ago
  There's an open issue discussing the idea of an explicit protocol support somehow in Python: https://github.com/python/typing/issues/11 https://github.com/python/typing/issues/11 .
zardeh 10y ago
Well yes, but # optionally: Number = Union[int, float, complex] def sum_and_stringify(nums: Iterable[Number]) -> str: return str(sum(nums)) also works. Except no one knows what the Number abstract base class is (or even the Iterable abstract base class for that matter)
- clusmore 10y ago
  Yeah but as I said it can still go far more general than just int, float and complex. It could be any arbitrary class defined by somebody else that I could not possibly know about at design time, and so a whitelisting approach of concrete classes is simply not going to cut it. It would be nice to have a way to describe the actual requirements (only that the elements support __add__ and __radd__).
  - zardeh 10y ago
    Well, my joke was that numbers.Number exists, and I think supports those operations, but even someone complaining about this topic is unaware that it exists.
emmelaich 10y ago
You can't say it is more specific than it needs to be without knowing the author's intent AND the semantics of all the __add__ methods of all types that have these methods. Which is for practical purposes impossible. Semantically, the presence of __add__ and __radd__ is a minimum, not sufficient.
- clusmore 10y ago
  Well sure but that's just the danger of duck typing vs explicit interface implementation, something Python already suffers from. Without MyPy, Python will already let you use this function on all sequences of addable things, not just lists of ints. It would be good if MyPy reflected this as closely as possible, otherwise you're throwing away the advantages of Python's duck-typing. Edit to add: To be clear, sure the result of calling this on any arbitrary sequence of addables might be semantically nonsense, but that's not the purpose of a type checker. It's only to verify type safety, i.e. you won't get runtime type errors if you give me at least this kind of thing.
  - dllthomas 10y ago
    Part of the issue is the name, which is generic enough that with the generic implementation, it probably also wants a maximally generic type. A more constrained type throws away "the advantages of Python's duck-typing", but it also throws away disadvantages, and that can be precisely what you want.
    - clusmore 10y ago
      Yes, this is a good point too which I certainly agree with. Sometimes by using the most general type, you actually permit more than you intended and you might restrict yourself from making refactorings to the function because it would restrict the types.
  - dllthomas 10y ago
    > might be semantically nonsense, but that's not the purpose of a type checker. It's only to verify type safety, i.e. you won't get runtime type errors if you give me at least this kind of thing. We either disagree about the purpose of a type checker, or we disagree about what can constitute a "runtime type error". If a type checker can catch inconsistencies that would lead to my returning "semantic nonsense", at low enough cost, I absolutely want it to. Obviously it won't be able to catch all such inconsistencies in all cases.
rtpg 10y ago
protocols (think Java interfaces) are becoming a thing for the type checking, so that's what you can use in places like this in the future. something like `Iterable[Addable]`
- clusmore 10y ago
  This is great provided that Addable is not just something hardcoded to know about builtins like int and float. If I create my own class that quacks like an Addable, then it needs to be supported. That's the essence of duck-typing.
  - yxhuvud 10y ago
    I suggest you go and have a long hard look at Crystal. From your comments I am getting the feeling you could appreciate it.
    - clusmore 10y ago
      Thanks for the suggestion. For the record, I'm a fairly happy C# for work, Haskell for hobby programmer, using Python as my go-to for quick-and-dirty scripts. My comments here are mostly based on my own thoughts on the static vs dynamic debate, in this case specifically how to keep the advantages of dynamic languages whilst absorbing some of the benefits of static checking. They have nothing to do with the best usage of MyPy in production-grade Python projects, of which I have zero experience.
RubyPinch 10y ago
I find it a bit easier to treat it as documentation, instead of literal types mypy will then tell you if you used your documented code wrong, according to your documentation also sum excludes summing of strings if I remember correctly, so its any object that has an __add__ and __radd__ and isn't a string or subclass of string. which is just a bit insane to try and express on one line!
- clusmore 10y ago
  That's a fair point on the lack of support for strings by sum, but note this is only because the function is hard-coded explicitly to reject them and only because it's not the best way to build strings. By usage, strings should just work. On the point of documentation, this is potentially a fair point if you consider that you probably wrote this function for a particular usage and you know at design time that you're really only ever going to give it a list of ints (if you strip the types away, the parameter is still called nums which suggests that's what was really intended). In reality, the function is actually much more powerful than that, and if included in a library, many people could use it for many more things than just that.
  - dllthomas 10y ago
    > In reality, the function is actually much more powerful than that, and if included in a library, many people could use it for many more things than just that. I think all this says is that design and documentation considerations for functions in distributed libraries may be different than those for functions used locally. Something I certainly agree with, but not really a criticism of the parent point.
    - clusmore 10y ago
      Fair point. I realise I sound quite critical in my comments so perhaps I should add that I like the tool itself. The problems that I see are more in the usage as shown in the examples. I am not necessarily suggesting that in all circumstances, the most general types are the best. But I am yet to see this tool used in such a way as to permit the most general types, as would be appropriate for library functions.
dom96 10y ago
> I really feel like in order to get the best of duck typing, you need a good syntax for describing types by usage, not by explicit name. Not sure if this is of interest to you, but the Nim[1] programming language offers a feature that allows you to do exactly what you described. It is called concepts and is documented here: http://nim-lang.org/docs/manual.html#generics-concepts http://nim-lang.org/docs/manual.html#generics-concepts As a bonus, Nim is also quite Python-like. 1 - http://nim-lang.org http://nim-lang.org
- clusmore 10y ago
  Yep that sounds like what I mean. Haskell typeclasses, Go interfaces, TypeScript interfaces are further examples. Not Java/C# interfaces though, as they need to be declared at design time, and you can't force existing classes to implement interfaces they haven't already declared themselves as implementing.
prawks 10y ago
From the mypy docs linked in the article (http://mypy.readthedocs.io/en/latest/duck_type_compatibility.html http://mypy.readthedocs.io/en/latest/duck_type_compatibility...), it seems it supports a small number of built-in duck types currently, and has support for some protocols (http://mypy.readthedocs.io/en/latest/cheat_sheet.html#standard-duck-types http://mypy.readthedocs.io/en/latest/cheat_sheet.html#standa...) > mypy considers an int object to be valid whenever a float object is expected. > Use Iterable for generic iterables (anything usable in `for`) Thus you should be able to get a more broadly correct (though still not 100% if you have your own types implementing __add__ and __radd__) like this: from typing import Iterator def sum_and_stringify(nums: Iterator[float]) -> str: return str(sum(nums)) It also looks like implementing an abstract base class SupportsSum (to check for __add__ and __radd__) may not take a whole lot of work, see `SupportsInt`: https://github.com/python/mypy/blob/d7f6ea617317103534d8668658d176d7a8972cb2/lib-typing/3.2/typing.py#L1370 https://github.com/python/mypy/blob/d7f6ea617317103534d86686... http://mypy.readthedocs.io/en/latest/class_basics.html?highlight=protocol#abstract-base-classes-and-multiple-inheritance http://mypy.readthedocs.io/en/latest/class_basics.html?highl...
- clusmore 10y ago
  I had heard of the support for the hierarchy of some builtins like floats being more general than ints, but this isn't a very satisfactory approach because it's not extensible to user-defined numeric types. Perhaps what I'm suggesting is possible with either the existing classes in abc, or by defining more like them (and ensuring users can define their own). Would be nice to see more focus on this in the examples if that's the case.

dbcurtis 10y ago

Well, I like to put that another way -- I think Python is all about protocols (in the Python sense of the word protocol). I think that is what most people mean when they say Python is all about duck typing, and it communicates the concept more precisely. The concept of 'type' is overloaded with too many meanings these days. It started out as a what-storage-to-allocate and which-instruction-to-generate (int v float) concept (FORTRAN I, variables starting with I..N are ints, others real). Deep class hierarchies muddied the waters by overlaying the idea of whether or not a class implements a particular property or method, and how to locate the property or method that you want. Python shows us that types and protocols/interfaces are two very different thing. In Python, asking 'isinstance()' is usually a bad idea unless you are doing something very low level, low level as in next stop is a C extension. The missing predicate in Python is 'hasprotocol()'. Abstract base classes can be used to accomplish that: isinstance(foo, SomeABC), but I think the intent could be more transparent with different syntax.

clusmore 10y ago
Yeah I agree with you completely, I think we just use different terminology to describe the same thing. It sounds like you use "type" to mean "concrete type", i.e. the thing it actually is, and not the thing it acts like, and then "protocol/interface" to describe the behaviour, the thing we can use it as. As somebody who mostly uses statically-typed languages, I use "type" to mean "the thing my static type checker knows this as", which could be a concrete type or simply an interface, and then obviously I use the term "concrete type" when I need to be more explicit. I like the idea of a hasprotocol or supportsprotocol predicate, certainly over usage of isinstance, since one of the beauties of duck typing is that it works without requiring the type hierarchy. The tricky part will be in allowing you to define your own protocols, like "has these attributes/methods". An example would be the difference (or lack thereof) between classes and namedtuples.
- benhoyt 10y ago
  Interestingly, I think that's what isinstance() actually does when it's called on abstract base classes like collections.abc.Iterable -- it looks to see whether the type supports __iter__ and whatnot. See the abc module and it's implementation here: * https://docs.python.org/3.4/library/collections.abc.html https://docs.python.org/3.4/library/collections.abc.html * https://hg.python.org/cpython/file/3.4/Lib/_collections_abc.py https://hg.python.org/cpython/file/3.4/Lib/_collections_abc....
  - clusmore 10y ago
    Pasting the relevant snippet from your link: def __subclasshook__(cls, C): if cls is Iterable: if any("__iter__" in B.__dict__ for B in C.__mro__): return True return NotImplemented Wow, hadn't seen this before. That's exactly what I mean. If I can define my own, e.g. Addable, which does a similar check for the presence of __add__ (and __radd__) and MyPy uses the subclasshook, then that'd be perfect.
    - dbcurtis 10y ago
      Get _Fluent_Python_ by Ramalho and read chapter 11.
dllthomas 10y ago
> Python shows us that types and protocols/interfaces are two very different thing. In Python, asking 'isinstance()' is usually a bad idea Interesting perspective. This would seem to be pretty well described by parametric polymorphism with constraints (Haskell-style polymorphism with typeclasses). In that context, too, we are often well served staying as generic as possible, and asking at runtime about the type of something is a bit of a code smell (and only possible where you have a Typeable constraint available).
- clusmore 10y ago
  Yes. Haskell's type deduction is exactly what I was thinking of as I read the article. You write a function using what you need and it deducts the requirements based on usage, giving you the most general type. I think this is actually very close to compile-time duck-typing. Actually I think if you took the class declarations out of Haskell and removed the instance lines (but left the subsequent implementations), and had the compiler deduce typeclasses for you, you'd get something that felt very much like compile-time duck-typing.
  - int_19h 10y ago
    If you want an even closer example, it would be OCaml with its structurally typed object system with inferred types. For example: # let f obj x y = (obj#m x)#n y;; val f : < m : 'a -> < n : 'b -> 'c; .. >; .. > -> 'a -> 'b -> 'c = <fun> Here I defined a function taking 3 arguments, called method m on obj, passing x as argument, then called method n on the result of that, passing y as argument. Note the inferred type signature for this: the first argument is of type <m : 'a -> <n : 'b -> 'c; ..>; ..> - i.e. any object that has a method named m (and possibly some other unspecified members - that's what ".." means), with said method m having a signature that allows us to pass some 'a, and returning another object of type <n : 'b -> 'c; ..> - i.e. something with a method named n that accepts 'b and returns 'c. 'a and 'b' are then used as types of x and y, respectively, and 'c is the the result of f. So effectively, this is full-fledged duck typing in the vein of Python objects, but captured on type system level.
    - 10y ago
brettcannon 10y ago
There's an open issue discussing the idea of an explicit protocol support somehow in Python: https://github.com/python/typing/issues/11 https://github.com/python/typing/issues/11 .

zardeh 10y ago

Well yes, but # optionally: Number = Union[int, float, complex] def sum_and_stringify(nums: Iterable[Number]) -> str: return str(sum(nums)) also works. Except no one knows what the Number abstract base class is (or even the Iterable abstract base class for that matter)

clusmore 10y ago
Yeah but as I said it can still go far more general than just int, float and complex. It could be any arbitrary class defined by somebody else that I could not possibly know about at design time, and so a whitelisting approach of concrete classes is simply not going to cut it. It would be nice to have a way to describe the actual requirements (only that the elements support __add__ and __radd__).
- zardeh 10y ago
  Well, my joke was that numbers.Number exists, and I think supports those operations, but even someone complaining about this topic is unaware that it exists.

emmelaich 10y ago

You can't say it is more specific than it needs to be without knowing the author's intent AND the semantics of all the __add__ methods of all types that have these methods. Which is for practical purposes impossible. Semantically, the presence of __add__ and __radd__ is a minimum, not sufficient.

clusmore 10y ago
Well sure but that's just the danger of duck typing vs explicit interface implementation, something Python already suffers from. Without MyPy, Python will already let you use this function on all sequences of addable things, not just lists of ints. It would be good if MyPy reflected this as closely as possible, otherwise you're throwing away the advantages of Python's duck-typing. Edit to add: To be clear, sure the result of calling this on any arbitrary sequence of addables might be semantically nonsense, but that's not the purpose of a type checker. It's only to verify type safety, i.e. you won't get runtime type errors if you give me at least this kind of thing.
- dllthomas 10y ago
  Part of the issue is the name, which is generic enough that with the generic implementation, it probably also wants a maximally generic type. A more constrained type throws away "the advantages of Python's duck-typing", but it also throws away disadvantages, and that can be precisely what you want.
  - clusmore 10y ago
    Yes, this is a good point too which I certainly agree with. Sometimes by using the most general type, you actually permit more than you intended and you might restrict yourself from making refactorings to the function because it would restrict the types.
- dllthomas 10y ago
  > might be semantically nonsense, but that's not the purpose of a type checker. It's only to verify type safety, i.e. you won't get runtime type errors if you give me at least this kind of thing. We either disagree about the purpose of a type checker, or we disagree about what can constitute a "runtime type error". If a type checker can catch inconsistencies that would lead to my returning "semantic nonsense", at low enough cost, I absolutely want it to. Obviously it won't be able to catch all such inconsistencies in all cases.

rtpg 10y ago

protocols (think Java interfaces) are becoming a thing for the type checking, so that's what you can use in places like this in the future. something like `Iterable[Addable]`

clusmore 10y ago
This is great provided that Addable is not just something hardcoded to know about builtins like int and float. If I create my own class that quacks like an Addable, then it needs to be supported. That's the essence of duck-typing.
- yxhuvud 10y ago
  I suggest you go and have a long hard look at Crystal. From your comments I am getting the feeling you could appreciate it.
  - clusmore 10y ago
    Thanks for the suggestion. For the record, I'm a fairly happy C# for work, Haskell for hobby programmer, using Python as my go-to for quick-and-dirty scripts. My comments here are mostly based on my own thoughts on the static vs dynamic debate, in this case specifically how to keep the advantages of dynamic languages whilst absorbing some of the benefits of static checking. They have nothing to do with the best usage of MyPy in production-grade Python projects, of which I have zero experience.

RubyPinch 10y ago

I find it a bit easier to treat it as documentation, instead of literal types mypy will then tell you if you used your documented code wrong, according to your documentation also sum excludes summing of strings if I remember correctly, so its any object that has an __add__ and __radd__ and isn't a string or subclass of string. which is just a bit insane to try and express on one line!

clusmore 10y ago
That's a fair point on the lack of support for strings by sum, but note this is only because the function is hard-coded explicitly to reject them and only because it's not the best way to build strings. By usage, strings should just work. On the point of documentation, this is potentially a fair point if you consider that you probably wrote this function for a particular usage and you know at design time that you're really only ever going to give it a list of ints (if you strip the types away, the parameter is still called nums which suggests that's what was really intended). In reality, the function is actually much more powerful than that, and if included in a library, many people could use it for many more things than just that.
- dllthomas 10y ago
  > In reality, the function is actually much more powerful than that, and if included in a library, many people could use it for many more things than just that. I think all this says is that design and documentation considerations for functions in distributed libraries may be different than those for functions used locally. Something I certainly agree with, but not really a criticism of the parent point.
  - clusmore 10y ago
    Fair point. I realise I sound quite critical in my comments so perhaps I should add that I like the tool itself. The problems that I see are more in the usage as shown in the examples. I am not necessarily suggesting that in all circumstances, the most general types are the best. But I am yet to see this tool used in such a way as to permit the most general types, as would be appropriate for library functions.

dom96 10y ago

> I really feel like in order to get the best of duck typing, you need a good syntax for describing types by usage, not by explicit name. Not sure if this is of interest to you, but the Nim[1] programming language offers a feature that allows you to do exactly what you described. It is called concepts and is documented here: http://nim-lang.org/docs/manual.html#generics-concepts http://nim-lang.org/docs/manual.html#generics-concepts As a bonus, Nim is also quite Python-like. 1 - http://nim-lang.org http://nim-lang.org

clusmore 10y ago
Yep that sounds like what I mean. Haskell typeclasses, Go interfaces, TypeScript interfaces are further examples. Not Java/C# interfaces though, as they need to be declared at design time, and you can't force existing classes to implement interfaces they haven't already declared themselves as implementing.

prawks 10y ago

From the mypy docs linked in the article (http://mypy.readthedocs.io/en/latest/duck_type_compatibility.html http://mypy.readthedocs.io/en/latest/duck_type_compatibility...), it seems it supports a small number of built-in duck types currently, and has support for some protocols (http://mypy.readthedocs.io/en/latest/cheat_sheet.html#standard-duck-types http://mypy.readthedocs.io/en/latest/cheat_sheet.html#standa...) > mypy considers an int object to be valid whenever a float object is expected. > Use Iterable for generic iterables (anything usable in `for`) Thus you should be able to get a more broadly correct (though still not 100% if you have your own types implementing __add__ and __radd__) like this: from typing import Iterator def sum_and_stringify(nums: Iterator[float]) -> str: return str(sum(nums)) It also looks like implementing an abstract base class SupportsSum (to check for __add__ and __radd__) may not take a whole lot of work, see `SupportsInt`: https://github.com/python/mypy/blob/d7f6ea617317103534d8668658d176d7a8972cb2/lib-typing/3.2/typing.py#L1370 https://github.com/python/mypy/blob/d7f6ea617317103534d86686... http://mypy.readthedocs.io/en/latest/class_basics.html?highlight=protocol#abstract-base-classes-and-multiple-inheritance http://mypy.readthedocs.io/en/latest/class_basics.html?highl...

clusmore 10y ago
I had heard of the support for the hierarchy of some builtins like floats being more general than ints, but this isn't a very satisfactory approach because it's not extensible to user-defined numeric types. Perhaps what I'm suggesting is possible with either the existing classes in abc, or by defining more like them (and ensuring users can define their own). Would be nice to see more focus on this in the examples if that's the case.