5 ms·

I'm still amazed at how Python code with type hints looks so different to Python without them. Consider the program near the end of the article - if we remove t

by pansa2 6y ago

I'm still amazed at how Python code with type hints looks so different to Python without them. Consider the program near the end of the article - if we remove the type hints: - The `AstNode` abstract base class becomes completely unnecessary and can be deleted. - We can't use `@dataclass` and are forced to write an `__init__` method for each class. But this is good, right? "Explicit is better than implicit"? - It's then clear that our three classes each have two methods, one of which is __init__. Following the advice from "Stop Writing Classes" [0], each class should just be a function (in this case, a generator). These changes reduce the code above "AST representation of the Ruby statement" from 70 lines to 40, and simplify some of those lines as well - for example, `arg_reprs = (arg.representations() for arg in self.args or [])` becomes simply `arg_reprs = args`. The knock-on effects of type annotations seem to be doubling the complexity of the code. This is why I'm far from convinced that static type checking in Python is worth the cost. [0] https://www.youtube.com/watch?v=o9pEzgHorH0 https://www.youtube.com/watch?v=o9pEzgHorH0

josteink 6y ago
> The knock-on effects of type annotations seem to be doubling the complexity of the code. This is why I'm far from convinced that static type checking in Python is worth the cost. If adding types to your existing code cannot be done without creating new types, that’s a sign that your type-system lacks expressiveness. For instance C# 1 lacked a good and general way to describe types for functions (that is, decoupled from a class). In later C# revisions this have been greatly improved, making this pain go away, and accepting type-safe function-value parameters as easy as any other parameter. I’m guessing Python is lagging quite a bit in that regard. That doesn’t mean that type-safety (when done right) isn’t massively useful and worth the effort 200 times over.
- masklinn 6y ago
  I think you’re missing their point. They’re not necessarily saying types are not useful, they’re saying types encourages a form of thinking which tends towards class-based overcomplications. Now this is also a very yagni view, and maybe TFAA uses the structure they build for inspection before the final consumption, but their point that the posted script could be half the size and work the same without loss of readability… is compelling.
  - foxes 6y ago
    Then types in python are not expressive enough. Other languages most of the time you don't even necessarily need to specify the types - the compiler/interpreter can figure it out and still type check the program.
    - nerdponx 6y ago
      Maybe, maybe not. Data classes are meant to feel like Scala's case classes, or Haskell's record syntax. The point is to stop passing around dicts/hashmaps/tables and tuples, and to start passing around containers with a known and fixed set of keys/names, and known types for the values/elements. I don't know of any language where the compiler automatically infers field names and types, and I don't know if I would ever want such a thing.
      - josteink 6y ago
        > I don't know of any language where the compiler automatically infers field names and types, and I don't know if I would ever want such a thing. Typescript can do this to unannotated code, and it’s surprisingly useful.
        nerdponx 6y ago
        I've already started checking out Typescript as a general-purpose scripting language to replace Python, Ruby, Perl, and to some extent Bash/Zsh. Knowing this makes it even more attractive. The only annoyances I've had so far are: difficult-to-search docs, and the `tsc` command-line compiler is slow.
    - dragonwriter 6y ago
      Even in languages where this is possible, it's usually idiomatic to explicitly type the public interface including exposed fields, and often it's common to explicitly type them narrower than the inferred type to avoid implementation details leaking into an unintentionally broad backward compatibility contract restricting future evolution.
  - 6y ago
viraptor 6y ago
> We can't use `@dataclass` and are forced to write an `__init__` method for each class. But this is good, right? "Explicit is better than implicit"? I don't think that applies. @dataclass is an explicit wrapper with specific result and can be inspected and interacted with. Implicit would be something like "when a class has more than one attribute and doesn't define init or repr, add those methods assuming they're wanted/needed". > The `AstNode` abstract base class becomes completely unnecessary and can be deleted. You're mixing type annotations with class usage. You can have either one without the other. For example without AstNode you could still have an `Expr|MethodCall|Lambda` type hint on relevant variables. Or even alias that expression to AstNode without having the class and inheritance. Finally, classes help describe/construct the structure present in "stmt". Sure, you could replace those with functions, but that means your data is either reduced to untyped / string-tagged lists/dicts, or you're running the exact function immediately inline. It doesn't matter for this toy example, but it would in any practical use of this code.
- mumblemumble 6y ago
  @dataclass is one of those ones where it depends on how you look at it. To me, it's a great application of "explicit is better than implicit." Because @dataclass explicitly indicates that this class has a certain set of behavior, and is meant to be used a certain way. If you manually implement exactly the same behavior, you've made the individual statements more explicit, but you've made the class's actual function implicit. A reader needs to figure it out by reading the code and working backward from there. An editor of the code needs to recognize the intended behavior and not break it, otherwise the understanding that other people built up by reading the code may break in subtle ways.
gjulianm 6y ago
I don't think you can draw that conclusion from that code. I've been using Python for a few years and moved some projects to static type checking. It's not always worth the cost in every situation, of course. For such a simple script I wouldn't add type checking. But for larger projects where there are multiple classes and a lot of types, I've found that the effort is very much worth it: a lot of type errors are found faster (some of them hadn't been caught by unit testing), the code becomes more consistent and easier to read, and developing is easier if you have an IDE that understands the types and gives you appropriate suggestions. It's also worth it in plumbing code, as it tends to be more difficult to test properly and having types will catch a whole class of errors. The fact that the program at the bottom of the article can be simplified is not an issue of typing, in fact static typing could also be used in your approach without any issue.
BiteCode_dev 6y ago
It's more of a design issue here. You could do away from classes while keeping types by using 2 lines: Expr = Callable[[str]], Callable[..., Iterable[str]]] AstNode = Expr | Callable[... ], Callable[..., Iterable[str]]] You then don't event need to call representation() , as each node is callable. No classes needed. But my instinct tells me a node will eventually have more attributes and methods. Stuff to get parents, children, siblings, filtering and so on. So it's probably a good call to use a class for later. Also you want to keep AstNode as a node could be an expression, a statement or a comment.
- mumblemumble 6y ago
  > But my instinct tells me a node will eventually have more attributes and methods. I couldn't agree more with Jack Diederich's thoughts on that sentiment, at 5:40 in the video linked above: https://youtu.be/o9pEzgHorH0?t=340 https://youtu.be/o9pEzgHorH0?t=340 > Lots of times people think they might need something later. You don't. Or, you can just do it later.
  - BiteCode_dev 6y ago
    https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html Experience helps with this. It's a middle path as usual: naive approaches waste as much time as overenginering.
  - nerdponx 6y ago
    YAGNI is a double-edged sword. It's almost no additional overhead to write a class now if you think you might need to add methods and internal state later. But it's quite a bit of refactoring work if you decide not to write a class and then realize you need one after the fact. Maybe this is an indictment of Python as a language, but I think it's a case of not taking anything too literally. YAGNI fetishism is like DRY fetishism all over again.
    - mumblemumble 6y ago
      I guess? At the same time, working in some languages (Racket and C come to mind), it's surprising how long I can go without anything ever becoming a class, ever. Seemingly indefinitely. My sense is that many of us are taught to think in terms of classes and methods instead of data structures and functions, so it's very easy to slip into thinking you need objects in order to manage state, even when that's not the simplest way to do it. Strictly speaking, though, I think that the only time in Python that you really have no workable alternative to classes is when you also need dynamic dispatch.
jasfi 6y ago
If you're going to use Python with types, then you may as well use Nim. Nim has a similar syntax, but with better performance. You can also interface with Python code via Nimpy.
- craftinator 6y ago
  I love using Nim, it's an absolute pleasure after years of fighting with Python's "I hope I got this right..." style of type guessing. The biggest problem most people run into is that python has a huge library ecosystem. I haven't actually tried interfacing them, it's next on my docket. Have you found any good learning resources for that?
  - jasfi 6y ago
    Nimpy: https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy A good test as an example: https://github.com/yglukhov/nimpy/blob/master/tests/tpyfromnim.nim https://github.com/yglukhov/nimpy/blob/master/tests/tpyfromn... I've tried it and it works fine. The only problem I ever had was trying to call a specific Python function at the same time from multiple Nim threads. I'm not sure where the fault lay exactly, I just made that part of the code single-threaded and moved on.
- nerdponx 6y ago
  I like Nim, but I don't think there is much of anything similar between these languages except the syntax.

by pansa2 6y ago

josteink 6y ago

> The knock-on effects of type annotations seem to be doubling the complexity of the code. This is why I'm far from convinced that static type checking in Python is worth the cost. If adding types to your existing code cannot be done without creating new types, that’s a sign that your type-system lacks expressiveness. For instance C# 1 lacked a good and general way to describe types for functions (that is, decoupled from a class). In later C# revisions this have been greatly improved, making this pain go away, and accepting type-safe function-value parameters as easy as any other parameter. I’m guessing Python is lagging quite a bit in that regard. That doesn’t mean that type-safety (when done right) isn’t massively useful and worth the effort 200 times over.

masklinn 6y ago
I think you’re missing their point. They’re not necessarily saying types are not useful, they’re saying types encourages a form of thinking which tends towards class-based overcomplications. Now this is also a very yagni view, and maybe TFAA uses the structure they build for inspection before the final consumption, but their point that the posted script could be half the size and work the same without loss of readability… is compelling.
- foxes 6y ago
  Then types in python are not expressive enough. Other languages most of the time you don't even necessarily need to specify the types - the compiler/interpreter can figure it out and still type check the program.
  - nerdponx 6y ago
    Maybe, maybe not. Data classes are meant to feel like Scala's case classes, or Haskell's record syntax. The point is to stop passing around dicts/hashmaps/tables and tuples, and to start passing around containers with a known and fixed set of keys/names, and known types for the values/elements. I don't know of any language where the compiler automatically infers field names and types, and I don't know if I would ever want such a thing.
    - josteink 6y ago
      > I don't know of any language where the compiler automatically infers field names and types, and I don't know if I would ever want such a thing. Typescript can do this to unannotated code, and it’s surprisingly useful.
      - nerdponx 6y ago
        I've already started checking out Typescript as a general-purpose scripting language to replace Python, Ruby, Perl, and to some extent Bash/Zsh. Knowing this makes it even more attractive. The only annoyances I've had so far are: difficult-to-search docs, and the `tsc` command-line compiler is slow.
  - dragonwriter 6y ago
    Even in languages where this is possible, it's usually idiomatic to explicitly type the public interface including exposed fields, and often it's common to explicitly type them narrower than the inferred type to avoid implementation details leaking into an unintentionally broad backward compatibility contract restricting future evolution.
- 6y ago

viraptor 6y ago

> We can't use `@dataclass` and are forced to write an `__init__` method for each class. But this is good, right? "Explicit is better than implicit"? I don't think that applies. @dataclass is an explicit wrapper with specific result and can be inspected and interacted with. Implicit would be something like "when a class has more than one attribute and doesn't define init or repr, add those methods assuming they're wanted/needed". > The `AstNode` abstract base class becomes completely unnecessary and can be deleted. You're mixing type annotations with class usage. You can have either one without the other. For example without AstNode you could still have an `Expr|MethodCall|Lambda` type hint on relevant variables. Or even alias that expression to AstNode without having the class and inheritance. Finally, classes help describe/construct the structure present in "stmt". Sure, you could replace those with functions, but that means your data is either reduced to untyped / string-tagged lists/dicts, or you're running the exact function immediately inline. It doesn't matter for this toy example, but it would in any practical use of this code.

mumblemumble 6y ago
@dataclass is one of those ones where it depends on how you look at it. To me, it's a great application of "explicit is better than implicit." Because @dataclass explicitly indicates that this class has a certain set of behavior, and is meant to be used a certain way. If you manually implement exactly the same behavior, you've made the individual statements more explicit, but you've made the class's actual function implicit. A reader needs to figure it out by reading the code and working backward from there. An editor of the code needs to recognize the intended behavior and not break it, otherwise the understanding that other people built up by reading the code may break in subtle ways.

gjulianm 6y ago

I don't think you can draw that conclusion from that code. I've been using Python for a few years and moved some projects to static type checking. It's not always worth the cost in every situation, of course. For such a simple script I wouldn't add type checking. But for larger projects where there are multiple classes and a lot of types, I've found that the effort is very much worth it: a lot of type errors are found faster (some of them hadn't been caught by unit testing), the code becomes more consistent and easier to read, and developing is easier if you have an IDE that understands the types and gives you appropriate suggestions. It's also worth it in plumbing code, as it tends to be more difficult to test properly and having types will catch a whole class of errors. The fact that the program at the bottom of the article can be simplified is not an issue of typing, in fact static typing could also be used in your approach without any issue.

BiteCode_dev 6y ago

It's more of a design issue here. You could do away from classes while keeping types by using 2 lines: Expr = Callable[[str]], Callable[..., Iterable[str]]] AstNode = Expr | Callable[... ], Callable[..., Iterable[str]]] You then don't event need to call representation() , as each node is callable. No classes needed. But my instinct tells me a node will eventually have more attributes and methods. Stuff to get parents, children, siblings, filtering and so on. So it's probably a good call to use a class for later. Also you want to keep AstNode as a node could be an expression, a statement or a comment.

mumblemumble 6y ago
> But my instinct tells me a node will eventually have more attributes and methods. I couldn't agree more with Jack Diederich's thoughts on that sentiment, at 5:40 in the video linked above: https://youtu.be/o9pEzgHorH0?t=340 https://youtu.be/o9pEzgHorH0?t=340 > Lots of times people think they might need something later. You don't. Or, you can just do it later.
- BiteCode_dev 6y ago
  https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html Experience helps with this. It's a middle path as usual: naive approaches waste as much time as overenginering.
- nerdponx 6y ago
  YAGNI is a double-edged sword. It's almost no additional overhead to write a class now if you think you might need to add methods and internal state later. But it's quite a bit of refactoring work if you decide not to write a class and then realize you need one after the fact. Maybe this is an indictment of Python as a language, but I think it's a case of not taking anything too literally. YAGNI fetishism is like DRY fetishism all over again.
  - mumblemumble 6y ago
    I guess? At the same time, working in some languages (Racket and C come to mind), it's surprising how long I can go without anything ever becoming a class, ever. Seemingly indefinitely. My sense is that many of us are taught to think in terms of classes and methods instead of data structures and functions, so it's very easy to slip into thinking you need objects in order to manage state, even when that's not the simplest way to do it. Strictly speaking, though, I think that the only time in Python that you really have no workable alternative to classes is when you also need dynamic dispatch.

jasfi 6y ago

If you're going to use Python with types, then you may as well use Nim. Nim has a similar syntax, but with better performance. You can also interface with Python code via Nimpy.

craftinator 6y ago
I love using Nim, it's an absolute pleasure after years of fighting with Python's "I hope I got this right..." style of type guessing. The biggest problem most people run into is that python has a huge library ecosystem. I haven't actually tried interfacing them, it's next on my docket. Have you found any good learning resources for that?
- jasfi 6y ago
  Nimpy: https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy A good test as an example: https://github.com/yglukhov/nimpy/blob/master/tests/tpyfromnim.nim https://github.com/yglukhov/nimpy/blob/master/tests/tpyfromn... I've tried it and it works fine. The only problem I ever had was trying to call a specific Python function at the same time from multiple Nim threads. I'm not sure where the fault lay exactly, I just made that part of the code single-threaded and moved on.
nerdponx 6y ago
I like Nim, but I don't think there is much of anything similar between these languages except the syntax.