5 ms·

Yes, this higher abstractions tendency is clearly away from "Flat is better than nested". Creating good abstractions is so difficult (and important) that it sho

by janto 5y ago

Yes, this higher abstractions tendency is clearly away from "Flat is better than nested". Creating good abstractions is so difficult (and important) that it should not be done lightly by everyone. The recent proliferation of type annotations is also trending towards the static typing mindset, which is also cargo culting on a massive scale.

smohare 5y ago
I think the comment about types takes a rather narrow view of Python use cases. Sure, if you’re some lone sysadmin using Python as an alternative to bash to implement some questionable ETL pipeline comprising a bunch of random scripts running off a single machine somewhere then I agree. You probably are not the target audience for type annotations. But even moderately complex libraries and applications are such a huge pain to develop, read, and maintain without some of the tooling that leverages type annotations. This is especially true in the ML / AI / Data Science world where a lot of the people implementing models have dubious coding practices. Pure dynamic typing paradigms simply have not delivered on their promises over the last 30 years. There are definitely some areas where they make sense, but I doubt we will see a massive readoption until our tooling becomes sufficiently intelligent. Imagine, for example, a probabilistic type inference based on both the structural aspects of the code and previous runs over actual data.
- janto 5y ago
  People in the data science world with dubious coding practices should keep their data in commonly used types such as Pandas tables. The last thing we should want from them is encoding their data with ill-conceived class/type systems. Not sure if that is what you mean. I'd say Python's large scale adoption is exactly a good example of dynamic typing delivering on its promises.
kortex 5y ago
Static typing in python isn't "cargo culting." In fact I'd go even farther in the opposite direction, and say that if you are using python for production software engineering, and aren't typing most (>50%) of your function signatures, you are wasting everybody's time, your teams', code reviewers', and most of all your own time. Opinion, but empirical experience bears this out for me. I'm not talking mypy --strict here, heck I've yet to achieve that myself. But if your are writing functions with obvious realizations of interfaces and not typing them, all you are doing is creating more mental strain for the next consumer (often future you) of that function down the road. At $LASTCO I inherited some jupyter-notebook-copy-pasted datascience ML abomination of a pipeline. Dicts everywhere, mutation everywhere, zero docstrings, nary a type hint in sight. Took two exceptionally stressful months, with constant back and forth with the authors, to get it working in prod ("it works on my machine, I don't understand the problem"). $THISCO embraces type hints, functional style, etc. My stress level has actually normalized. Typing `foo(bar: Optional[float])` takes what, 2s, more than `foo(bar)`? Asking "hey Steve, what does "bar" take in function "foo"? on slack is already more time and characters than just annotating it, times every dev that doesn't know and has to ask.
- maleldil 5y ago
  Type hints just make everything so much easier. When you're writing a function, you _know_ that parameter is an int, or float, or something you can use a list comprehension on (iterable). Why not just say it right there in the function definition? It's better than writing it in a docstring, because a type checker will tell you to change the type if you change how you use a variable. Does everyone need to go all the way and type 100% of things and use heavily generic code to represent all possible cases? Well, that would be wonderful, but just sprinkling built-in types is already a massive improvement over no types at all.
  - janto 5y ago
    A sprinkling of type annotations to help with ambiguity is nice. However, I don't recall the last time I spent ages figuring out what type I need to pass to a function if it simply wants a builtin type. It's usually the semantics or a library-unique type (ugh!) that I have to look up.
    - icedchai 5y ago
      Definitely. This to be especially terrible with some libraries (example: sqlalchemy) which have a crap load of types. You're not really sure what's being returned, etc.
- janto 5y ago
  The problem with creating excessive types in production systems (which is what I want people to avoid and can happen if you go too far with Pydantic models + type annotations into pseudo-"static typing") is that you move outside the area that Python's core is already designed to generically cover in idiomatic ways. You end up wasting your own (and reviewers') time creating unnecessary code that future you will have to now maintain and extend to more generic cases. I'm skeptical that the people that produce the kind of abomination you encountered would produce something less horrible if they went to town with static typing. I've seen too much Java to fall for that ;) I'm not averse to putting in some annotations and data validation chokepoints here and there. I'd ask Steve why he called it `foo(bar)` rather than `set_width(width)`. Static typing fans shift the semantics of code onto a type system. I'd prefer it if they focused more on better naming than spending time on designing complicated Pydantic models.

by janto 5y ago

smohare 5y ago

I think the comment about types takes a rather narrow view of Python use cases. Sure, if you’re some lone sysadmin using Python as an alternative to bash to implement some questionable ETL pipeline comprising a bunch of random scripts running off a single machine somewhere then I agree. You probably are not the target audience for type annotations. But even moderately complex libraries and applications are such a huge pain to develop, read, and maintain without some of the tooling that leverages type annotations. This is especially true in the ML / AI / Data Science world where a lot of the people implementing models have dubious coding practices. Pure dynamic typing paradigms simply have not delivered on their promises over the last 30 years. There are definitely some areas where they make sense, but I doubt we will see a massive readoption until our tooling becomes sufficiently intelligent. Imagine, for example, a probabilistic type inference based on both the structural aspects of the code and previous runs over actual data.

janto 5y ago
People in the data science world with dubious coding practices should keep their data in commonly used types such as Pandas tables. The last thing we should want from them is encoding their data with ill-conceived class/type systems. Not sure if that is what you mean. I'd say Python's large scale adoption is exactly a good example of dynamic typing delivering on its promises.

kortex 5y ago

Static typing in python isn't "cargo culting." In fact I'd go even farther in the opposite direction, and say that if you are using python for production software engineering, and aren't typing most (>50%) of your function signatures, you are wasting everybody's time, your teams', code reviewers', and most of all your own time. Opinion, but empirical experience bears this out for me. I'm not talking mypy --strict here, heck I've yet to achieve that myself. But if your are writing functions with obvious realizations of interfaces and not typing them, all you are doing is creating more mental strain for the next consumer (often future you) of that function down the road. At $LASTCO I inherited some jupyter-notebook-copy-pasted datascience ML abomination of a pipeline. Dicts everywhere, mutation everywhere, zero docstrings, nary a type hint in sight. Took two exceptionally stressful months, with constant back and forth with the authors, to get it working in prod ("it works on my machine, I don't understand the problem"). $THISCO embraces type hints, functional style, etc. My stress level has actually normalized. Typing `foo(bar: Optional[float])` takes what, 2s, more than `foo(bar)`? Asking "hey Steve, what does "bar" take in function "foo"? on slack is already more time and characters than just annotating it, times every dev that doesn't know and has to ask.

maleldil 5y ago
Type hints just make everything so much easier. When you're writing a function, you _know_ that parameter is an int, or float, or something you can use a list comprehension on (iterable). Why not just say it right there in the function definition? It's better than writing it in a docstring, because a type checker will tell you to change the type if you change how you use a variable. Does everyone need to go all the way and type 100% of things and use heavily generic code to represent all possible cases? Well, that would be wonderful, but just sprinkling built-in types is already a massive improvement over no types at all.
- janto 5y ago
  A sprinkling of type annotations to help with ambiguity is nice. However, I don't recall the last time I spent ages figuring out what type I need to pass to a function if it simply wants a builtin type. It's usually the semantics or a library-unique type (ugh!) that I have to look up.
  - icedchai 5y ago
    Definitely. This to be especially terrible with some libraries (example: sqlalchemy) which have a crap load of types. You're not really sure what's being returned, etc.
janto 5y ago
The problem with creating excessive types in production systems (which is what I want people to avoid and can happen if you go too far with Pydantic models + type annotations into pseudo-"static typing") is that you move outside the area that Python's core is already designed to generically cover in idiomatic ways. You end up wasting your own (and reviewers') time creating unnecessary code that future you will have to now maintain and extend to more generic cases. I'm skeptical that the people that produce the kind of abomination you encountered would produce something less horrible if they went to town with static typing. I've seen too much Java to fall for that ;) I'm not averse to putting in some annotations and data validation chokepoints here and there. I'd ask Steve why he called it `foo(bar)` rather than `set_width(width)`. Static typing fans shift the semantics of code onto a type system. I'd prefer it if they focused more on better naming than spending time on designing complicated Pydantic models.