12 ms·
Python types have an expectations problem
- tyingq 3y agoThere are some other warts also, like hard-to-grok errors if you're using typing syntax that's not yet supported in your version of python. Like: somevar: dict[int, int] = {0: 0, 1: 1} Produces "TypeError: 'type' object is not subscriptable"
- miohtama 3y agoYep, it's a good catch. Hard-to-grok errors tend to appear for every language and every backwards-compatible change unless the language itself specifies the target version in the source code file itself. The only language I know is doing this is Solidity.
- sco1 3y agoFor context, if others aren't familiar: type hinting generics were added to Python 3.9 by PEP 585 (also available in Python 3.7+ with the annotations future import). PEP 484 previously added type hints to Python 3.5 as explicit imports from the stdlib's typing module. https://peps.python.org/pep-0585/ https://peps.python.org/pep-0585/ https://peps.python.org/pep-0484/ https://peps.python.org/pep-0484/
- misnome 3y agoUnless you both a) Don't evaluate them at runtime, and b) import annotations from __future__, in which case it is generally safe to write them like this (and AFAIK all of the usual type-checking tools read them fine).
- tyingq 3y agoNope. See https://bugs.python.org/issue45117 https://bugs.python.org/issue45117
- misnome 3y agoYep. This is explicitly what I listed as "as long as you don't (a), evaluating at runtime". This extremely specific and rare case is outside of the scope of "Generally, this works". It means that some, very few definitions still need to be in old style, but the majority of your code is fine.
- tyingq 3y agoI didn't say you couldn't make it work by writing it differently. The point was obscure errors can pop up, which is a drawback to the way they chose to do things. That holds.
- teddyh 3y agoSummarizing quote: “Python type hints are a core part of the language, they even have standard library modules (typing), and yet they don’t do anything when used in that language without some external tooling. That, to me, is a bit of an expectations mismatch.” I.e. they’re complaining that a type checker like mypy isn’t run by default.
- miohtama 3y agoIt's also good to note that all popular editors and IDEs like Visual Studio Code and PyCharm support type hints quite well by default. Some external tooling needed, but it's your editor and you are going to have any case.
- olejorgenb 3y agoPyCharm unfortunately does not supports type hints fully. It certainly utilize the hints, but there's a bunch of things it simply does not understand which make hinting much less useful when using PyCharm.
- d0mine 3y agoCould you provide an example?
- pphysch 3y agoI was a bit surprised when I was diving into the Django type hints and realized that it was VSCode, not Django library, supplying some of the types via a "stubs" feature that includes extra type support for popular libraries like Django and pandas, the latter of which wasn't even installed on my system.
- empath-nirvana 3y agoI think it's a reasonable complaint and I think that there should be some "strict" annotation that forces python to do that type checking before running the code, which should be completely backwards compatible. Let's think of this in a deploy environment like kubernetes -- if you don't have such a check, then you could deploy code that fails _at runtime_, causing an outage, because someone made a mistake with types. If you fail _at startup_, then the deploy will never go healthy and will fail, leaving the old pods still running, causing no interruption in service. And yes you _should_ have this check in CI, but there's no reason not to have defense in depth.
- ehutch79 3y agoA note; typescript does nothing to ensure types are correct at runtime. Especially in the browser. You need to do runtime checking even if you're using typescript. In Python, yeah, they're called Type _Hints_ for a reason. Don't count on them at runtime here either. Both are dynamic languages, it's hard doing anything meta/schema driven with rigid types. If you really want a hard type system, just move to GO or Rust or C or something with a real type system enforced.
- thadt 3y agoC's type checker is the MMU - when you mess up types, it give you a segfault. If you're lucky.
- ehutch79 3y agoFair.
- dboreham 3y agoFunny, but for the readers who don't know C -- this isn't true. C has type checking at compile time.
- robertlagrant 3y agoIt has something, but not what you'd call modern type checking: #include <stdio.h> int main(void) { unsigned int positive_number = -1; printf("%d", positive_number); return 0; } Prints -1.
- maxfurman 3y agoI must be missing something here. Wouldn't an unsigned int be unable to represent -1 since the sign bit is part of the value?
- SAI_Peregrinus 3y agoC implicitly converts integer types as needed. The specific conversion rules are rather convoluted, see section 6.3 of the ISO C standard for the details. Before C23 this was implementation defined, but since C23 now mandates two's complement representation of signed integers and wrapping for unsigned overflow (as before) the `-1` is converted to UINT_MAX. Then the `%d` format specifier performs a conversion from `unsigned int` back to `int` for printing, resulting in `-1`. Implicit conversions between types of the same rank are guaranteed to "round-trip", you get back the value you started with when converting back to the starting type. If they'd specified an unsigned format specifier for printing then they wouldn't get `-1` printed.
- PurpleRamen 3y agoSo basically all they want is a wrapper for python which checks first with mypy if a script should be executed, and throws an error otherwise?
- sesgoe 3y agoFrom my experience working on an older Python codebase, this issue is definitely a headache. It's extremely difficult to gradually adopt typing in an older Python codebase with almost no typing information because the only real "enforcement" option seems to be a CI pipeline running something like `mypy`. This issue compounds in a painful way. Because 99% of your codebase is starting out untyped, you have a couple of options, neither of which I have found to be practically very useful. For the first option, you can run a blanket `mypy` invocation on your entire codebase and have a massive blast of errors you ignore for some time. Because it's necessarily going to error in the beginning, you can't really fail your CI pipeline as a result of this yet. If you can convince your team to gradually improve typing or set a deadline for eventual CI failure based on types, you might be able to move the needle and eventually get your codebase typed. From my experience though, this basically just became a CI step everyone ignored, and for people on the team not passionate about typing, they never worried about it. The other option, which is far more annoying in practice, is to pick a few "seed" files in your codebase that you can add typing information to quickly. Then, supplement your `mypy` invocation with a list of these seed files so it becomes something like `mypy file1.py file2.py ...` As you continue to improve typing "at the edges" of your codebase, you gradually add more and more files to the `mypy` invocation until you're eventually (hopefully) adding entire subfolders, and then maybe eventually the entire codebase. Starting at the edges means you can enforce the CI check from the beginning and get value quickly. The issue here is mostly remembering to continue to add files to the `mypy` invocation, which means you're constantly altering your CI pipeline. You know how when you alter a CI command it breaks sometimes because you got the encantation slightly wrong? Multiply this effect across basically every member of your team 1x a week because most people probably haven't edited your CI pipeline before. With even a small team (~7-10) making constant changes to a codebase, this quickly becomes extremely painful, and pipeline failures start eating a significant chunk of time just trying to debug if the encantation is wrong or if the types are actually broken. We mitigated this by having only one dev add new files to the `mypy` invocation, which worked well for the CI side of the story. The local side of the story is what ultimately led to enough fatigue to give up. It was hard to get in the rhythm of using local `mypy ...` invocations to check your types as you made changes, and so the experience for most of our team was to push changes, and then the types would break in CI, which was frustrating. They'd go in and try to fix it, and sometimes Python typing gets weird, and a fix wasn't immediately obvious. Eventually you get to `#type: ignore` or `Any`s being thrown around to sidestep the CI pipeline, and your typing story has collapsed again. The real kicker for us was the painful juxtaposition between `mypy` and `import`s. Is the giant swath of errors I'm seeing from this file or from a file I imported? Asking the entire team to become Python typing gurus to sort out these issues was a non-starter. Does anyone have experience gradually adopting Python typing in a large, older codebase successfully? If so, would you mind sharing the methodology you found success with?
- simonw 3y ago"I’d have to set up some CI with the type checking step and make it impossible to deploy the code that doesn’t pass it, maybe." Yes. That's how people successfully use type checking in Python.
- lpapez 3y agoI did that for Python, and now I do it for PHP as well. Trivial to set up but ends up being a huge time saver, especially when reviewing junior colleagues code - don't even ping me to review your code until you've managed to convince the static analyzer it will work!
- magnusmundus 3y agoRe: junior colleagues, do you not instead arrive at "why does the static analyzer complain", or worse, an avoidance of proper types that you then have to point out in review?
- lpapez 3y agoYou definitely do, for the first few times. But after that they learn to write code which makes both the static analyzer and the reviewer happy, and such code tends to be much more maintainable down the line. When you don't have strict typing discipline enforced by the CI, you will likely have to enforce it manually anyway because projects in a gradually typed language without simple types enforced tend to become a complete mess IMO.
- deleted 3y ago[deleted]
- posix_monad 3y agoIn Python it's common to have functions whose return types depend on the run-time arguments. For example: def fooify(x): if isinstance(x, list): map(fooify, x) else: x * 2 I guess this is to make scripting more forgiving. In typical typed languages, you would have two functions instead: fooify : number -> number fooifyMany : [number] -> [number] But in the Python community, it's common to have a big function with many behaviors. However, then the type annotations cannot be so precise: fooify : any -> any Not an experienced Python dev, so curious how this works in practice.
- yoyohello13 3y agoYou can annotate OR types in python so in this case you could do def fooify(x: int | list[int])
- pharmakom 3y agoBut if x is an int, the result is an int If x is a list, the result is a list I don’t think that the type system can describe this.
- olejorgenb 3y agoIt can: https://news.ycombinator.com/item?id=39179144 https://news.ycombinator.com/item?id=39179144
- evrimoztamur 3y agoThat could be a Union[float, list[float]]. Union types are very common! In fact, I think TypeScript will, for your given example, with the 'else' clause accurately identify the type of x to be a float if it's a union like the one I wrote down above.
- taeric 3y agoSorta? The function does return a union type, in isolation. At most callsites, you would know which of the two you are getting. This is much closer to generic invocation, if I remember the name correctly. Was very common in a lot of older dynamic languages. In fact, in a lot of languages, you can't tell this is a float, statically. It would work with whatever type was passed in that supports multiplication. And return the appropriate type. Right?
- mg 3y agoI don't really like types on the function declaration line. Instead of this: def check_permission(user: User, perm: str, obj: BaseModel | None) -> bool: I find it much nicer to read something this: """ Check if user is allowed to drive a car :user: User # The application's User model :perm: str # Our magic permission string. Tab seperated. :obj : Basemodel | None # Base model of all our objects since 2017 :returns bool # True if the user is allowed to drive """ def check_permission(user, perm, obj): This way I can grok the code much faster and only have to look into the type declaration when I want to. Due to syntax highlighting, it will look really nice. Because the whole type part can be styled in a dimmer color which puts it into the background. And I can define a key combo to show/hide the whole type part.
- Malcolmlisk 3y agoYou can do that as a docstring inside the function. There are even some basic rules like... def extract_coordiates( unreseted_idx_from_df, json_data: dict, notna_idxs: list, na_idxs: list ) -> list: """ Extracting coordinates from the json_data['data'] and returning the 4 positions of those as an array Parameters ---------- json_data: json Data extracted from tabula.read_pdf and using output_format='json' notna_idxs: list List of id's from the dataframe that ha no na values. na_idxs: list List of id's from the dataframe with na Returns ------- List of tuple coordinates: [(1, 2), (2, 2), (4, 3), (1, 4)] """ Thing is, that you may want to get fast information in the linter with this type hinting. If you need to read the documentation on big functions over and over, that means that's not clear at all, and having basic type hinting while typing the atributes is going to be clearer. You are right, that some docstrings are necesary. But that does not mean that is the best practice. The best practice is use both, type hinting and docstrings.
- zokier 3y agoI think you misinterpreted parent comment. My reading was that they would have preferred type-hints to be defined as part of docstring style comments instead of having the type hints inline with the code. From tooling point of view it doesn't make any difference, both forms should be functionally equivalent.
- ringofchaos 3y agoI have been using python library Pydantic with FastApi for input and output data validation for Rest API and it solves some of the problems. I wanted to check if the use case can be generalized for all situations. For example, the code below will throw a runtime error * Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='A', input_type=str] * ``` from pydantic import BaseModel class User(BaseModel): id: int name: str def print_user(user: User): print(user.name, user.id) user_1 = User(id="A", name="John") print_user(user_1) ``` I have also been using typescript with React Application and really find it better compared to python due to type checking. But I can still imagine a future when type-checking gets incorporated in native python, unlike javascript/typescript. The groundwork has been laid already with type hints.
- ReflectedImage 3y agoUsing static typing in Python is a serious mistake. The point of static typing is to allow an compiler to compile your code for performance reasons and nothing else. People do not normally compile Python code with a compiler. Static typing significantly increases code length compared to duck typing, significantly increases development times and significantly increases bug count per delivered software feature. It additionally gives developers the false impression that you can write large projects in Python without splitting the codebase into small micro-services. Hint: You can't, it is always a complete disaster as Python's support for static typing isn't good enough for large projects.
- empath-nirvana 3y ago> significantly increases bug count per delivered software feature. Going to need a source for this.
- ReflectedImage 3y agoIt should be fair obvious that when you use a very verbose very boiler plate heavy coding style like static typing that because the programmer is writing a lot more lines of code per software feature and because the programmers make errors on a per line written basis that the number of the bugs will drastically increase. You can easily find information on duck typed programs being significantly shorter and that statically typed and ducked typed programs have the same bug count per line. You can also look at the language defective tables showing ducked type Clojure as having the least bugs in commerical software and statically typed C++ having the most bugs in commerical software. Google is your friend here.
- dolni 3y agoI have written type-annotated Python for as long as type annotations have been a thing. "Number of bugs will drastically increase" is simply wrong. Type checking prevents bugs before they ship.
- 3y ago
- pbhowmic 3y agoThere is no protecting against pre conceived notions. If a Typescript programmer, for example, expects Python to behave like Typescript then that is on them. Typescript has types. Python has type hints. RTFM for God’s sake.
- noone_youknow 3y agoSounds to me like the author has the “expectations problem”. I’m not a huge fan of type hints (though they are useful especially with modern IDEs) but it seems a stretch to say a thing is inherently flawed simply because it doesn’t work the way I expect it to…
- amomchilov 3y agoSorbet is a really interesting point in this space, because in addition to its static type checker, it has a runtime component which can do runtime type checking. It installs shims around all your methods which type check arguments on the way in, and results on the way out. Of course this comes with a performance penalty, so it’s often enabled for development and testing, but disabled in production.