4 ms·

i've often thought a more interesting approach for python's duck typing would be if the interpreter looked for anomalous types on assignment. so build a compact

by a-dub 3y ago

i've often thought a more interesting approach for python's duck typing would be if the interpreter looked for anomalous types on assignment. so build a compact distribution of type "shape" for every assignment and then either throw exceptions or log warnings when apparent outliers are encountered.

pepoluan 3y ago
Use lots of typing.Protocol and you get all the goodness of duck-typing.
MrJohz 3y ago
That's roughly a structural type system, and it's part of what makes Typescript work well (although it has some disadvantages in that you can't easily describe the type "an object with only these attributes). You can somewhat get that effect with Protocols in Python, but it often feels a bit boilerplate-y to me.
- a-dub 3y ago
  sortof. except two differences: 1) i'm primarily interested in data types, rather than objects, as i believe most runtime errors are the result of unexpected changes to data types. 2) i'm suggesting a type system that does away with hand annotations and rather builds them on the fly by keeping assignment history and using that to determine when statistically anomalous types appear.
  - MrJohz 3y ago
    That would be a structural type system with Hindley-Milner type inference. A good example of that might be OCaml - it's largely structurally typed, and the types can be completely inferred without type annotations. However, while Hindley-Milner will find all the type errors in your program, it won't necessarily report those errors in the most logical places. That's why it's generally still the norm to annotate functions with type annotations except in the most trivial cases: knowing that a function will take a string and return a number is useful information not just for the compiler, but also the programmer. To come back to your first point, one common solution here is the "parse, don't validate" mantra for loading data. This is the idea that when accepting unknown data (e.g. from a file or API), you first parse the data and convert it into a representation that enforces the invariants your program believes about that data. For example, that names can never be longer than twenty characters, or that ages are always positive numbers. By doing this parsing as soon as the data enters your program, you immediately handle unexpected changes to the data formats - if data changes such that it can no longer be parsed, this is an immediate error, rather than something that will only show deep in your application. With something like Pydantic (alternatives are available), you can describe what your data should look like in terms of types, and use Pydantic to do the parsing for you.
    - a-dub 3y ago
      cool library. although worth keeping in mind that strong typing doesn't solve all your problems. here's an example, taking the second example on this page https://docs.pydantic.dev/latest/ https://docs.pydantic.dev/latest/ with the external_data dictionary, if i add a collision like: 'wine': 9, 'wine': 1, it seems to just silently pick the last one, which seems error prone to me. maybe there's a flag to fix it, but i couldn't find any documentation or switches that controlled this behavior. :/ with respect to the approach i mentioned above, it's more of an idea (or probably a research topic) to do quantitative analysis on the behavior of programs while they execute, rather than static analysis of the code that defines them. the latter is obviously preferable, assuming the approach can capture all of the types of analyses of interest, the quantitative approach may be more suitable for large or legacy systems where detecting changes in data shape (even if of valid type) and branching behavior may be useful for tracking down causes of, or detecting conditions for, errant behaviors.
      - MrJohz 3y ago
        Pydantic operates on Python dictionaries, so questions about duplicate keys will come down to whatever the Python behaviour is. (I.e. if you have a dictionary literal with duplicate keys, Python will select which key "wins" before Pydantic gets a chance to intervene.) When loading directly from JSON, it's possible to hook into Python's JSON-parsing mechanism and throw on duplicate keys, which probably ought to be the default given the number of security risks that come from this case.

by a-dub 3y ago

pepoluan 3y ago

Use lots of typing.Protocol and you get all the goodness of duck-typing.

MrJohz 3y ago

That's roughly a structural type system, and it's part of what makes Typescript work well (although it has some disadvantages in that you can't easily describe the type "an object with only these attributes). You can somewhat get that effect with Protocols in Python, but it often feels a bit boilerplate-y to me.

a-dub 3y ago
sortof. except two differences: 1) i'm primarily interested in data types, rather than objects, as i believe most runtime errors are the result of unexpected changes to data types. 2) i'm suggesting a type system that does away with hand annotations and rather builds them on the fly by keeping assignment history and using that to determine when statistically anomalous types appear.
- MrJohz 3y ago
  That would be a structural type system with Hindley-Milner type inference. A good example of that might be OCaml - it's largely structurally typed, and the types can be completely inferred without type annotations. However, while Hindley-Milner will find all the type errors in your program, it won't necessarily report those errors in the most logical places. That's why it's generally still the norm to annotate functions with type annotations except in the most trivial cases: knowing that a function will take a string and return a number is useful information not just for the compiler, but also the programmer. To come back to your first point, one common solution here is the "parse, don't validate" mantra for loading data. This is the idea that when accepting unknown data (e.g. from a file or API), you first parse the data and convert it into a representation that enforces the invariants your program believes about that data. For example, that names can never be longer than twenty characters, or that ages are always positive numbers. By doing this parsing as soon as the data enters your program, you immediately handle unexpected changes to the data formats - if data changes such that it can no longer be parsed, this is an immediate error, rather than something that will only show deep in your application. With something like Pydantic (alternatives are available), you can describe what your data should look like in terms of types, and use Pydantic to do the parsing for you.
  - a-dub 3y ago
    cool library. although worth keeping in mind that strong typing doesn't solve all your problems. here's an example, taking the second example on this page https://docs.pydantic.dev/latest/ https://docs.pydantic.dev/latest/ with the external_data dictionary, if i add a collision like: 'wine': 9, 'wine': 1, it seems to just silently pick the last one, which seems error prone to me. maybe there's a flag to fix it, but i couldn't find any documentation or switches that controlled this behavior. :/ with respect to the approach i mentioned above, it's more of an idea (or probably a research topic) to do quantitative analysis on the behavior of programs while they execute, rather than static analysis of the code that defines them. the latter is obviously preferable, assuming the approach can capture all of the types of analyses of interest, the quantitative approach may be more suitable for large or legacy systems where detecting changes in data shape (even if of valid type) and branching behavior may be useful for tracking down causes of, or detecting conditions for, errant behaviors.
    - MrJohz 3y ago
      Pydantic operates on Python dictionaries, so questions about duplicate keys will come down to whatever the Python behaviour is. (I.e. if you have a dictionary literal with duplicate keys, Python will select which key "wins" before Pydantic gets a chance to intervene.) When loading directly from JSON, it's possible to hook into Python's JSON-parsing mechanism and throw on duplicate keys, which probably ought to be the default given the number of security risks that come from this case.