3 ms·

ex-pytype dev here - we knew this was coming and it's definitely the right thing to do, but it's still a little sad to see the end of an era. in particular, pyt

by zem 1y ago

ex-pytype dev here - we knew this was coming and it's definitely the right thing to do, but it's still a little sad to see the end of an era. in particular, pytype's ability to do flow-based analysis across function boundaries (type checking calls to unannotated functions by symbolically executing the function body with the types of the call arguments) has not been implemented by any of the other checkers (again for good reasons; it's a performance hit and the world is moving towards annotations over pure inference anyway, but I still think it's a nice feature to have and makes for more powerful checking). as an aside, while I agree that bytecode-based analysis has its drawbacks, I think it's a tool worth having in the overall python toolbox. I spun off pycnite from pytype in the hope that anyone else who wanted to experiment with it would have an easier time getting started - https://github.com/google/pycnite https://github.com/google/pycnite I have recently jumped onto the "write python tooling in rust" bandwagon and might look into a rust reimplementation of pycnite at some point, because I still feel that bytecode analysis lets you reuse a lot of work the compiler has already done for you.

almostgotcaught 1y ago
> while I agree that bytecode-based analysis has its drawbacks abstract interpretation of the bytecode like y'all were doing is the only way to robustly do type inference in python. > https://github.com/google/pycnite https://github.com/google/pycnite there's also https://github.com/MatthieuDartiailh/bytecode https://github.com/MatthieuDartiailh/bytecode which is a good collection
- zem 1y ago
  yeah, that's a really nice project too!
- jpfr 1y ago
  MOPSA does abstract interpretation for both C and Python. It even works across language boundaries. https://mopsa.lip6.fr/#features https://mopsa.lip6.fr/#features It also has more abstraction domains than „just“ the type of objects.
librasteve 1y ago
why not just go with a language that has gradual typing built in - eg raku
- zem 1y ago
  because there is a ton of existing python code that people find a lot of value in, and that no one wants to abandon. the return on investment for making python better is insanely higher than that on porting hundreds of millions of lines of code to another language.
  - librasteve 1y ago
    that’s why raku has modules like Inline::Python Inline::Perl5 and strong FFI (Foreign Function Interface) chops certainly I see the economic sense in continuing with Python, but for some folks there’s a limit to how much lipstick you want on your pig
xenophonf 1y ago
> I have recently jumped onto the "write python tooling in rust" bandwagon I know Go and Rust are the belles du jour, but this kind of thing really hampers integrators' ability to support platforms other than x86-64 and armv8. In my particular case, it results in me being unable to build software that depends on pyca/cryptography on platforms like s390x, which makes me sad. It also makes development environment management, including CI/CD pipeline maintenance, that much more complicated. It was already bad enough when I was trying to compile binary distributions on Windows, and that was just with the Visual C++ toolchain mess that is the Microsoft development experience.
- zem 1y ago
  I can sympathise with that, but to argue a bit for the other side, these tools are mainly intended to run on the developer's machine or in the CI pipeline. in both cases they overwhelmingly use architectures that rust supports, and in the case of CI surely it's easier to deploy a single rust binary than a python binary and all its library dependencies. I used to be very much in the "write your language tools in the language and the community will contribute" camp, but astral has really shown what a big difference pure speed can make, and I now have more of a "write tools in rust and ideally make sure they also expose libraries that you can call from python" mindset.
- endgame 1y ago
  On top of all this, pretty much everything depends on Python (even glibc's build system depends on it now), and Rust is relatively hard to bootstrap. So bootstrapping a usable Python on glibc could one day involve bootstrapping all the way up to Rust on musl (including e.g. llvm) just to get Python just to build the final system's libc. That doesn't feel great to me.
  - zem 1y ago
    please note that I am not talking about introducing rust into the python interpreter (where, I agree, bootstrapping concerns would make the gain in code maintainability not really worth it), but in writing developer tools that work with python in rust or a mix of rust and python rather than in pure python. these are tools that run on the developer's machine or on test/ci servers, not in the target environment.
lostmsu 1y ago
I worked on https://github.com/Microsoft/PTVS https://github.com/Microsoft/PTVS (not being in MSFT) around 2019 so I know they did type check calls across function boundaries.
- zem 1y ago
  neat. did they also do it by symbolically executing the function body?
  - lostmsu 1y ago
    In a nutshell it would build a dependency graph first using simplified algorithm that only looked at imports. Then it would pick a function, eval types statement by statement (I think that's what you call symbolic execution), and if any named value as result would expand the list of possible types it could have, all the functions that use that value (global var, passed as argument, etc) would be queued for reevaluation. Until either the set of possible types for all values is steady or some reevaluation limit is reached.

by zem 1y ago

almostgotcaught 1y ago

> while I agree that bytecode-based analysis has its drawbacks abstract interpretation of the bytecode like y'all were doing is the only way to robustly do type inference in python. > https://github.com/google/pycnite https://github.com/google/pycnite there's also https://github.com/MatthieuDartiailh/bytecode https://github.com/MatthieuDartiailh/bytecode which is a good collection

zem 1y ago
yeah, that's a really nice project too!
jpfr 1y ago
MOPSA does abstract interpretation for both C and Python. It even works across language boundaries. https://mopsa.lip6.fr/#features https://mopsa.lip6.fr/#features It also has more abstraction domains than „just“ the type of objects.

librasteve 1y ago

why not just go with a language that has gradual typing built in - eg raku

zem 1y ago
because there is a ton of existing python code that people find a lot of value in, and that no one wants to abandon. the return on investment for making python better is insanely higher than that on porting hundreds of millions of lines of code to another language.
- librasteve 1y ago
  that’s why raku has modules like Inline::Python Inline::Perl5 and strong FFI (Foreign Function Interface) chops certainly I see the economic sense in continuing with Python, but for some folks there’s a limit to how much lipstick you want on your pig

xenophonf 1y ago

> I have recently jumped onto the "write python tooling in rust" bandwagon I know Go and Rust are the belles du jour, but this kind of thing really hampers integrators' ability to support platforms other than x86-64 and armv8. In my particular case, it results in me being unable to build software that depends on pyca/cryptography on platforms like s390x, which makes me sad. It also makes development environment management, including CI/CD pipeline maintenance, that much more complicated. It was already bad enough when I was trying to compile binary distributions on Windows, and that was just with the Visual C++ toolchain mess that is the Microsoft development experience.

zem 1y ago
I can sympathise with that, but to argue a bit for the other side, these tools are mainly intended to run on the developer's machine or in the CI pipeline. in both cases they overwhelmingly use architectures that rust supports, and in the case of CI surely it's easier to deploy a single rust binary than a python binary and all its library dependencies. I used to be very much in the "write your language tools in the language and the community will contribute" camp, but astral has really shown what a big difference pure speed can make, and I now have more of a "write tools in rust and ideally make sure they also expose libraries that you can call from python" mindset.
endgame 1y ago
On top of all this, pretty much everything depends on Python (even glibc's build system depends on it now), and Rust is relatively hard to bootstrap. So bootstrapping a usable Python on glibc could one day involve bootstrapping all the way up to Rust on musl (including e.g. llvm) just to get Python just to build the final system's libc. That doesn't feel great to me.
- zem 1y ago
  please note that I am not talking about introducing rust into the python interpreter (where, I agree, bootstrapping concerns would make the gain in code maintainability not really worth it), but in writing developer tools that work with python in rust or a mix of rust and python rather than in pure python. these are tools that run on the developer's machine or on test/ci servers, not in the target environment.

lostmsu 1y ago

I worked on https://github.com/Microsoft/PTVS https://github.com/Microsoft/PTVS (not being in MSFT) around 2019 so I know they did type check calls across function boundaries.

zem 1y ago
neat. did they also do it by symbolically executing the function body?
- lostmsu 1y ago
  In a nutshell it would build a dependency graph first using simplified algorithm that only looked at imports. Then it would pick a function, eval types statement by statement (I think that's what you call symbolic execution), and if any named value as result would expand the list of possible types it could have, all the functions that use that value (global var, passed as argument, etc) would be queued for reevaluation. Until either the set of possible types for all values is steady or some reevaluation limit is reached.