7 ms·

Using type checking in Python is always the wrong thing to do. You need to learn to use Microservices where you split up a large code base into seperate 3k line

Using type checking in Python is always the wrong thing to do. You need to learn to use Microservices where you split up a large code base into seperate 3k lines Microservice chunks of code that interact over well documented APIs. That's just how effective Python is done.

codekansas 4y ago
I don't think you're understanding their use case, they aren't building some Flask service, they're an ML drug discovery company supporting research scientists and as such they're more or less building a framework. There's a lot of code reuse and most large orgs building similar frameworks have adopted similar code quality standards - think PyTorch, Fairseq, Pytorch Lightning, Huggingface - which is why tools like MyPy / PyLint / Black exist. For people who have used those libraries there is a night-and-day difference between the anything-goes ones and the ones that do linting and unit testing.
- ReflectedImage 4y ago
  In my personal experience codebases using MyPy tend to be considerably worse than codebases that don't. By using a static type checker on a dynamically typed language, you have admitted you don't know what are you doing right of the bat. This means the software engineers in the project are very bad and therefore the code quality overall will also be bad. Tools like MyPy exist to make Python appear to be more like Java to help Java developers, who aren't willing to learn how to code in a different programming paradigm. That code will always be far worse than Python code written using the Python development paradigms.
  - kortex 4y ago
    What are some examples of popular python libraries that use this ideal python development paradigm?
    - ReflectedImage 4y ago
      Most of the Python libraries that are not locked to specific versions of Python, which is often the case with a lot of the badly written ML libraries.
  - throwaway343244 4y ago
  - LtWorf 4y ago
    > This means the software engineers in the project are very bad and therefore the code quality overall will also be bad. Conversely, I do think you're very bad and don't have much experience with large python codebases. For example the suggestion to split and use microservices makes no sense. Microservices are even harder to refactor.
  - eloff 4y ago
    Actually now I'm thinking you might be trolling.
CleverLikeAnOx 4y ago
I don't think one size fits all. There are plenty of successful python projects that don't match this. E.g., every python library (numpy as microservices doesn't make sense). Plenty of large successful Django apps exist. I am using typing effectively in solo dev flask webapp that doesn't need to be split into microservices. Sorry if your post was sarcastic and it went over my head.
- ReflectedImage 4y ago
  I don't think Numpy is mostly written in Python. It's C. So using type hints to annotate your flask api, etc. to generate docs is great. No problem. The issue is that there have been a lot of Java developers switching to Python and they want to pretend that Python is a statically typed language (and use tools like MyPy) because it's more familar to them. Python is not a statically typed language and treating it as one leads to terrible results. Especially in large projects. The issue of course if that the Java developers have no point of reference on what a successful large Python project looks like. So they think they are doing great with MyPy when if you compare what they are outputting to a proper large Python project, it's pretty clear they are doing terribly. When you use Python properly you write self contained microservices. Typing information exists but only on the external interfaces e.g. API. You also don't share business logic code between the self contained microservices because that massively decreases the maintainabily of the overall system. You show your Python code is correct by using Unit Testing and Mocking. Basically, there is a way to do large Python projects and MyPy (and other static type checking tools) have no place in that story. It only exists to support people in making bad decisions on their codebase.
  - Mehdi2277 4y ago
    Numpy has a large amount of python code. C is definitely used in places, but majority of numpy is python. Not all operations need C implementations especially as many can be built on top of other python functions which do wrap C. Tensorflow/pytorch similarly have a large amount of python code. I mostly work on developing a library that builds on tensorflow and is like 50K lines of code. Dividing that library into services makes little sense. One of the recent typing new features is mostly devoted to numerical ecosystem in python (variadic generics to do template like types for matrices). I don't want to use dynamically typed language, but python is clear leader in ML ecosystem. A lot of my department does work that builds on ML research/libraries and those are mostly found in python. If other languages had comparable ML ecosystems then a different language may have been chosen. But at moment today there's no competitor anywhere close. One lazy metric, I would estimate 90%+ of research papers to ML conferences are in python.
yawaramin 4y ago
You forgot to end with /s
- ReflectedImage 4y ago
  I've seen Python projects using Monorepo MyPy and Python projects using Microservices. Python Microservices style when it comes to code quality, maintainability, etc... wipe the floor with Monorepo MyPy style projects. It's not even close.
  - kortex 4y ago
    And I've seen the exact opposite. Messy, poorly/under/incorrectly-documented buggy microservices which barf on corner cases, painful to refactor, no way to verify correctness without tons of unit and integration tests. Conversely, I've seen huge heavily-typed mono repos which are a breeze to operate on, wrap my head around, jump to definitions, automatically refactor, and actually run with confidence. So do our anecdotes cancel out?
    - ReflectedImage 4y ago
      No, because you can always replace bad microservices wholesale. It can't be "painful to refactor", because you literally just delete the code and rewrite it. If you have introduced static typing, you then have to start doing refactoring and verifying correctness. "heavily-typed mono repos which is a breeze to operate on" I highly doubt such a thing exists, more likely you are used to a certain level of bad code and don't understand it can be better.
      - kortex 4y ago
        > I highly doubt such a thing exists, more likely you are used to a certain level of bad code and don't understand it can be better. Ohhh trust me, I know bad code. And I know good code. My unit of measurement is how fried I feel at the end of the day. Dynamic loosy-goosy python? Brain fried, constant debugging, little confidence in deploys. Static types, pydantic, pycharm, mypy, DI? I'm in the zone all day. I don't think you are the arbiter of all code, so I'm not sure what grounds you have to tell me what my taste in code is. Additionally, you would be better served by writing comments with less presumption in them. It makes the discourse more adversarial than it needs to be.
thunky 4y ago
This is sarcasm, right?
Barrin92 4y ago
the enraged responses are pretty funny given that this is literally just what the BEAM vm is conceptually. Trying to bolt a static type system on a dynamic object oriented language literally violates any benefit you get from using a dynamic language in the first place. (that is to say, change things as they run). I have no idea why people try to enforce the programming paradigm of Java on Python. if you want a huge, static program... write it in a static language, don't write it in Python.
- geekraver 4y ago
  If I have a function that returns something in a list, why not declare it as such? The pain in Python typing comes more from bolting a largely (but not entirely) nominal type system on a structurally typed language. That can be addressed by Protocols.
eloff 4y ago
That's horrific. The crazy thing is I think you might be serious and not trolling. It's like something out of a Dilbert comic. Typing is good, anyplace you can get the computer to check more of your work - the better. There are practical limits and trade-offs as always, but some typing is better than none. Microservices are usually a terrible idea. I know they're popular, but I've only had bad experiences with them. I strongly recommend against microservices in most situations.

codekansas 4y ago

I don't think you're understanding their use case, they aren't building some Flask service, they're an ML drug discovery company supporting research scientists and as such they're more or less building a framework. There's a lot of code reuse and most large orgs building similar frameworks have adopted similar code quality standards - think PyTorch, Fairseq, Pytorch Lightning, Huggingface - which is why tools like MyPy / PyLint / Black exist. For people who have used those libraries there is a night-and-day difference between the anything-goes ones and the ones that do linting and unit testing.

ReflectedImage 4y ago
In my personal experience codebases using MyPy tend to be considerably worse than codebases that don't. By using a static type checker on a dynamically typed language, you have admitted you don't know what are you doing right of the bat. This means the software engineers in the project are very bad and therefore the code quality overall will also be bad. Tools like MyPy exist to make Python appear to be more like Java to help Java developers, who aren't willing to learn how to code in a different programming paradigm. That code will always be far worse than Python code written using the Python development paradigms.
- kortex 4y ago
  What are some examples of popular python libraries that use this ideal python development paradigm?
  - ReflectedImage 4y ago
    Most of the Python libraries that are not locked to specific versions of Python, which is often the case with a lot of the badly written ML libraries.
- throwaway343244 4y ago
- LtWorf 4y ago
  > This means the software engineers in the project are very bad and therefore the code quality overall will also be bad. Conversely, I do think you're very bad and don't have much experience with large python codebases. For example the suggestion to split and use microservices makes no sense. Microservices are even harder to refactor.
- eloff 4y ago
  Actually now I'm thinking you might be trolling.

CleverLikeAnOx 4y ago

I don't think one size fits all. There are plenty of successful python projects that don't match this. E.g., every python library (numpy as microservices doesn't make sense). Plenty of large successful Django apps exist. I am using typing effectively in solo dev flask webapp that doesn't need to be split into microservices. Sorry if your post was sarcastic and it went over my head.

ReflectedImage 4y ago
I don't think Numpy is mostly written in Python. It's C. So using type hints to annotate your flask api, etc. to generate docs is great. No problem. The issue is that there have been a lot of Java developers switching to Python and they want to pretend that Python is a statically typed language (and use tools like MyPy) because it's more familar to them. Python is not a statically typed language and treating it as one leads to terrible results. Especially in large projects. The issue of course if that the Java developers have no point of reference on what a successful large Python project looks like. So they think they are doing great with MyPy when if you compare what they are outputting to a proper large Python project, it's pretty clear they are doing terribly. When you use Python properly you write self contained microservices. Typing information exists but only on the external interfaces e.g. API. You also don't share business logic code between the self contained microservices because that massively decreases the maintainabily of the overall system. You show your Python code is correct by using Unit Testing and Mocking. Basically, there is a way to do large Python projects and MyPy (and other static type checking tools) have no place in that story. It only exists to support people in making bad decisions on their codebase.
- Mehdi2277 4y ago
  Numpy has a large amount of python code. C is definitely used in places, but majority of numpy is python. Not all operations need C implementations especially as many can be built on top of other python functions which do wrap C. Tensorflow/pytorch similarly have a large amount of python code. I mostly work on developing a library that builds on tensorflow and is like 50K lines of code. Dividing that library into services makes little sense. One of the recent typing new features is mostly devoted to numerical ecosystem in python (variadic generics to do template like types for matrices). I don't want to use dynamically typed language, but python is clear leader in ML ecosystem. A lot of my department does work that builds on ML research/libraries and those are mostly found in python. If other languages had comparable ML ecosystems then a different language may have been chosen. But at moment today there's no competitor anywhere close. One lazy metric, I would estimate 90%+ of research papers to ML conferences are in python.

yawaramin 4y ago

You forgot to end with /s

ReflectedImage 4y ago
I've seen Python projects using Monorepo MyPy and Python projects using Microservices. Python Microservices style when it comes to code quality, maintainability, etc... wipe the floor with Monorepo MyPy style projects. It's not even close.
- kortex 4y ago
  And I've seen the exact opposite. Messy, poorly/under/incorrectly-documented buggy microservices which barf on corner cases, painful to refactor, no way to verify correctness without tons of unit and integration tests. Conversely, I've seen huge heavily-typed mono repos which are a breeze to operate on, wrap my head around, jump to definitions, automatically refactor, and actually run with confidence. So do our anecdotes cancel out?
  - ReflectedImage 4y ago
    No, because you can always replace bad microservices wholesale. It can't be "painful to refactor", because you literally just delete the code and rewrite it. If you have introduced static typing, you then have to start doing refactoring and verifying correctness. "heavily-typed mono repos which is a breeze to operate on" I highly doubt such a thing exists, more likely you are used to a certain level of bad code and don't understand it can be better.
    - kortex 4y ago
      > I highly doubt such a thing exists, more likely you are used to a certain level of bad code and don't understand it can be better. Ohhh trust me, I know bad code. And I know good code. My unit of measurement is how fried I feel at the end of the day. Dynamic loosy-goosy python? Brain fried, constant debugging, little confidence in deploys. Static types, pydantic, pycharm, mypy, DI? I'm in the zone all day. I don't think you are the arbiter of all code, so I'm not sure what grounds you have to tell me what my taste in code is. Additionally, you would be better served by writing comments with less presumption in them. It makes the discourse more adversarial than it needs to be.

thunky 4y ago

This is sarcasm, right?

Barrin92 4y ago

the enraged responses are pretty funny given that this is literally just what the BEAM vm is conceptually. Trying to bolt a static type system on a dynamic object oriented language literally violates any benefit you get from using a dynamic language in the first place. (that is to say, change things as they run). I have no idea why people try to enforce the programming paradigm of Java on Python. if you want a huge, static program... write it in a static language, don't write it in Python.

geekraver 4y ago
If I have a function that returns something in a list, why not declare it as such? The pain in Python typing comes more from bolting a largely (but not entirely) nominal type system on a structurally typed language. That can be addressed by Protocols.

eloff 4y ago

That's horrific. The crazy thing is I think you might be serious and not trolling. It's like something out of a Dilbert comic. Typing is good, anyplace you can get the computer to check more of your work - the better. There are practical limits and trade-offs as always, but some typing is better than none. Microservices are usually a terrible idea. I know they're popular, but I've only had bad experiences with them. I strongly recommend against microservices in most situations.