4 ms·

This is an honest question, what's the size of projects you have written in Python? For me, once a project gets beyond some size, the aggravation only grows of

This is an honest question, what's the size of projects you have written in Python? For me, once a project gets beyond some size, the aggravation only grows of debugging old code and adding new. Mostly, I see this is as a lack of compilation and types in Python. How do others overcome this and enjoy large Python projects?

andybak 9y ago
OK. Bear with me here. This might be overstating my case... In some sense - there are no large projects. Only projects that have failed at being modular. Maintainability involves keeping clear boundaries between functional units so that you can keep enough code in your head at any one time to reason about it. If we assume that Python is capable of handling more code than you can keep in your head at any given time, then surely the problem is simply one of making sure your modules are nicely self-contained from each other.
- vturner 9y ago
  Agreed, modularity definitely helps us minimize the mental load of understanding all the objects at once. But how does it handle this situation: Write function in module A that expects some complex data structure that comes in. In a static typed language, you put a type on the argument and thus guarantee, to some degree, the data in the object can be safely manipulated. Over in module B we call that function. Because Python doesn't enforce types, I can pass in some type of data structure that might behave well in many situations, but in some that function is going to fail because I didn't properly construct the data structure. How do other Python projects avoid this? Do they avoid large data structures of multiple fields?
  - outworlder 9y ago
    > Write function in module A that expects some complex data structure that comes in. In a static typed language, you put a type on the argument and thus guarantee, to some degree, the data in the object can be safely manipulated. Let's question some of the assumptions. Do you NEED the entire thing to be passed around? If your functions are small and only do one thing, can't they be fed the portion of the data they need and only manipulate that? If they are getting unnecessary data, soon someone will like the convenience and that piece of data will now become necessary. I have seen Java projects with humongous "DAO" objects being passed across all apps. Even if Java had Haskell's type system, at some point it breaks down. So, you do not enforce that on Python itself. Rather, you have to apply due diligence and ensure your data structures are clean, and functions do what they are supposed to do and no more. > I can pass in some type of data structure that might behave well in many situations, but in some that function is going to fail because I didn't properly construct the data structure. Then you construct the structure in a single place, and that should have some sanity checks and sane defaults. Even if you are using a static typed language, there's no guarantee that what's inside the data structure even makes sense, only that the right type of structure is being passed on. That's helpful, but not as much as people would think. And you write tests. Lots of tests. It is amazing the amount of Python code that gets written without proper testing. And Java, for that matter. Only Python programmers have less excuses, it's easier to replace what you need for testing.
    - simion314 9y ago
      >Even if you are using a static typed language, there's no guarantee that what's inside the data structure even makes sense. You should always put validation in constructor so you are 100% sure that the object created is valid.
      - xapata 9y ago
        How can you be sure your validation covered all cases?
        simion314 9y ago
        Say you have a Circle class, you need a center point so you validate that the center is not null and you check that radius is greater then zero. Do you have an example where is impossible to validate all cases? if so how will your unit tests validate/test all those cases?
        xapata 9y ago
        In your example, you can't anticipate that center points approximately but not quite equal to pi will cause a ... whatever bug in ... whatever downstream use of the class. One can never really know what states are acceptable.
        simion314 9y ago
        The circle is valid, what you say is a bug in other section that crashes with valid circles,so types could prevent invalid inputs to functions. What you describe is a bug in an algorithm I don't think that is possible for a compiler to prevent bugs in algorithms, division by zero or precision errors
        xapata 9y ago
        Type systems can do some pretty miraculous things. For example, you could distinguish between a connected and disconnected socket with types so that you don't accidentally send on a disconnected socket. I guess I'm arguing both sides now. My point is that one needs to root out the weird cases somehow and that it's often hard to have "100%" validation of inputs. One's initial understanding of what is the valid and invalid state space is often incorrect.
  - mring33621 9y ago
    Everything's a server, er, microservice, passing JSON back and forth!
    - marcosdumay 9y ago
      Microservices are great, because you can enforce static typing within their interfaces. \s
  - abc_lisper 9y ago
    Well, there's schema. https://pypi.python.org/pypi/schema/0.6.6 https://pypi.python.org/pypi/schema/0.6.6
  - asddddd 9y ago
    Twisted is a massive codebase, and uses zope.interface. There are definitely options for stricter guarantees, but it's optional, and generally you'd rather be treating input as ducks. Also, even without actual type checking (mypy), IDE support + type hinting (PEP 484 or reST et al) means you can find most issues before the interpreter/compiler is involved. PyCharm is fantastic at this.
  - crdoconnor 9y ago
    >How do other Python projects avoid this? https://github.com/jonathanslenders/pymux/blob/master/pymux/process.py#L54 https://github.com/jonathanslenders/pymux/blob/master/pymux/... Here's one way. Those asserts will typically pick up on errors during spikes, development and tests and turn a head scratching problem into a straightforward issue. A couple of times on a disastrously tech debt ridden project I used it to pick up obscure configuration errors in prod, too (shouldn't see it in prod on a relatively well written project tho).
    - andybak 9y ago
      You might consider this too much of an edge case but what if I want to pass something that quacks like an EventLoop to Process init ?
      - crdoconnor 9y ago
        Unless it's a subclass (which would pass), I don't imagine that would be at all likely. Never been much of a fan of duck typing. It only really makes sense when you have objects that approximate built in types. Take a random object with a .run() method, for example, and you don't have a fucking clue what it might be doing so there's no point interpreting that as a quack.
flavio81 9y ago
> Mostly, I see this is as a lack of compilation and types in Python. How do others overcome this and enjoy large Python projects? Divide code into modules. If you are worried about types, use object oriented programming to define the classes you want and then use the assert() instruction to check variables to comply with said classes/types. A good IDE like PyCharm will take advantage of such assert() statements.
crdoconnor 9y ago
A combination of: * Realistic integration tests that cover as many user stories as possible. * Raise exceptions ASAP for any kind of invalid data or state (could be checking types, but could also be checking for file/directory existence). Could also be pip installing and using 'schema'. * Do not build on poor quality modules and strive to decouple and uninvent poorly reinvented wheels. * Just in general: loosely couple everything. I don't see this as a static typing issue at all. Static typing only helps with a small part of the problem and it usually does that at some expense (usually more verbose & less flexible code). I'm a little skeptical that super-strong type systems like haskell necessarily help all that much either. They clearly help eliminate classes of bugs, but the overhead is very high (the amount of everyday software written in haskell is seemingly small relative to its apparent popularity).

andybak 9y ago

OK. Bear with me here. This might be overstating my case... In some sense - there are no large projects. Only projects that have failed at being modular. Maintainability involves keeping clear boundaries between functional units so that you can keep enough code in your head at any one time to reason about it. If we assume that Python is capable of handling more code than you can keep in your head at any given time, then surely the problem is simply one of making sure your modules are nicely self-contained from each other.

vturner 9y ago
Agreed, modularity definitely helps us minimize the mental load of understanding all the objects at once. But how does it handle this situation: Write function in module A that expects some complex data structure that comes in. In a static typed language, you put a type on the argument and thus guarantee, to some degree, the data in the object can be safely manipulated. Over in module B we call that function. Because Python doesn't enforce types, I can pass in some type of data structure that might behave well in many situations, but in some that function is going to fail because I didn't properly construct the data structure. How do other Python projects avoid this? Do they avoid large data structures of multiple fields?
- outworlder 9y ago
  > Write function in module A that expects some complex data structure that comes in. In a static typed language, you put a type on the argument and thus guarantee, to some degree, the data in the object can be safely manipulated. Let's question some of the assumptions. Do you NEED the entire thing to be passed around? If your functions are small and only do one thing, can't they be fed the portion of the data they need and only manipulate that? If they are getting unnecessary data, soon someone will like the convenience and that piece of data will now become necessary. I have seen Java projects with humongous "DAO" objects being passed across all apps. Even if Java had Haskell's type system, at some point it breaks down. So, you do not enforce that on Python itself. Rather, you have to apply due diligence and ensure your data structures are clean, and functions do what they are supposed to do and no more. > I can pass in some type of data structure that might behave well in many situations, but in some that function is going to fail because I didn't properly construct the data structure. Then you construct the structure in a single place, and that should have some sanity checks and sane defaults. Even if you are using a static typed language, there's no guarantee that what's inside the data structure even makes sense, only that the right type of structure is being passed on. That's helpful, but not as much as people would think. And you write tests. Lots of tests. It is amazing the amount of Python code that gets written without proper testing. And Java, for that matter. Only Python programmers have less excuses, it's easier to replace what you need for testing.
  - simion314 9y ago
    >Even if you are using a static typed language, there's no guarantee that what's inside the data structure even makes sense. You should always put validation in constructor so you are 100% sure that the object created is valid.
    - xapata 9y ago
      How can you be sure your validation covered all cases?
      - simion314 9y ago
        Say you have a Circle class, you need a center point so you validate that the center is not null and you check that radius is greater then zero. Do you have an example where is impossible to validate all cases? if so how will your unit tests validate/test all those cases?
        xapata 9y ago
        In your example, you can't anticipate that center points approximately but not quite equal to pi will cause a ... whatever bug in ... whatever downstream use of the class. One can never really know what states are acceptable.
        simion314 9y ago
        The circle is valid, what you say is a bug in other section that crashes with valid circles,so types could prevent invalid inputs to functions. What you describe is a bug in an algorithm I don't think that is possible for a compiler to prevent bugs in algorithms, division by zero or precision errors
        xapata 9y ago
        Type systems can do some pretty miraculous things. For example, you could distinguish between a connected and disconnected socket with types so that you don't accidentally send on a disconnected socket. I guess I'm arguing both sides now. My point is that one needs to root out the weird cases somehow and that it's often hard to have "100%" validation of inputs. One's initial understanding of what is the valid and invalid state space is often incorrect.
- mring33621 9y ago
  Everything's a server, er, microservice, passing JSON back and forth!
  - marcosdumay 9y ago
    Microservices are great, because you can enforce static typing within their interfaces. \s
- abc_lisper 9y ago
  Well, there's schema. https://pypi.python.org/pypi/schema/0.6.6 https://pypi.python.org/pypi/schema/0.6.6
- asddddd 9y ago
  Twisted is a massive codebase, and uses zope.interface. There are definitely options for stricter guarantees, but it's optional, and generally you'd rather be treating input as ducks. Also, even without actual type checking (mypy), IDE support + type hinting (PEP 484 or reST et al) means you can find most issues before the interpreter/compiler is involved. PyCharm is fantastic at this.
- crdoconnor 9y ago
  >How do other Python projects avoid this? https://github.com/jonathanslenders/pymux/blob/master/pymux/process.py#L54 https://github.com/jonathanslenders/pymux/blob/master/pymux/... Here's one way. Those asserts will typically pick up on errors during spikes, development and tests and turn a head scratching problem into a straightforward issue. A couple of times on a disastrously tech debt ridden project I used it to pick up obscure configuration errors in prod, too (shouldn't see it in prod on a relatively well written project tho).
  - andybak 9y ago
    You might consider this too much of an edge case but what if I want to pass something that quacks like an EventLoop to Process init ?
    - crdoconnor 9y ago
      Unless it's a subclass (which would pass), I don't imagine that would be at all likely. Never been much of a fan of duck typing. It only really makes sense when you have objects that approximate built in types. Take a random object with a .run() method, for example, and you don't have a fucking clue what it might be doing so there's no point interpreting that as a quack.

flavio81 9y ago

> Mostly, I see this is as a lack of compilation and types in Python. How do others overcome this and enjoy large Python projects? Divide code into modules. If you are worried about types, use object oriented programming to define the classes you want and then use the assert() instruction to check variables to comply with said classes/types. A good IDE like PyCharm will take advantage of such assert() statements.

crdoconnor 9y ago

A combination of: * Realistic integration tests that cover as many user stories as possible. * Raise exceptions ASAP for any kind of invalid data or state (could be checking types, but could also be checking for file/directory existence). Could also be pip installing and using 'schema'. * Do not build on poor quality modules and strive to decouple and uninvent poorly reinvented wheels. * Just in general: loosely couple everything. I don't see this as a static typing issue at all. Static typing only helps with a small part of the problem and it usually does that at some expense (usually more verbose & less flexible code). I'm a little skeptical that super-strong type systems like haskell necessarily help all that much either. They clearly help eliminate classes of bugs, but the overhead is very high (the amount of everyday software written in haskell is seemingly small relative to its apparent popularity).