6 ms·

I'm surprised you're hitting that boundary at just 5-10K lines. I work on a team of 6* with about 60,000 logical Python statements (Not including tests) and are

I'm surprised you're hitting that boundary at just 5-10K lines. I work on a team of 6* with about 60,000 logical Python statements (Not including tests) and aren't running into any language or framework induced roadblocks. And this isn't even a case of "If you do everything perfect like me, [Language] works great!" - We only have 45% code coverage, and our code architecture, in certain places, is really sub-optimal (Caused by us, not the language). When I'm about to make a major change to a function (different inputs or different outputs) I'll always start by grepping first to get an idea of what I'm about to get myself into, write or adjust tests for the new 'signature', change the function, and code/debug/test until it works. My team is in the loop before my code is committed to a shared branch. * We have other responsibilities besides the Python code. Edit: If your team is stepping all over each other at such a low LOC, my hunch is that its related to a communication problem with the team, your code is too tightly coupled, or there's not enough architecture planning (Too much is bad, but so is too little).

runT1ME 13y ago
It's also possible you have a 60k loc python app because you aren't able to get near the amount of code reuse or abstractions due to the lack of static typing, and this is why you haven't run into issues with refactoring.
- Buttons840 13y ago
  If Wikipedia is to be believed: "Dynamic typing typically allows duck typing, which enables easier code reuse." I would expect to see more code reuse in Python than in any statically typed language.
  - coolsunglasses 13y ago
    It's a true statement if you qualify it properly. It wins you more code reusability with less code written to accomplish aforementioned reusability. I always run into silly inheritance chains, type coercion or adapter fns/methods to handle reusing code for unlike-types, rather than relying on a common base of functionality regardless of where it was derived from (function API, inheritance, mixin, etc). Best way I've seen this done is in Clojure. Good mix of the best of both worlds in terms of static and dynamic typing, multimethods, protocols, essentially structurally typed arguments, etc.
- spamizbad 13y ago
  If that was the case our codebase would be a serious pain to work on and I'd be much worse off than the OP. Also I should clarify, 60KLS is not all 1 app: it's spread across 8 discrete apps, an API, and data layer (models).
bsaul 13y ago
I guess it really depends on what you're building. Just to give you a taste here's the kind of thing i'm doing: - tree-structured sqlalchemy managed objects comparaison , generating diffs, then applying diffs to those trees, and persist everything. I'm using sqlalchemy declarative approach. That diff applying is performed in a celery background task, reusing my flask configuration. So, in the worst case, i have to deal at the same time with : - Business logic on a bunch of SqlAlchemy ORM object ( declarative approach) - a Flask request context - a celery task context - and sqlalchemy session At that point, i'm changing the signature of a function that takes pieces of those three parts to perform some business logic. Now i'm telling that the IDE (pyCharm, the best one) and python "compile phase" doesn't give you a CLUE on what you're doing. You're dealing with so much "magic" that it becomes unmanageable. You don't need that many lines of code to reach that point. EDIT : you've got 6 people working on 60 LOC spread on 8 discrete apps. That's about the same amount of isolated group of LOC per person than me (a person having to deal with a group of 5-10K LOC)
- aidos 13y ago
  I'm not sure the complexity breaks down that way. More people + more code means things need to be better organised because things might change under you without warning (or you may have to work with an unfamiliar part of the system). How is your app structured? I'm guessing that Flask is the top level glue and everything else is scattered around the Flask app. That's the general approach (in most modern MVC frameworks) and I think it's also the root cause of complexity. Celery and Flask (and sql alchemy too) should really be asides to the main codebase. The code should be layered and discrete libraries for handling different parts of the system. If you have 6k loc that all cross reference one another then you have problems in any language. Presumably there are a number of different components in there. Each should stand on its own with as simpler api as possible. As ever, too much coupling is going to make it impossible to reason about your code. If you're about to change a signature for a function, it should already be fairly obvious as to where it is called from. If not, you need to ask yourself why. What is this function that's so fundamental to the system that it could be called by any module? Why is it buried in another module an being accessed from elsewhere? My current app has about 4k loc in python and the same again in js (angular). It's broken into dozens of parts that I only connect where needed through a simple api. At the core is a sort of image processing library (that itself contains lots of different components). On top of that is a system that works with the image processing. Above that another system that interacts with the data models, uses the system below and farms out processing to picloud (though could use celery). Finally, the Flask layer just provides a web interface to talk to the system that handles that business processing. I can tap into any of those layers to drive them. The point is that I can operate at a high level without needing to consider any details of deeper parts of the system. These are the layers of abstraction that make a system understandable and stop it from being brittle.

runT1ME 13y ago

It's also possible you have a 60k loc python app because you aren't able to get near the amount of code reuse or abstractions due to the lack of static typing, and this is why you haven't run into issues with refactoring.

Buttons840 13y ago
If Wikipedia is to be believed: "Dynamic typing typically allows duck typing, which enables easier code reuse." I would expect to see more code reuse in Python than in any statically typed language.
- coolsunglasses 13y ago
  It's a true statement if you qualify it properly. It wins you more code reusability with less code written to accomplish aforementioned reusability. I always run into silly inheritance chains, type coercion or adapter fns/methods to handle reusing code for unlike-types, rather than relying on a common base of functionality regardless of where it was derived from (function API, inheritance, mixin, etc). Best way I've seen this done is in Clojure. Good mix of the best of both worlds in terms of static and dynamic typing, multimethods, protocols, essentially structurally typed arguments, etc.
spamizbad 13y ago
If that was the case our codebase would be a serious pain to work on and I'd be much worse off than the OP. Also I should clarify, 60KLS is not all 1 app: it's spread across 8 discrete apps, an API, and data layer (models).

bsaul 13y ago

I guess it really depends on what you're building. Just to give you a taste here's the kind of thing i'm doing: - tree-structured sqlalchemy managed objects comparaison , generating diffs, then applying diffs to those trees, and persist everything. I'm using sqlalchemy declarative approach. That diff applying is performed in a celery background task, reusing my flask configuration. So, in the worst case, i have to deal at the same time with : - Business logic on a bunch of SqlAlchemy ORM object ( declarative approach) - a Flask request context - a celery task context - and sqlalchemy session At that point, i'm changing the signature of a function that takes pieces of those three parts to perform some business logic. Now i'm telling that the IDE (pyCharm, the best one) and python "compile phase" doesn't give you a CLUE on what you're doing. You're dealing with so much "magic" that it becomes unmanageable. You don't need that many lines of code to reach that point. EDIT : you've got 6 people working on 60 LOC spread on 8 discrete apps. That's about the same amount of isolated group of LOC per person than me (a person having to deal with a group of 5-10K LOC)

aidos 13y ago
I'm not sure the complexity breaks down that way. More people + more code means things need to be better organised because things might change under you without warning (or you may have to work with an unfamiliar part of the system). How is your app structured? I'm guessing that Flask is the top level glue and everything else is scattered around the Flask app. That's the general approach (in most modern MVC frameworks) and I think it's also the root cause of complexity. Celery and Flask (and sql alchemy too) should really be asides to the main codebase. The code should be layered and discrete libraries for handling different parts of the system. If you have 6k loc that all cross reference one another then you have problems in any language. Presumably there are a number of different components in there. Each should stand on its own with as simpler api as possible. As ever, too much coupling is going to make it impossible to reason about your code. If you're about to change a signature for a function, it should already be fairly obvious as to where it is called from. If not, you need to ask yourself why. What is this function that's so fundamental to the system that it could be called by any module? Why is it buried in another module an being accessed from elsewhere? My current app has about 4k loc in python and the same again in js (angular). It's broken into dozens of parts that I only connect where needed through a simple api. At the core is a sort of image processing library (that itself contains lots of different components). On top of that is a system that works with the image processing. Above that another system that interacts with the data models, uses the system below and farms out processing to picloud (though could use celery). Finally, the Flask layer just provides a web interface to talk to the system that handles that business processing. I can tap into any of those layers to drive them. The point is that I can operate at a high level without needing to consider any details of deeper parts of the system. These are the layers of abstraction that make a system understandable and stop it from being brittle.