3 ms·
I'm not sure the complexity breaks down that way. More people + more code means things need to be better organised because things might change under you without
by aidos 13y ago
I'm not sure the complexity breaks down that way. More people + more code means things need to be better organised because things might change under you without warning (or you may have to work with an unfamiliar part of the system).
How is your app structured? I'm guessing that Flask is the top level glue and everything else is scattered around the Flask app. That's the general approach (in most modern MVC frameworks) and I think it's also the root cause of complexity.
Celery and Flask (and sql alchemy too) should really be asides to the main codebase. The code should be layered and discrete libraries for handling different parts of the system. If you have 6k loc that all cross reference one another then you have problems in any language. Presumably there are a number of different components in there. Each should stand on its own with as simpler api as possible. As ever, too much coupling is going to make it impossible to reason about your code.
If you're about to change a signature for a function, it should already be fairly obvious as to where it is called from. If not, you need to ask yourself why. What is this function that's so fundamental to the system that it could be called by any module? Why is it buried in another module an being accessed from elsewhere?
My current app has about 4k loc in python and the same again in js (angular). It's broken into dozens of parts that I only connect where needed through a simple api.
At the core is a sort of image processing library (that itself contains lots of different components). On top of that is a system that works with the image processing. Above that another system that interacts with the data models, uses the system below and farms out processing to picloud (though could use celery). Finally, the Flask layer just provides a web interface to talk to the system that handles that business processing. I can tap into any of those layers to drive them. The point is that I can operate at a high level without needing to consider any details of deeper parts of the system.
These are the layers of abstraction that make a system understandable and stop it from being brittle.
- bsaul 13y agoThe problem lies in the interfaces. Components, even when they are independents, expose interfaces. Those interfaces acts as a contract between the component and its users. Python needs a way to make those interfaces automatically verifiable. It's even worse once you start to use big libraries. If you're using top level functions, then maybe your IDE can help you, but as soon as you're dealing with magical properties or parameters, that becomes a mess. Take for example the "desc" magical function in SQLAlchemy, on things like "order_by" on relationships. That's extremely useful and clever, but i'd really like python to give me some "ok, you're not doing things wrong" message as I'm typing. Even better, once i enter a relationship declaration, it should give me a list of all the parameter i can use, along with the things thoses properties accept for value. This way i wouldn't have to check the documentation every time i'm writing one. It could also let me discover new things as i'm typing ("hey what's that property doing ? that looks interesting..."). Autocompletion is another way of discovering APIs. But for that, you need type declarations. EDIT : as for my api, it's really nothing fancy. It's structured in three big parts "admin / common / public". They each have their "model / business / service" layers, and each have their modules. Only the service layer is impacted by flask. I have some "utils" modules for very low-level stuffs (json serialization, etc). Flask configuration are used a bit everywhere, because i want my api to have only one configuration file. Nothing special, really.
- boothead 13y agoIt's not a total solution but I find the zope.interface and zope.component libraries really good for this. I believe that zope.component provides some stuff to build unit tests that verify if and interface is correctly implemented/provided too.
- aidos 13y agoI definitely understand where you're coming from. On the discovery aspect, maybe that's just not something I notice because I work in ipython so it's interactive anyway.