3 ms·

Most of it is pretty obvious. However, what I am still fighting myself on with Python is a project layout/module structure. In languages with C++-like OOP mode

by krick 11y ago

Most of it is pretty obvious. However, what I am still fighting myself on with Python is a project layout/module structure. In languages with C++-like OOP model (Java, PHP) it's most of the time reasonable to settle on "1 file, 1 class" and to use directory structure the same way you use namespaces. Actually, in some such languages it's enforced. For PHP it still leaves a problem with where you shoud put/expect to find anything beside classes, but it's fine, more or less. I hope some day we'll decide even more generally what works best, but for now it's decided per project/framework and it's mostly bearable. For Ruby it's mostly the same as for PHP, even though the language is a bit more flexible. But it's still normal to use class as the main entity, so pretty much the same guidelines/problems apply. For Python… I cannot decide yet, and what's worse, looking at the code on, say, Github, it doesn't seem like it's less of a problem for many others. Because some of "solutions" are seriously horrible. No project structure whatsoever, tens of classes in the same file. Classes, functions and even app-specific code (like GUI, CLI or HTTP-routing) all in the same file. Multi-thousand LOC files are considered completely normal. Ridiculous amount of code in __init__.py, that completely re-arranges submodule structure or even defines a bunch of new code. Hell, I've seen projects (more than 1) almost entirely written in the __init__.py! Please don't say you think it's normal. I would say that things remind of "project-structure" in C, but (it's scary to suddenly realize that) it would actually be a compliment for Python, because even though large files are considered normal as well, with modern practices of writing C-code and all that "C-lang OOP" model stuff it's mostly clear where you should put something. With Python I'm never sure. The only way to find something is to use grep. With things like that in Python, it's even no use to speak of a "project structure" in the more general sense, aside from what's enforced by the frameworks like Django. So, any suggestions?

encoderer 11y ago
In PHP you create classes for everything because there is no other encapsulating scope. In that circumstance, with the class loader hook the language gives you, one class per file is a clear win. In Python, with module-level scope, it makes sense to group like-functionality in the same module (aka file). Maybe it only seems "wrong" to you because it's different than other languages you're more familiar with?
- krick 11y ago
  Yes, it makes sense "to group like-functionality in the same module", but that seems to be the very problem. Because I already describes what it leads to, and compared with state of things in languages with more rigid code structure I cannot say it satisfies me. And, just to clarify, I consider myself quite familiar with Python, because it's probably my first language I'm still using a lot, even though the language itself changed a lot during this time. That's about 10 years. And I am still never sure where I'm supposed to put anything. When 1 file contains everything there is to say about `time` — it will contain a lot, because there really is a lot to say about time. It's much easier to move around the codebase, when there are many files with clear semantic scope. Say, there's a module Time with a file (and a class) Timezone in it. Then, even though the class Timezone may be 4000 lines long — it doesn't bother me. If I assume that there's no crazy person working on that project, that would write something about product energy value in the class named "Timezone" — there's pretty much zero cognitive load of keeping that file in my text editor. Because I know, that even though 4000 LOC is quite a lot — there's nothing else in that file. There's no "before" and "after" that class. If I am not finding something here — it isn't here. And the best thing is, that the directory structure can be quite nested, but feel pretty flat. The directory tree is completely semantic, there's no 1 directory with 200 files in that, there's clear meaning in that: src/Time/Timezone, src/Time/Datetime, src/Food/Components/Protein, src/Food/Components/Fat. I can execute `tree -f` in the console, and I already know a lot about what I can expect to find in the codebase. Discoverability is important. It is also easier to review commits in CVS with a project structure like that. I can look only at files changed and I know, that somebody changed something in Timezone class, not just something "time-related". With python it's a great deal harder to follow project structure like that. If everything can be grouped by file based just on "related-ness" principle — files can grow huge and I have to apply constant, significant effort to notice that something can become a sub-module, not just a file with bunch of functionality. And looking at the file with the same 4000 LOC becomes way harder: I don't know what else there is in that class, judging only by filename, assumption of developers being more or less sane and even the piece of code I'm looking at right now. There can be many things "related to time" before or after that piece of code. 4000 suddenly seems much larger than before, and the cognitive load now is quite significant. That's why I'm looking for some "project-structure style guide" for Python. There needs to be some set of simple rules to follow, so that python project would not become the mess it always becomes. If we think it's reasonable to have a style-guide about "backslash vs parens in multi-line expressions" it's unreasonable to assume that project structure will grow just fine all by itself. Well, it contradicts my observations.
majika 11y ago
I would suggest working on how to improve your Python tooling; Vim with Jedi, or PyCharm are both great options. I think Python's module semantics are harder to use and understand, but when you know how to wield them (or reading the code of someone who does) they serve you far better than other systems. Python's module semantics are closer to Java's (explicit packages) rather than Ruby or JavaScript (global namespace), but unlike Java, Python's modules are derived from the file structure and not `package` statements. The consequence of this is that most Python developers throw everything that should be in the same module into the same file, because that's "easiest". It stops being easy when the file grows large. What you can do to address the problem of large modules in a single file is to turn the module into a directory of the same name, create an `__init__.py`, then put separate files in that directory to hold sections of the module (e.g. a file for each class if that's appropriate), and then in the `__init__.py` import the classes and whatever else you want to export from the module. Done - and client code needn't change. Python's module system (and namespace system in general) is one of the largest reasons I prefer it over other dynamic languages. Knowing where every identifier came from in a file makes it so much easier to learn a codebase.

by krick 11y ago

encoderer 11y ago

In PHP you create classes for everything because there is no other encapsulating scope. In that circumstance, with the class loader hook the language gives you, one class per file is a clear win. In Python, with module-level scope, it makes sense to group like-functionality in the same module (aka file). Maybe it only seems "wrong" to you because it's different than other languages you're more familiar with?

krick 11y ago
Yes, it makes sense "to group like-functionality in the same module", but that seems to be the very problem. Because I already describes what it leads to, and compared with state of things in languages with more rigid code structure I cannot say it satisfies me. And, just to clarify, I consider myself quite familiar with Python, because it's probably my first language I'm still using a lot, even though the language itself changed a lot during this time. That's about 10 years. And I am still never sure where I'm supposed to put anything. When 1 file contains everything there is to say about `time` — it will contain a lot, because there really is a lot to say about time. It's much easier to move around the codebase, when there are many files with clear semantic scope. Say, there's a module Time with a file (and a class) Timezone in it. Then, even though the class Timezone may be 4000 lines long — it doesn't bother me. If I assume that there's no crazy person working on that project, that would write something about product energy value in the class named "Timezone" — there's pretty much zero cognitive load of keeping that file in my text editor. Because I know, that even though 4000 LOC is quite a lot — there's nothing else in that file. There's no "before" and "after" that class. If I am not finding something here — it isn't here. And the best thing is, that the directory structure can be quite nested, but feel pretty flat. The directory tree is completely semantic, there's no 1 directory with 200 files in that, there's clear meaning in that: src/Time/Timezone, src/Time/Datetime, src/Food/Components/Protein, src/Food/Components/Fat. I can execute `tree -f` in the console, and I already know a lot about what I can expect to find in the codebase. Discoverability is important. It is also easier to review commits in CVS with a project structure like that. I can look only at files changed and I know, that somebody changed something in Timezone class, not just something "time-related". With python it's a great deal harder to follow project structure like that. If everything can be grouped by file based just on "related-ness" principle — files can grow huge and I have to apply constant, significant effort to notice that something can become a sub-module, not just a file with bunch of functionality. And looking at the file with the same 4000 LOC becomes way harder: I don't know what else there is in that class, judging only by filename, assumption of developers being more or less sane and even the piece of code I'm looking at right now. There can be many things "related to time" before or after that piece of code. 4000 suddenly seems much larger than before, and the cognitive load now is quite significant. That's why I'm looking for some "project-structure style guide" for Python. There needs to be some set of simple rules to follow, so that python project would not become the mess it always becomes. If we think it's reasonable to have a style-guide about "backslash vs parens in multi-line expressions" it's unreasonable to assume that project structure will grow just fine all by itself. Well, it contradicts my observations.

majika 11y ago

I would suggest working on how to improve your Python tooling; Vim with Jedi, or PyCharm are both great options. I think Python's module semantics are harder to use and understand, but when you know how to wield them (or reading the code of someone who does) they serve you far better than other systems. Python's module semantics are closer to Java's (explicit packages) rather than Ruby or JavaScript (global namespace), but unlike Java, Python's modules are derived from the file structure and not `package` statements. The consequence of this is that most Python developers throw everything that should be in the same module into the same file, because that's "easiest". It stops being easy when the file grows large. What you can do to address the problem of large modules in a single file is to turn the module into a directory of the same name, create an `__init__.py`, then put separate files in that directory to hold sections of the module (e.g. a file for each class if that's appropriate), and then in the `__init__.py` import the classes and whatever else you want to export from the module. Done - and client code needn't change. Python's module system (and namespace system in general) is one of the largest reasons I prefer it over other dynamic languages. Knowing where every identifier came from in a file makes it so much easier to learn a codebase.