4 ms·

> -m` means to run the code as a module, i.e., to figure out what package it's in (by importing its ancestors first) and to set its `name` to the actual nam

by MrJohz 2y ago

> -m` means to run the code as a module, i.e., to figure out what package it's in (by importing its ancestors first) and to set its `__name__` to the actual name of the module (so that code inside `if __name__ == '__main__':` will not run). This is only partially true. The code will be executed as a module in some senses, but importantly `__name__` will be set to `__main__` for this module. This is what enables e.g. `python -m http.server` to have a different effect to `import http.server`. I also disagree with some other parts of this as well. For example, if you were okay with breaking changes, it's quite easy to differentiate between local imports (any import path that begins with `.`) and imports from the package store (all other imports). This is how imports are generally handled in Javascript, for example. Looking for the package root also doesn't have to be that complicated if you have a well-known project structure or a well-known metadata file that can specify what the project should look like. Again, look at Javascript and the `package.json` file, where all of the module entry points can be defined explicitly. I understand why Python doesn't work like this, and why it probably never can at this point, but that doesn't make it a good system, and it doesn't mean that the quirks are there for a good reason. IIRC, there's some quite clever logic built into the import system just so that you can import a third-party Flask plugin from `flask.plugins.*`, which is cool but also completely unnecessary complexity that makes the whole system more difficult to comprehend for almost no benefit. If there were ever to be a Python 4, I wish it would focus solely on the necessary breaking changes to get Python packaging back into a manageable state: explicit and mandatory metadata, one fully-featured blessed package manager (and not the anemic, it-just-installs-things approach of pip, but a genuine package manager with lock files, workspaces, proper dependency groups, etc), and clear, obvious imports that can be explained to an everyday user of Python without them having to resort to badly-curated SO answers. But unfortunately I think that ship has sailed.

zahlman 2y ago
Thanks for the informal invitation to write even more on one of my favourite topics. ;) >but importantly `__name__` will be set to `__main__` for this module Correct. Should have tested that, sorry - I don't use the feature often. >It's quite easy to differentiate between local imports (any import path that begins with `.`) and imports from the package store (all other imports). This is how imports are generally handled in Javascript, for example. JavaScript generally runs in a fundamentally different environment, and it has the luxury of letting the environment define these semantics (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#module_specifier_resolution https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...). But the problem I was trying to describe was more of a design problem. As far as I know, JavaScript doesn't have to worry about the hierarchical package structure thing. When you use an import for a name like `foo.bar.baz`, it's easy to see the package structure. If you import a path like, I don't know, `file://./foo/bar/baz.py`, you have to decide whether `foo` and `bar` are packages or just containing folders (the problem that the original article is about). You also have to decide whether that's relative to the current file or the project root (hopefully not the current working directory!) You have to decide whether you're forbidding absolute paths for "security" or allowing them for "convenience", and you realistically need to support support the symbolic-name syntax as well anyway. And you either restrict the user to know the actual file type source, or have to come up with some pseudo-path thing that allows for discovering other sources. >I understand why Python doesn't work like this, and why it probably never can at this point, but that doesn't make it a good system There are definitely things I don't like about this system, but it demonstrably solves the important problems, and can be worked with more straightforwardly (and understood more readily) than a lot of users seem to think. There's a lot of history to this - see e.g. https://stackoverflow.com/a/75068706/523612 https://stackoverflow.com/a/75068706/523612 - which indeed has a lot to do with "why it probably never can", but I think a bigger issue is that Python doesn't actually expect users to create a "project", and thus there isn't necessarily a good place to put a marker file. Also, "project root" and "package root" are different concepts, especially if you use src layout (as I do nowadays). >If there were ever to be a Python 4, I wish it would focus solely on the necessary breaking changes to get Python packaging back into a manageable state I used to think along these lines, too. But nowadays, the idea of a greenfield, Python-inspired language project makes a lot more sense to me. I expect to be blogging a fair bit about my own design in 2025. That said, it seems pretty clear that the Python community sees the language and the packaging ecosystem as more or less separate concerns that can be worked on separately. (Notwithstanding the special status of Pip - whereby it's still distributed with Python, bootstrapped by other standard library modules and commonly expected to be copied into every environment, even though it's separately developed and versioned, doesn't have a formal programmatic API and is perfectly capable of installing into environments other than its host.) The current attitude is still that there will never be a Python 4 - "Python 3 is the brand", PEP 2026 proposes dropping the pretense and making it use real calver, etc. But I don't think that means they'll never drop support for legacy builds. (Of course, they'll never drop support for `setup.py` - because that's Setuptools' responsibility now; PEP 517 allows the build backend to make its own design decisions, and building non-Python code and integrating it with Python is not at all straightforward. But they can eventually require `pyproject.toml`.) >explicit and mandatory metadata Yes, this is embarrassing. They've supposedly been working on it, and I've been following the conversation in some detail. But the community's over-accommodating attitude towards backwards compatibility is a serious handicap here. I have a special "celebration" planned for the official Pip 24.3 release, which should be soon (apparently .dev0 is available). >one fully-featured blessed package manager (and not the anemic, it-just-installs-things approach of pip, but a genuine package manager... A lot of people say this and I haven't understood it. To me, it doesn't make a difference whether the various tasks a "package manager" does are separate applications or separate subcommands of the same application. The "just-installs-packages" thing is actually pretty complex already (it still includes dependency resolution, for example, and at least in Python it includes orchestrating local builds). And for the rest of the packaging loop, there is poor agreement on what the expected set of tasks is. The competing third-party options are competing for a reason. I have a lot more thoughts about this topic - not yet very organized ones. >clear, obvious imports that can be explained to an everyday user of Python without them having to resort to badly-curated SO answers. Honestly, I blame SO for this (saying this as an SO curator with a special interest in the topic). The old Q&A is full of randomly-scoped Qs that covered the individual needs of some random individuals, rather than coherently dividing up the problem space. The answers are very often just bad despite huge upvote totals - partly because things have changed over time, partly because some myths are extremely persistent, and partly because everyone wants to share their personal experiences. At the time a lot of this content was being written, SO hadn't really fully established its "sense of self" yet - there doesn't appear to have been a core community on the meta site that really understood what the format is capable of, what Stack Exchange sites ought to be, or how to achieve those goals. (Try digging into the history of how the standard reasons for closing questions have changed over time.) It took until 2014 or so, when new question volume was at its peak, for people to get frustrated enough to actually start defining those things properly. Anyway, people still ask lots of "new" questions about Python that should have a straightforward answer and be recognized as common duplicates - but it's hell finding proper duplicate targets for them. There is maddeningly little community will to do anything about it, even though the problem is well recognized (https://meta.stackoverflow.com/questions/423857 https://meta.stackoverflow.com/questions/423857). It's a big part of why I'm trying to start over on Codidact.

> -m` means to run the code as a module, i.e., to figure out what package it's in (by importing its ancestors first) and to set its `__name__` to the actual nam

by MrJohz 2y ago

zahlman 2y ago

Thanks for the informal invitation to write even more on one of my favourite topics. ;) >but importantly `__name__` will be set to `__main__` for this module Correct. Should have tested that, sorry - I don't use the feature often. >It's quite easy to differentiate between local imports (any import path that begins with `.`) and imports from the package store (all other imports). This is how imports are generally handled in Javascript, for example. JavaScript generally runs in a fundamentally different environment, and it has the luxury of letting the environment define these semantics (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#module_specifier_resolution https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...). But the problem I was trying to describe was more of a design problem. As far as I know, JavaScript doesn't have to worry about the hierarchical package structure thing. When you use an import for a name like `foo.bar.baz`, it's easy to see the package structure. If you import a path like, I don't know, `file://./foo/bar/baz.py`, you have to decide whether `foo` and `bar` are packages or just containing folders (the problem that the original article is about). You also have to decide whether that's relative to the current file or the project root (hopefully not the current working directory!) You have to decide whether you're forbidding absolute paths for "security" or allowing them for "convenience", and you realistically need to support support the symbolic-name syntax as well anyway. And you either restrict the user to know the actual file type source, or have to come up with some pseudo-path thing that allows for discovering other sources. >I understand why Python doesn't work like this, and why it probably never can at this point, but that doesn't make it a good system There are definitely things I don't like about this system, but it demonstrably solves the important problems, and can be worked with more straightforwardly (and understood more readily) than a lot of users seem to think. There's a lot of history to this - see e.g. https://stackoverflow.com/a/75068706/523612 https://stackoverflow.com/a/75068706/523612 - which indeed has a lot to do with "why it probably never can", but I think a bigger issue is that Python doesn't actually expect users to create a "project", and thus there isn't necessarily a good place to put a marker file. Also, "project root" and "package root" are different concepts, especially if you use src layout (as I do nowadays). >If there were ever to be a Python 4, I wish it would focus solely on the necessary breaking changes to get Python packaging back into a manageable state I used to think along these lines, too. But nowadays, the idea of a greenfield, Python-inspired language project makes a lot more sense to me. I expect to be blogging a fair bit about my own design in 2025. That said, it seems pretty clear that the Python community sees the language and the packaging ecosystem as more or less separate concerns that can be worked on separately. (Notwithstanding the special status of Pip - whereby it's still distributed with Python, bootstrapped by other standard library modules and commonly expected to be copied into every environment, even though it's separately developed and versioned, doesn't have a formal programmatic API and is perfectly capable of installing into environments other than its host.) The current attitude is still that there will never be a Python 4 - "Python 3 is the brand", PEP 2026 proposes dropping the pretense and making it use real calver, etc. But I don't think that means they'll never drop support for legacy builds. (Of course, they'll never drop support for `setup.py` - because that's Setuptools' responsibility now; PEP 517 allows the build backend to make its own design decisions, and building non-Python code and integrating it with Python is not at all straightforward. But they can eventually require `pyproject.toml`.) >explicit and mandatory metadata Yes, this is embarrassing. They've supposedly been working on it, and I've been following the conversation in some detail. But the community's over-accommodating attitude towards backwards compatibility is a serious handicap here. I have a special "celebration" planned for the official Pip 24.3 release, which should be soon (apparently .dev0 is available). >one fully-featured blessed package manager (and not the anemic, it-just-installs-things approach of pip, but a genuine package manager... A lot of people say this and I haven't understood it. To me, it doesn't make a difference whether the various tasks a "package manager" does are separate applications or separate subcommands of the same application. The "just-installs-packages" thing is actually pretty complex already (it still includes dependency resolution, for example, and at least in Python it includes orchestrating local builds). And for the rest of the packaging loop, there is poor agreement on what the expected set of tasks is. The competing third-party options are competing for a reason. I have a lot more thoughts about this topic - not yet very organized ones. >clear, obvious imports that can be explained to an everyday user of Python without them having to resort to badly-curated SO answers. Honestly, I blame SO for this (saying this as an SO curator with a special interest in the topic). The old Q&A is full of randomly-scoped Qs that covered the individual needs of some random individuals, rather than coherently dividing up the problem space. The answers are very often just bad despite huge upvote totals - partly because things have changed over time, partly because some myths are extremely persistent, and partly because everyone wants to share their personal experiences. At the time a lot of this content was being written, SO hadn't really fully established its "sense of self" yet - there doesn't appear to have been a core community on the meta site that really understood what the format is capable of, what Stack Exchange sites ought to be, or how to achieve those goals. (Try digging into the history of how the standard reasons for closing questions have changed over time.) It took until 2014 or so, when new question volume was at its peak, for people to get frustrated enough to actually start defining those things properly. Anyway, people still ask lots of "new" questions about Python that should have a straightforward answer and be recognized as common duplicates - but it's hell finding proper duplicate targets for them. There is maddeningly little community will to do anything about it, even though the problem is well recognized (https://meta.stackoverflow.com/questions/423857 https://meta.stackoverflow.com/questions/423857). It's a big part of why I'm trying to start over on Codidact.