4 ms·

So, you have two choices for directory layout in mono either: root/ requirements.txt / pyproject.toml src/ --- in which case you need to be caref

So, you have two choices for directory layout in mono either: root/ requirements.txt / pyproject.toml src/ --- in which case you need to be careful about when your dependencies get installed during deployment because ie pytorch can be ~500mb your other option is root/ src/ A/ requirements.txt / pyproject.toml B/ requirements.txt / pyproject.toml which means you need to come up with a bespoke dependency management solution. in either case, you're doing dependency management. I encourage you to google "monorepo {python, poetry, pip}" and you're going to land on many multiple page blog posts describe arcane dependency management solutions.

animal_spirits 2y ago
So this example is a single repo with many services? I don’t think OP was talking about that. Why would you have a single repo with different requirements? At that point they should be independent projects
- lijok 2y ago
  Because that’s the monorepo pattern. What you’re thinking of is a monolith. Two separate concepts
- selcuka 2y ago
  > So this example is a single repo with many services? Yes, that's the meaning of monorepo. If it was a single service it would be a monolith. > Why would you have a single repo with different requirements? Real world example: I implemented a client/server system using Python, and they have completely different requirements (even the Python versions are different). I still want to share code between client and server codebases, so a monorepo is the perfect choice.
gen220 2y ago
The second option is traditionally what people mean by "monorepo" (multiple executables in a repo with a dependency tree). First one you'd call a "monolith" (single executable in a repo). The second one you'd want to manage with a tool like Bazel, which will use whatever plugins are appropriate for the language (pip in Python's case). It's a legitimate ambition to try and build a tool within a single language's ecosystem to manage the second option, but it's a really hard problem that Bazel (and others) have already solved well, so you might as well use them instead.
- drawnwren 2y ago
  So you can do a somewhat hacky thing with poetry workspaces to get the first one to contain multiple binaries like i.e. https://github.com/bleu/balpy-poc https://github.com/bleu/balpy-poc But yes, you're right -- Bazel is the most correct monorepo solution. We started with a combination of the second structure, poetry, and shell scripts but have since moved to nix. I use nix btw.
  - gen220 2y ago
    Yea nix is an interesting angle of approach. It rhymes a lot with Bazel (reproducibility, explicit/declarative dependency tree, etc.), but nix approaches the problem from a different, lower layer of abstraction (by replacing the OS-level "pip" in apt, pacman, snap, brew, and the like). I think, in the long run, something like nix will win out. It makes sense for your OS "package build system" to be the same as your project build system. If your python project depends on some mildly obscure python lib, do you write your own nix "package" (or whatever nix chooses to call them) wrappers for each release you end up using? Any other tripwires in your experience so far?
    - drawnwren 2y ago
      Nix calls them "derivations", but yeah you're right again. Lots of weird language in this space. Specifically, we use this nix project [1] which provides a nice translation layer between a poetry project and a nix derivation. It allows our devs to use poetry where they want (with local relative paths in the pyproject.toml) and ci/cd to have more granular control of deps at build time. There are some corner cases. As you correctly guessed, some more obscure python libraries might require a few extra lines (i.e. to specify that this package needs setuptools etc). of code. 1 - https://github.com/nix-community/poetry2nix https://github.com/nix-community/poetry2nix
crabbone 2y ago
You should never use requirements.txt, nor pyproject.toml and this has nothing to do with whether you put multiple projects into the same repository or not. This is just an all-around bad idea. But, anyways. In a monorepo project you would have nothing to do with requirements.txt or pyproject.toml because all dependencies are already there. There's no need to install anything from anywhere...
- drawnwren 2y ago
  How can you not have a pyproject.toml/requirements.txt, nor manage dependencies yourself? Do you have no external dependencies at all?
  - crabbone 2y ago
    1. In monorepo you don't have dependencies. This is the whole point of having a monorepo: everything is included. You can build on airgapped system, no unexpected inputs into your builds, no network partitioning problems. This is the whole reason why people do that. Total control and stability. But you pay for it by having to maybe manage third-party code yourself. By having to use more space. Probably, you will need more infrastructure to side-step tools that cannot be made to not use network etc. Hope this also answers the question about external dependencies: they become internal dependencies. 2. Why you should never use requirements.txt: the tradition of using this approach comes from total misunderstanding of the goals of project deployment. This is surprising and upsetting, because this isn't a difficult concept. This is due to most developers not wanting to understand how infrastructure of their project works, and the willingness to settle on the first "solution" that "worked". The goal of deploying a project must be that byte-compiled Python code is placed in the platlib directory, accompanying data in the data directory and so on. The reliable way to accomplish this is to make a Python package and install it. Requirements.txt plays no role in this process, since package requirements need to be written into META file in the package info directory. Instead, the process that involves using requirements.txt typically ends up installing "something" that at the time of writing allowed the authors of the project to somehow make their code work, due to the combination of such factors as current working directory, some Python path files placed without their knowledge into platlib etc. This is a very fragile setup and projects designed in this way usually don't survive multiple years w/o updates that keep modifying requirements.txt to chase the latest changes in the ever changing environment. 3. pyproject.toml had more potential, in principle, but turned out to be a disaster. The idea behind this contraption was to organize configurations of multiple Python-related tools under one roof. But the chosen format was way too simplistic to realistically replace the configuration of other tools, and the configuration started to slide down two bad directions: either "string programming" (i.e. a technique where strings in the code develop special meaning and parsing), or delegation (i.e. pyproject.toml would contain a minimal code necessary to redirect to the real configuration). Second problem with pyproject.toml esp. when it comes to dependencies: there was no plan on how to solve this problem. No system. So, every tool that decided to support pyproject.toml decided to do it in a way that suits it. So, in the end, it's always better to just use the native configuration format of the tool that does the package building, instead of involving the middle-man: pyproject.toml. It always stands between the developer and their ability to debug the code, to fine tune the tool they need to run. It's the MS Windows registry all over again, but even more poorly executed. ---- So, what should you do instead? -- Well, there's a problem... there aren't any good Python tools :( And, like I mentioned before, the reason is even not the tools / their authors, it's that the underlying design of Python infrastructure is broken. So, you are bound to choose between the bad and the worse. On the bright side: the problem, when you really understand what needs to be done is very simple. For any individual project you can solve all your deployment and packaging problems very easily writing in any language that can do infrastructure-related work. I've done this multiple times (as a result of frustration with Python's own tools) and had never a reason to regret it.

animal_spirits 2y ago

So this example is a single repo with many services? I don’t think OP was talking about that. Why would you have a single repo with different requirements? At that point they should be independent projects

lijok 2y ago
Because that’s the monorepo pattern. What you’re thinking of is a monolith. Two separate concepts
selcuka 2y ago
> So this example is a single repo with many services? Yes, that's the meaning of monorepo. If it was a single service it would be a monolith. > Why would you have a single repo with different requirements? Real world example: I implemented a client/server system using Python, and they have completely different requirements (even the Python versions are different). I still want to share code between client and server codebases, so a monorepo is the perfect choice.

gen220 2y ago

The second option is traditionally what people mean by "monorepo" (multiple executables in a repo with a dependency tree). First one you'd call a "monolith" (single executable in a repo). The second one you'd want to manage with a tool like Bazel, which will use whatever plugins are appropriate for the language (pip in Python's case). It's a legitimate ambition to try and build a tool within a single language's ecosystem to manage the second option, but it's a really hard problem that Bazel (and others) have already solved well, so you might as well use them instead.

drawnwren 2y ago
So you can do a somewhat hacky thing with poetry workspaces to get the first one to contain multiple binaries like i.e. https://github.com/bleu/balpy-poc https://github.com/bleu/balpy-poc But yes, you're right -- Bazel is the most correct monorepo solution. We started with a combination of the second structure, poetry, and shell scripts but have since moved to nix. I use nix btw.
- gen220 2y ago
  Yea nix is an interesting angle of approach. It rhymes a lot with Bazel (reproducibility, explicit/declarative dependency tree, etc.), but nix approaches the problem from a different, lower layer of abstraction (by replacing the OS-level "pip" in apt, pacman, snap, brew, and the like). I think, in the long run, something like nix will win out. It makes sense for your OS "package build system" to be the same as your project build system. If your python project depends on some mildly obscure python lib, do you write your own nix "package" (or whatever nix chooses to call them) wrappers for each release you end up using? Any other tripwires in your experience so far?
  - drawnwren 2y ago
    Nix calls them "derivations", but yeah you're right again. Lots of weird language in this space. Specifically, we use this nix project [1] which provides a nice translation layer between a poetry project and a nix derivation. It allows our devs to use poetry where they want (with local relative paths in the pyproject.toml) and ci/cd to have more granular control of deps at build time. There are some corner cases. As you correctly guessed, some more obscure python libraries might require a few extra lines (i.e. to specify that this package needs setuptools etc). of code. 1 - https://github.com/nix-community/poetry2nix https://github.com/nix-community/poetry2nix

crabbone 2y ago

You should never use requirements.txt, nor pyproject.toml and this has nothing to do with whether you put multiple projects into the same repository or not. This is just an all-around bad idea. But, anyways. In a monorepo project you would have nothing to do with requirements.txt or pyproject.toml because all dependencies are already there. There's no need to install anything from anywhere...

drawnwren 2y ago
How can you not have a pyproject.toml/requirements.txt, nor manage dependencies yourself? Do you have no external dependencies at all?
- crabbone 2y ago
  1. In monorepo you don't have dependencies. This is the whole point of having a monorepo: everything is included. You can build on airgapped system, no unexpected inputs into your builds, no network partitioning problems. This is the whole reason why people do that. Total control and stability. But you pay for it by having to maybe manage third-party code yourself. By having to use more space. Probably, you will need more infrastructure to side-step tools that cannot be made to not use network etc. Hope this also answers the question about external dependencies: they become internal dependencies. 2. Why you should never use requirements.txt: the tradition of using this approach comes from total misunderstanding of the goals of project deployment. This is surprising and upsetting, because this isn't a difficult concept. This is due to most developers not wanting to understand how infrastructure of their project works, and the willingness to settle on the first "solution" that "worked". The goal of deploying a project must be that byte-compiled Python code is placed in the platlib directory, accompanying data in the data directory and so on. The reliable way to accomplish this is to make a Python package and install it. Requirements.txt plays no role in this process, since package requirements need to be written into META file in the package info directory. Instead, the process that involves using requirements.txt typically ends up installing "something" that at the time of writing allowed the authors of the project to somehow make their code work, due to the combination of such factors as current working directory, some Python path files placed without their knowledge into platlib etc. This is a very fragile setup and projects designed in this way usually don't survive multiple years w/o updates that keep modifying requirements.txt to chase the latest changes in the ever changing environment. 3. pyproject.toml had more potential, in principle, but turned out to be a disaster. The idea behind this contraption was to organize configurations of multiple Python-related tools under one roof. But the chosen format was way too simplistic to realistically replace the configuration of other tools, and the configuration started to slide down two bad directions: either "string programming" (i.e. a technique where strings in the code develop special meaning and parsing), or delegation (i.e. pyproject.toml would contain a minimal code necessary to redirect to the real configuration). Second problem with pyproject.toml esp. when it comes to dependencies: there was no plan on how to solve this problem. No system. So, every tool that decided to support pyproject.toml decided to do it in a way that suits it. So, in the end, it's always better to just use the native configuration format of the tool that does the package building, instead of involving the middle-man: pyproject.toml. It always stands between the developer and their ability to debug the code, to fine tune the tool they need to run. It's the MS Windows registry all over again, but even more poorly executed. ---- So, what should you do instead? -- Well, there's a problem... there aren't any good Python tools :( And, like I mentioned before, the reason is even not the tools / their authors, it's that the underlying design of Python infrastructure is broken. So, you are bound to choose between the bad and the worse. On the bright side: the problem, when you really understand what needs to be done is very simple. For any individual project you can solve all your deployment and packaging problems very easily writing in any language that can do infrastructure-related work. I've done this multiple times (as a result of frustration with Python's own tools) and had never a reason to regret it.