10 ms·

I would really encourage you to try your hand at at a monorepo. I manage a python monorepo in prod and dependency management is hell. Poetry has some newer feat

I would really encourage you to try your hand at at a monorepo. I manage a python monorepo in prod and dependency management is hell. Poetry has some newer features that I am looking at trying to implement, but the state of the ecosystem wrt big monorepos is horrible.

crabbone 2y ago
[flagged]
- miohtama 2y ago
  Generally all Python software use third party packages which are fetched from PyPi package site. These packages are not stored in monorepo.
  - TheCleric 2y ago
    Not necessarily. Some people do vendor their packages in the repo for consistency (though admittedly it’s rarer).
  - crabbone 2y ago
    This is maybe common, but this contradicts the definition of monorepo. You just use the word incorrectly. "Mono" means "one". If you pull packages from elsewhere, that stops being "mono". There's really no difference in this situation between your team publishing multiple packages from multiple repositories and then assembling them together for the purpose of deployment, or doing so, but with the third-party packages.
- dudus 2y ago
  You still have dependencies. But they are included in the repo. What happens when you have 2 different apps in your monorepo but one uses an older version of Django and wasn't upgraded yet. The monorepo doesn't handle that automatically, you need tooling. It's not as black and white as you say.
  - crabbone 2y ago
    In a company I worked for that used monorepo we had multiple versions of Linux (CentOS 6 variants) all at the same time. Trust me, something like different versions of Django is not a really big problem in comparison. But, to answer your question in a more practical way: what are you going to do with these two versions of Django? Are you planning on running a single pre-fork server with two different Django application servers? Are there going to be two different pre-fork servers? Do both Django versions have to be loaded by the same Python interpreter? Or maybe they don't even need to be deployed on the same compute node? Once you can answer questions like these, the solution becomes obvious. Most likely, what you want is something like two pre-fork servers proxying HTTP traffic into two separate instances of Django-based Python Web applications. So, in your deployment script, you create eg. two virtual environments and place two different Django packages into these two environments, together with associated code for the Web application. There are, of course, ways to improve on that. Deploying while using Python packaging is, in general, wasteful. A better way is to merge all the packages you need to deploy your application into a single filesystem snapshot (removing all the info directories, and, potentially, all the source files, replacing them with bytecompiled ones, while also pruning all other irrelevant data from Python packages, s.a. readmes, test files etc.) Going even further, you can Cythonize your code, or even embed Python interpreter into your http server so that you deploy your application as a single binary. And there are plenty more of other options. The sky is the limit really.
- drawnwren 2y ago
  So, you have two choices for directory layout in mono either: root/ requirements.txt / pyproject.toml src/ --- in which case you need to be careful about when your dependencies get installed during deployment because ie pytorch can be ~500mb your other option is root/ src/ A/ requirements.txt / pyproject.toml B/ requirements.txt / pyproject.toml which means you need to come up with a bespoke dependency management solution. in either case, you're doing dependency management. I encourage you to google "monorepo {python, poetry, pip}" and you're going to land on many multiple page blog posts describe arcane dependency management solutions.
  - animal_spirits 2y ago
    So this example is a single repo with many services? I don’t think OP was talking about that. Why would you have a single repo with different requirements? At that point they should be independent projects
    - lijok 2y ago
      Because that’s the monorepo pattern. What you’re thinking of is a monolith. Two separate concepts
    - selcuka 2y ago
      > So this example is a single repo with many services? Yes, that's the meaning of monorepo. If it was a single service it would be a monolith. > Why would you have a single repo with different requirements? Real world example: I implemented a client/server system using Python, and they have completely different requirements (even the Python versions are different). I still want to share code between client and server codebases, so a monorepo is the perfect choice.
  - gen220 2y ago
    The second option is traditionally what people mean by "monorepo" (multiple executables in a repo with a dependency tree). First one you'd call a "monolith" (single executable in a repo). The second one you'd want to manage with a tool like Bazel, which will use whatever plugins are appropriate for the language (pip in Python's case). It's a legitimate ambition to try and build a tool within a single language's ecosystem to manage the second option, but it's a really hard problem that Bazel (and others) have already solved well, so you might as well use them instead.
- dang 2y ago
  Personal attacks are not allowed on HN and we ban accounts that cross into that, so please don't do that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful. p.s. Fortunately most of your HN posts appear to be pretty good so this should be easy to fix!
  - crabbone 2y ago
    How is this personal? I have no idea who the person is... I'm commenting on something they wrote, not on something they are. If a person in a home-cooking forum advises someone to use a butter knife to slice tomatoes, it's highly appropriate to tell them they are incompetent and shouldn't be advising anyone. This post is exactly the same: an absurd advise that can only be explained by either an honest mistake or incompetence.
    - kstrauser 2y ago
      They didn’t say they were doing something goofy like that. They said they were running into problems at work. It could well be that they have a million-LOC repo that’s enormously complex with decades of legacy setup to consider. Presume that they’re competent at their job, and maybe what they’re trying to do isn’t trivially easy.
    - pvg 2y ago
      it's highly appropriate to tell them they are incompetent It's not on HN because it's a personal swipe and the HN guidelines ask you not to do those. It might work on your cooking forum but we know, empirically, that it doesn't work well here.
akdor1154 2y ago
I'm in a similar boat - uv's workspaces look tantalising.
kapilvt 2y ago
yeah. its not great, we had to build out a poetry plugin that worked for our cases to support a mono repo, https://github.com/cloud-custodian/poetry-plugin-freeze?tab=readme-ov-file#mono-repo-support https://github.com/cloud-custodian/poetry-plugin-freeze?tab=... the uv support on workspaces (virtual and concrete) has me intrigued.
forrestthewoods 2y ago
> I manage a python monorepo in prod and dependency management is hell. Why does it work for FAANG but not for you?
- tadfisher 2y ago
  FAANGs vendor everything so there's only one version of a given dependency, and they pay people to make this work.
  - forrestthewoods 2y ago
    Why does that work for FAANG but not for drawnwren? How difficult is it to support “one version of a given dependency”?
    - dwattttt 2y ago
      It's as difficult as: Package A depends on C 1.0 and B depends on C 2.0. How much work it is to get down to one version of C in your dependencies is up to how different 1.0 & 2.0 is, and how A & B use it. But if you want them resolved, it's up to you to do the engineering to A or B.
      - forrestthewoods 2y ago
        But pip doesn’t and can’t solve that issue, right?
        dwattttt 2y ago
        Correct. The solution is to modify packages A and or B, which is a high cost approach (hence why FAANGs could throw warm bodies at it, but most everyone else throws their hands up in exasperation).
  - zokier 2y ago
    Bingo, this is what I was referring to in my comment. And yes, I assume it is quite a lot of labor. But I feel that lot of this perceived friction is from people trying to cut corners and avoid doing that labor, while still getting the benefits.
0x008 2y ago
Poetry support for mono repos is really horrible and all the maintainers are saying is „it’s a tool indented to publish packages, not manage your dev environment“.
- Narushia 2y ago
  Poetry has a non-package mode, though…
  - 0x008 2y ago
    What is this non-package mode you are referring to? I was talking about the maintainers not wanting to include features a lot of the userbase would like (like monorepo stuff), because they are saying their target audience is package authors, while in fact most of the users aren't.
zokier 2y ago
Are you managing your deps in the monorepo too, or do you have some sort of half-mono setup?
carderne 2y ago
I’ve been working on a thing [1] to make monorepo (“workspace”) builds easier, works well with Rye/uv. Doesn’t do anything during dev, just removes the need for hacks and scripts at build-time. Would be curious if anyone thinks this is a useful direction, ultimately hope uv/hatch include something like this. [1] https://github.com/carderne/una https://github.com/carderne/una

crabbone 2y ago

[flagged]

miohtama 2y ago
Generally all Python software use third party packages which are fetched from PyPi package site. These packages are not stored in monorepo.
- TheCleric 2y ago
  Not necessarily. Some people do vendor their packages in the repo for consistency (though admittedly it’s rarer).
- crabbone 2y ago
  This is maybe common, but this contradicts the definition of monorepo. You just use the word incorrectly. "Mono" means "one". If you pull packages from elsewhere, that stops being "mono". There's really no difference in this situation between your team publishing multiple packages from multiple repositories and then assembling them together for the purpose of deployment, or doing so, but with the third-party packages.
dudus 2y ago
You still have dependencies. But they are included in the repo. What happens when you have 2 different apps in your monorepo but one uses an older version of Django and wasn't upgraded yet. The monorepo doesn't handle that automatically, you need tooling. It's not as black and white as you say.
- crabbone 2y ago
  In a company I worked for that used monorepo we had multiple versions of Linux (CentOS 6 variants) all at the same time. Trust me, something like different versions of Django is not a really big problem in comparison. But, to answer your question in a more practical way: what are you going to do with these two versions of Django? Are you planning on running a single pre-fork server with two different Django application servers? Are there going to be two different pre-fork servers? Do both Django versions have to be loaded by the same Python interpreter? Or maybe they don't even need to be deployed on the same compute node? Once you can answer questions like these, the solution becomes obvious. Most likely, what you want is something like two pre-fork servers proxying HTTP traffic into two separate instances of Django-based Python Web applications. So, in your deployment script, you create eg. two virtual environments and place two different Django packages into these two environments, together with associated code for the Web application. There are, of course, ways to improve on that. Deploying while using Python packaging is, in general, wasteful. A better way is to merge all the packages you need to deploy your application into a single filesystem snapshot (removing all the info directories, and, potentially, all the source files, replacing them with bytecompiled ones, while also pruning all other irrelevant data from Python packages, s.a. readmes, test files etc.) Going even further, you can Cythonize your code, or even embed Python interpreter into your http server so that you deploy your application as a single binary. And there are plenty more of other options. The sky is the limit really.
drawnwren 2y ago
So, you have two choices for directory layout in mono either: root/ requirements.txt / pyproject.toml src/ --- in which case you need to be careful about when your dependencies get installed during deployment because ie pytorch can be ~500mb your other option is root/ src/ A/ requirements.txt / pyproject.toml B/ requirements.txt / pyproject.toml which means you need to come up with a bespoke dependency management solution. in either case, you're doing dependency management. I encourage you to google "monorepo {python, poetry, pip}" and you're going to land on many multiple page blog posts describe arcane dependency management solutions.
- animal_spirits 2y ago
  So this example is a single repo with many services? I don’t think OP was talking about that. Why would you have a single repo with different requirements? At that point they should be independent projects
  - lijok 2y ago
    Because that’s the monorepo pattern. What you’re thinking of is a monolith. Two separate concepts
  - selcuka 2y ago
    > So this example is a single repo with many services? Yes, that's the meaning of monorepo. If it was a single service it would be a monolith. > Why would you have a single repo with different requirements? Real world example: I implemented a client/server system using Python, and they have completely different requirements (even the Python versions are different). I still want to share code between client and server codebases, so a monorepo is the perfect choice.
- gen220 2y ago
  The second option is traditionally what people mean by "monorepo" (multiple executables in a repo with a dependency tree). First one you'd call a "monolith" (single executable in a repo). The second one you'd want to manage with a tool like Bazel, which will use whatever plugins are appropriate for the language (pip in Python's case). It's a legitimate ambition to try and build a tool within a single language's ecosystem to manage the second option, but it's a really hard problem that Bazel (and others) have already solved well, so you might as well use them instead.
dang 2y ago
Personal attacks are not allowed on HN and we ban accounts that cross into that, so please don't do that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful. p.s. Fortunately most of your HN posts appear to be pretty good so this should be easy to fix!
- crabbone 2y ago
  How is this personal? I have no idea who the person is... I'm commenting on something they wrote, not on something they are. If a person in a home-cooking forum advises someone to use a butter knife to slice tomatoes, it's highly appropriate to tell them they are incompetent and shouldn't be advising anyone. This post is exactly the same: an absurd advise that can only be explained by either an honest mistake or incompetence.
  - kstrauser 2y ago
    They didn’t say they were doing something goofy like that. They said they were running into problems at work. It could well be that they have a million-LOC repo that’s enormously complex with decades of legacy setup to consider. Presume that they’re competent at their job, and maybe what they’re trying to do isn’t trivially easy.
  - pvg 2y ago
    it's highly appropriate to tell them they are incompetent It's not on HN because it's a personal swipe and the HN guidelines ask you not to do those. It might work on your cooking forum but we know, empirically, that it doesn't work well here.

akdor1154 2y ago

I'm in a similar boat - uv's workspaces look tantalising.

kapilvt 2y ago

yeah. its not great, we had to build out a poetry plugin that worked for our cases to support a mono repo, https://github.com/cloud-custodian/poetry-plugin-freeze?tab=readme-ov-file#mono-repo-support https://github.com/cloud-custodian/poetry-plugin-freeze?tab=... the uv support on workspaces (virtual and concrete) has me intrigued.

forrestthewoods 2y ago

> I manage a python monorepo in prod and dependency management is hell. Why does it work for FAANG but not for you?

tadfisher 2y ago
FAANGs vendor everything so there's only one version of a given dependency, and they pay people to make this work.
- forrestthewoods 2y ago
  Why does that work for FAANG but not for drawnwren? How difficult is it to support “one version of a given dependency”?
  - dwattttt 2y ago
    It's as difficult as: Package A depends on C 1.0 and B depends on C 2.0. How much work it is to get down to one version of C in your dependencies is up to how different 1.0 & 2.0 is, and how A & B use it. But if you want them resolved, it's up to you to do the engineering to A or B.
    - forrestthewoods 2y ago
      But pip doesn’t and can’t solve that issue, right?
      - dwattttt 2y ago
        Correct. The solution is to modify packages A and or B, which is a high cost approach (hence why FAANGs could throw warm bodies at it, but most everyone else throws their hands up in exasperation).
- zokier 2y ago
  Bingo, this is what I was referring to in my comment. And yes, I assume it is quite a lot of labor. But I feel that lot of this perceived friction is from people trying to cut corners and avoid doing that labor, while still getting the benefits.

0x008 2y ago

Poetry support for mono repos is really horrible and all the maintainers are saying is „it’s a tool indented to publish packages, not manage your dev environment“.

Narushia 2y ago
Poetry has a non-package mode, though…
- 0x008 2y ago
  What is this non-package mode you are referring to? I was talking about the maintainers not wanting to include features a lot of the userbase would like (like monorepo stuff), because they are saying their target audience is package authors, while in fact most of the users aren't.

zokier 2y ago

Are you managing your deps in the monorepo too, or do you have some sort of half-mono setup?

carderne 2y ago

I’ve been working on a thing [1] to make monorepo (“workspace”) builds easier, works well with Rye/uv. Doesn’t do anything during dev, just removes the need for hacks and scripts at build-time. Would be curious if anyone thinks this is a useful direction, ultimately hope uv/hatch include something like this. [1] https://github.com/carderne/una https://github.com/carderne/una