3 ms·

Way back when, I used to vendor all the libraries for a project (Java/Cpp/Python) into a mono repo and integrate building everything into the projects build fil

Way back when, I used to vendor all the libraries for a project (Java/Cpp/Python) into a mono repo and integrate building everything into the projects build files so anyone could rebuild the entire app stack with whatever compiler flags they wanted. It worked great, but it took diligence, it also forces you to interact with your deps in ways that adding a line to a deps file does not.

cmrdporcupine 1y ago
This is the default way of doing things in the monorepo(s) at Google. It feels like torture until you see the benefits, and the opposite ... the tangled mess of multiple versions and giant transitive dependency chains... agony. I would prefer to work in shops that manage their dependencies this way. It's hard to find.
- I_AM_A_SMURF 1y ago
  I've never seen a place that does it quite like Google. Is there one? It only works if you have one product or are a giant company as it's really expensive to do. Being able to change a dependency very deep and recompile the entire thing is just magic though. I don't know if I can ever go back from that.
- ArthurStacks 1y ago
  [flagged]
  - cmrdporcupine 1y ago
    It absolutely is so, or was for the 10 years I was there. I worked on Google3 (in Ads, on Google WiFi, on Fiber, and other stuff), in Chromium/Chromecast, Fiber, and on Stadia, and every single one of those repos -- all different repositories -- used vendored deps.
- bonzini 1y ago
  It's the same that we're doing for external crates in QEMU's experiments with Rust. Each new dependency is added to the build by hand.
ardit33 1y ago
Maven was the one the started the downfall into dependency hell. (Ant as well, but it was harder to blindly include things into it) Kids today don't know how to do that anymore...
- deepsun 1y ago
  Yet Maven repository is still not that bloated even after 20+ years Java et al. being one of the most popular language. Compared to Rust where my experience with protobuf lib some time ago was that there is a choice of not 1 but even 3 different libraries, one of which doesn't support services, another didn't support the syntax we had to support, and the third one was unmaintained. So out of 3 choices no single one worked. Compared that to Maven, where you have only one official supported choice that works well and well maintained.
  - bonzini 1y ago
    More time enables more consolidation.
    - deepsun 1y ago
      No, there were never several unofficial libraries, one of which eventually won the popularity contest. There was always only one official. There is some barrier to add your project there, so might be that helped. It's even more pronounced with the main Java competitor: .Net. They look at what approach won in Java ecosystem and go all in. For example there were multiple ORM tools competing, where Microsoft adopted the most popular one. So it's even easier choice there, well supported and maintained.
      - bonzini 1y ago
        > Microsoft adopted the most popular one That's still consolidation, and it also needs time. Even in Rust crates like hashbrown or parkinglot have been basically subsumed in the standard library.
saagarjha 1y ago
One nice thing about cargo is that it builds all your code together, which means you can pass a unified set of flags to everything. The feature of building everything all the time as a whole has a bunch of downsides, many which are mentioned elsewhere, but the specific problem of not being able to build dependencies the way you want isn't one.
silon42 1y ago
I would absolutely do this for any non-toy project. Alternatively, for some project it might be enough to only depend on stuff provided by Debian stable or some other LTS distro.

cmrdporcupine 1y ago

This is the default way of doing things in the monorepo(s) at Google. It feels like torture until you see the benefits, and the opposite ... the tangled mess of multiple versions and giant transitive dependency chains... agony. I would prefer to work in shops that manage their dependencies this way. It's hard to find.

I_AM_A_SMURF 1y ago
I've never seen a place that does it quite like Google. Is there one? It only works if you have one product or are a giant company as it's really expensive to do. Being able to change a dependency very deep and recompile the entire thing is just magic though. I don't know if I can ever go back from that.
ArthurStacks 1y ago
[flagged]
- cmrdporcupine 1y ago
  It absolutely is so, or was for the 10 years I was there. I worked on Google3 (in Ads, on Google WiFi, on Fiber, and other stuff), in Chromium/Chromecast, Fiber, and on Stadia, and every single one of those repos -- all different repositories -- used vendored deps.
bonzini 1y ago
It's the same that we're doing for external crates in QEMU's experiments with Rust. Each new dependency is added to the build by hand.

ardit33 1y ago

Maven was the one the started the downfall into dependency hell. (Ant as well, but it was harder to blindly include things into it) Kids today don't know how to do that anymore...

deepsun 1y ago
Yet Maven repository is still not that bloated even after 20+ years Java et al. being one of the most popular language. Compared to Rust where my experience with protobuf lib some time ago was that there is a choice of not 1 but even 3 different libraries, one of which doesn't support services, another didn't support the syntax we had to support, and the third one was unmaintained. So out of 3 choices no single one worked. Compared that to Maven, where you have only one official supported choice that works well and well maintained.
- bonzini 1y ago
  More time enables more consolidation.
  - deepsun 1y ago
    No, there were never several unofficial libraries, one of which eventually won the popularity contest. There was always only one official. There is some barrier to add your project there, so might be that helped. It's even more pronounced with the main Java competitor: .Net. They look at what approach won in Java ecosystem and go all in. For example there were multiple ORM tools competing, where Microsoft adopted the most popular one. So it's even easier choice there, well supported and maintained.
    - bonzini 1y ago
      > Microsoft adopted the most popular one That's still consolidation, and it also needs time. Even in Rust crates like hashbrown or parkinglot have been basically subsumed in the standard library.

saagarjha 1y ago

One nice thing about cargo is that it builds all your code together, which means you can pass a unified set of flags to everything. The feature of building everything all the time as a whole has a bunch of downsides, many which are mentioned elsewhere, but the specific problem of not being able to build dependencies the way you want isn't one.

silon42 1y ago

I would absolutely do this for any non-toy project. Alternatively, for some project it might be enough to only depend on stuff provided by Debian stable or some other LTS distro.