21 ms·

Dependencies Belong in Version Control

59 points by crummy 3y ago

fonnyx 3y ago
[flagged]
o11c 3y ago
> Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies and that will never happen. And here is the true root of the problem - people use build systems as if they were package managers. Use a real package manager (and install your dependencies before the build) and suddenly it is clear why dependencies do not, in fact, belong in version control.
- api 3y ago
  If you mean an OS package manager, now your dependencies are all really old and you have only one version to choose from. There’s a reason Rust and Go built their own dependency fetching systems and other languages like Java has had them for years.
  - reactordev 3y ago
    And you should use them instead of checking in Jar files into your repo…
    - api 3y ago
      So your builds fail if a remote site goes down, and the speed of that site rate limits your CI.
      - reactordev 3y ago
        Nexus/Artifactory my friend. If your site is down, you have bigger issues on your hands. If the remote end is down, cache is there to save you.
  - shortrounddev2 3y ago
    vcpkg is really awesome, but the versioning system has something to be desired
- gravypod 3y ago
  I don't think a package manager solves this case. If you own a repo with software in it you'll make sure to back up everything in that repo. If part of the build of your code in your repo fetches something from the network that will eventually fail and you will no longer be able to build your software again. I recently had someone ping me for a build of software that I wrote >10years ago. I couldn't build the code in the repo since I didn't contain the deps in my repo and the servers my build scripts reached out to where down.
  - amluto 3y ago
    It’s worse than just that. Say I pick Fedora as my base. Now my project is a pile of SRPMs, that need a nasty bootstrap to build (okay, this is hard to avoid), and I’m stuck using the Fedora build system. Which sucks, sorry. (Okay, at least there is only one Fedora build system.) Also a whole OS is dragged along. If I want to edit a dependency, eww, I have to commit patch files and muck with .spec files. Good times. Good tooling like the author describes could exist. No distro I’ve ever used is it. Gentoo is probably closer than Fedora, as Gentoo is actually intended to build packages as opposed to just running them.
jiggawatts 3y ago
This wouldn’t be a problem if the default in VCS systems was to use something like S3 blob storage by default for large binary files. Just store the torrent-like Merkle tree hash in the VCS database.
- eternityforest 3y ago
  I wish packages managers had P2P fetching from machines on the LAN as a fully supported built-in feature. Unfortunately, if they did so, they'd probably use IPFS, and I don't know when they're going to fix their idle bandwidth issues, a lot of people seem to just give up and use web gateways, completely defeating the purpose of working without single points of failure.
- dharmab 3y ago
  https://github.com/microsoft/p4vfs https://github.com/microsoft/p4vfs
gravypod 3y ago
I think something like this, if someone could build it, would be amazing. Essentially, this would bring the benefits of a BigCo monorepo (and all the SWE-time performance benefits it has) to the rest of the world. Lots of nice things could come from tech like this being adopted. I don't think it would get mass adoption though. Git+GitHub+$PackageManager is "good enough" and this approach wouldn't be significantly better for every use case.
- cxr 3y ago
  > Git+GitHub+$PackageManager is "good enough" Weird way to frame it. Surely if that's good enough, then Git + repo host alone, sans package manager, satisfies the same criterion. It didn't/doesn't become more complicated by omitting package managers like npm, cargo, etc., and their associated methodology from the equation. It's the other way around. Adding their package management philosophy/methodology into the fray is strictly more complicated than not. It's extra.
api 3y ago
“Vendoring” dependencies was a good idea that fell out of favor because it wasn’t quite implemented right. It should be revisited.
rch 3y ago
I'm annoyed that ever few months I'll start something new that I know will eventually run on Kudu and Impala (per my employer), but the local build requirements are such that it's more effective to start with Postgres and figure out porting later on. As a NixOS user, I know the answer, but I just haven't allocated the time. Maybe this holiday season... Advent of Nix or something.
echelon 3y ago
This top-down, prescriptive suggestion is wrong. The truth is that it depends on project construction, language, build toolchain, operating system support, and libraries. Python projects are a particular hell as the multiple attempts to solve dependencies didn't capture transitive dependencies well. Python also builds against native dynamically linked libraries, which introduces additional hell. But that's Python. The author is trying to use ML projects on Windows, and he probably hasn't realized that academics authoring this code aren't familiar with best practices, aren't writing multi-platform support, and aren't properly and hermetically packaging their code. To compare Python to a few other languages: - Rust projects are almost always incredibly hermetic. You really only need to vendor something in if you want to make changes to it. - Modern Javascript is much better than the wild west of ten years ago. Modern tools capture repeatable builds. Don't go checking in built binaries unless you understand that your particular set of software needs it. Know the right tools for the job.
jmyeet 3y ago
No. No, they don't. Specifically, binary dependencies don't belong in a repo and you want to use binary dependencies rather than source dependencies where possible. Once again, we see 20 years of dependency systems that have failed to do what Maven established as the bare minimum. Specifically: 1. Create signed versions of dependencies so you can't rebuild a given release; 2. Allow you to specify a specific version, the latest version or even the latest minor version of a specific major version; 3. Allow you to run internal repos in corporate environments where you might want to publish private libraries; and 4. Version information is nowhere near any source code. Putting github URLs in Go source files is the most egregious example of bad dependency management from a language in recent history. Every line of source code, whether its yours or third-party comes at a cost. Depending on your toolchain, this may well increase compilation time and required resources. You want reproducible builds. If you can do that without putting every dependency in a repo then you should. If you can't then you have a bad dependency system.
- shortrounddev2 3y ago
  I think the only C/C++ package manager that is even close to maven is vcpkg
  - jmyeet 3y ago
    Header file is both a huge strength and weakness of C/C++ and are really an anachronism now. It's very general purpose text substitution, which is very powerful, but means you have to include .h files to get types, signatures and so on. This kills binary compatibility (specificially, if you're linking to static or shared libraries, you still need a .h file). Even C++ templates aren't really much better and they're still text replacement. More modern languages have taken the use cases for .h files and incorporated them without the general purpose text substitution. Like Rust's macros a superior to #define macros. Type aliases (in various langauges) are better than #define uint32 unsigned int.
    - shortrounddev2 3y ago
      Modern C++ also has constexpr, consteval, and using aliases
justinwp 3y ago
> This is exactly how Google and Meta operate today I wish it was that great. Works until you need to have a different version than some other piece of code, import a dependency that requires 100 others, or you need to build for some other platform than the "blessed" target. I choose google3 only when I have to or when I am certain of the scope. (I am in DevRel, so I have more options than typical engineers for my work)
reactordev 3y ago
>“My background is C++ gamedev” This is why you think this way. Your proposal is not new. The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Every other language has had package managers for this very reason. Where do you draw the line? OS libs and package manager is ok but it’s not for a developer? Go learn vcpkg and come back to us when you learn why everyone does it this way.
- codetrotter 3y ago
  > The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Exactly. With Rust you can commit your Cargo.lock file and you will then be able to rebuild your project with the exact same version of all your deps in the future. No need to commit the deps themselves.
  - kibwen 3y ago
    And cargo-vendor is available out of the box as well (although it doesn't go quite so far as to add the entire Rust compiler toolchain to your repo): https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
    - reactordev 3y ago
      Exactly, In case you need to transport your dependencies and build air-gapped (happens) or you need to make some modifications (keep them as patches please, to be re-applied). Not so you can check them all into your repo. You should .gitignore your vendor folder.
  - HumanOstrich 3y ago
    Of course, just rewrite everything in Rust! It's so easy!
    - 3y ago
pmorici 3y ago
Isn't this the same kind of attitude around containers. "I don't want to think about or document the dependencies" lets just throw it all in a container full of crap that no one fully understands and it will "just work" because it worked for someone once before. One of the things that I find very useful is to start from a base install of a particular OS and then be very meticulous about documenting each package I need to install to get software to build. You can even put this into the documentation and automate checking the dependencies are there with the system package manager. The dependencies and how you check them will be different across different distros and versions but at least you had an understanding at one point to work from if you need to figure it out going forward.
- drewcoo 3y ago
  The putting into containers is a repeatable recipe. One would hope that everything you put in is also built from a repeatable recipe. The point is that you can reuse the container - that environment - and know that it was always built the same as any other copy. Because, especially with the way some people use package mangers, those "repeatable" recipes are not always repeatable.
- lsaferite 3y ago
  > lets just throw it all in a container full of crap that no one fully understands I mean, my usage of containers is very tightly controlled. I prefer to start with the scratch container in most cases. If I'm using software that needs more, then I'm very deliberate is what packages make it into the build container and the deploy container. So, maybe some just toss in the kitchen sink, others are deliberate about build and runtime deps.
malkia 3y ago
This came out this year from Perforce - https://www.perforce.com/blog/vcs/what-is-virtual-file-sync-p4vfs https://www.perforce.com/blog/vcs/what-is-virtual-file-sync-...
- malkia 3y ago
  There is also this - https://github.com/microsoft/p4vfs https://github.com/microsoft/p4vfs and several other solutions - just need to dig around.
  - forrestthewoods 3y ago
    Microsoft also has VFSforGit. Sadly they abandoned it to pursue sparse clones. I'm not sure the full story why. :(
eternityforest 3y ago
Seems like what all we would need on the VCS side is something like Git LFS but with global chunk based deduplication, binary file patching algorithms, which serves the actual files over bittorent. The last part is essential, because GitHub LFS is too expensive for anyone to just try out on their own. But then, on the dev tool side, we would need automated ways to get all that stuff in the repo in the first place, and make sure that the IDE linting and autocomplete knows about it. I used to put my python dependencies in a thirdparty folder, and have a line that alters the sys.path to point at it. I just spent a week getting rid of all that and using PyPi, because it didn't play nice with linters and I couldn't swap different versions of dependencies with virtualenv, updating required manual work, and there was no management to make sure there wasn't version conflicts in subtle ways. I like the idea of deps in VCS, but not as much as I like using tools as the devs intended and sticking to the official workflow everyone else uses.
- reactordev 3y ago
  This is horrible. Just use vcpkg or any number of other c++ package managers. Pypi exists for a reason. Maven and gradle exist for a reason. Nuget exists for a reason. NPM/Yarn as well. Storing your dependencies with your code ensures you will be out of date, vulnerable to whatever vulnerabilities have been patched since then, and that your build will produce a different hash so windows defender will do a full binary scan on you every time. Not to mention an all-hands on deck weekend holiday to upgrade.
  - cxr 3y ago
    > Storing your dependencies with your code ensures you will be out of date, vulnerable to whatever vulnerabilities have been patched since then I missed the part where having a copy of the dependencies you last built with means that you are required to never ever ever make use of any bugfixes that have come out since the first time those dependencies were committed.
    - eternityforest 3y ago
      If you don't have automated management, you will probably not regularly check for updates on every one of dozens of dependencies manually.
adrianmonk 3y ago
> Source code, binary assets, third-party libraries, and even compiler toolchains. Everything. How far down the stack of turtles do you go, though? Should you include the libc that the compiler requires to run? Other parts of the operating system? The kernel? An emulator for hardware that can run that kernel? Eventually, all of those things will stop being produced or easy to find. Even if you have the libraries and compiler in version control, can you build a game that ran on MS-DOS 5.x or CP/M or DEC VMS? My point is that you may want to just designate a stable interface somewhere (a language standard, some libc ABI, etc.) as the part you expect to not change. Be aware of what it is, and account for it in your plans. If a language is the interface that you expect to be stable, then don't upgrade to a compiler version that breaks compatibility. Or do upgrade, but do it in an orderly manner in conjunction with porting your code over. If you want your code to be runnable by our descendants 1000 years from now, you should probably have an incredibly simple VM that can be described in a picture and a binary that will run on that VM. (In other words, you go down so many turtles that you reach "anyone who knows how to code or build machines can implement this VM".)
- lebean 3y ago
  Everything! Check in your OS's system files! Build a new PC with parts identical to the one you're building on! Build many of them, one for each commit, and put them all in a warehouse!
  - adrianmonk 3y ago
    I mean, if you're the military, this might be the right answer. If you're making a personal web site, probably not.
- hducy4 3y ago
  This is a strawman. You're extending their argument to an extreme so it sounds silly when it's not what has been proposed. The argument is clearly to keep direct dependencies required for building in source control so that if you have a working build system, you can build the software indefinitely and independently from the internet. Build systems and operating systems don't disappear overnight. Leftpad does.
matisseverduyn 3y ago
"Security" would be a useful benefit/section to add to this post: A.) If maintainers of your dependencies edited an existing/previous version, or B.) If your dependencies did not pin their dependencies. For instance, if you installed vue-cli in May of last year from NPM with --prefer-offline (using the cache / basically the same as checking in your node_modules), you were fine. But because vue-cli doesn't pin its dependencies ("node-ipc"), installing fresh/online would create WITH-LOVE-FROM-AMERICA.txt on your desktop [1], which was at the very least a scare, but for some, incredibly problematic. [1] https://github.com/vuejs/vue-cli/issues/7054 https://github.com/vuejs/vue-cli/issues/7054
brendoncarroll 3y ago
I don't think you need to go quite so far as checking gigabytes of executables into version control. If you download some dependencies at build time, that's fine as long as you know exactly what they are ahead of time. "Exactly what they are" means a hash, not a name and version tag. The dockerized build approach is actually a good strategy, unfortunately it's done by image name instead of image hash in practice. Upgrading dependencies, or otherwise resolving a name and version to a hash is a pre-source task, not a from-source task. Maybe it can be automated, and a bot can generate pull requests to bump the versions, but that happens as a proposed change to the source, not in a from-source task like build, test, publish, or deploy.
malkia 3y ago
.exe may belong in p4, .pdb's not! :)
jmisavage 3y ago
I agree in principle, but you also need the same machine too otherwise new OS and hardware might introduce issues too. I resurrected an old project that needed to convert LESS to CSS and the node version required couldn’t run on my machine. Upgrading it to a version that could introduced filesystem changes that broke the packages that it was looking for. Now just imagine businesses in the middle of a platform shift like Macs going from Intel to Apple’s own ARM chips. Eventually you’ll going to be missing something and all this work of bundling everything will end up being busy work.
spiffytech 3y ago
Is it going to be a problem for your Windows/Linux developers that someone committed a node_modules with macOS-only binaries inside?
simonw 3y ago
If you're concerned about bloating your Git repository with non-unique binary files (as you should be) a trick I've used in the past that worked really well was to have a separate project-dependencies Git repository that all of my dependencies lived in. I was working with Python, so this was effectively a Git-backed cache of the various .tar.gz and .whl files that my project depended on. This worked, and it gave me reassurance that I'd still be able to build my project if I couldn't download packages from PyPI... but to be honest these days I don't bother, because PyPI has proven itself robust enough that it doesn't feel worth the extra effort. I keep meaning to set myself up a simple S3 bucket somewhere with my PyPI dependencies mirrored there, just in case I ever need to deploy a fix in a hurry while PyPI is having an outage.
matrss 3y ago
Nix (https://nix.dev/ https://nix.dev/) can provide all of this, although in a smarter way than just through dumping everything in the VCS. Some projects use it already to provide a reproducible development environment and if done right a clean build is just a `nix-build` away.
- delotrag 3y ago
  Honestly surprised the article didn't mention Nix or Guix. Seems like functional package management solves the exact problems the author is worried about.
  - FoodWThrow 3y ago
    The article starts with a gamedev disclaimer. Most gamedev folks would rather die on that Microsoft hill than use another OS.
- HumanOstrich 3y ago
  It does not provide "all of this" and would just make the experience worse for everyone.
  - matrss 3y ago
    What is it missing? Using nix with nixpkgs you can pin all dependencies to a specific version identified by a hash value of that dependencies source code and dependencies, recursively building a dependency graph down to nixpkgs' bootstrap toolchains. That is deeper down than you could reasonably go by dumping everything in the VCS. Building becomes a single command that fetches everything it needs. Even the offline requirement is satisfiable if you prefetch all dependencies, it is just not happening by default (and for good reason, you might already have the toolchain from another project or some other dependencies are already on your system).
jbverschoor 3y ago
I think you forgot to include: toolchains \win arm \linux x86 \linux arm \macos x86 \macos arm What I do think is that dependencies should be versioned, and their artifacts should be immutable. Dependency management is not the only thing wrong with gamedev
- TeMPOraL 3y ago
  > What I do think is that dependencies should be versioned, and their artifacts should be immutable. And you should keep a copy of those artifacts, and build from it. Having a hash is not enough. People have this weird assumption that you can just fetch all your dependencies by their hashes from the cloud. But cloud is just other people's servers; no one is going to be hosting build artifacts indefinitely just for you.
dboreham 3y ago
This is both right and wrong: If you're shipping shrink-wrap product, or equivalent, then you should freeze everything in carbonite so you can later recreate the thing you released. The article is written as if this is a novel idea but it isn't. Decades ago when I worked on products shipped on CD it was standard procedure to archive the entire build machine to tape and put that in a fire safe. In fact I subsequently (decades later) worked for law firms on patent cases where they were able to get those backup tapes to prove something about the content of software shipped on a specific date. otoh for the typical present-day software project you don't want to re-create an identical build result as someone else got six months ago. For example if it's a JavaScript project then Node is going to be two versions out of date and probably full of security bugs from that time. So you actually want "code that behaves as expected but built with current dependencies and toolchain". Admittedly experience shows that for some languages this is an unreasonable expectation. Some level of ongoing maintenance to the codebase is often required just to keep it building with security-fixed dependencies.
- cxr 3y ago
  > otoh for the typical present-day software project you don't want to re-create an identical build result as someone else got six months ago According to whom? I absolutely want to be able to do this. > if it's a JavaScript project then Node is going to be two versions out of date and probably full of security bugs from that time Sounds like building on something as unstable as Node is a significant drawback. That's a Node problem. (I missed how we got to talking about Node, anyway—you said "JavaScript project". NodeJS does not have a monopoly on JS. It's not even a great example of a particularly good JS runtime; for the reasons you mention—instability—it's actually a pretty good example of a bad JS runtime.) Fortunately if you're doing JS, there's an incredibly stable set of interfaces available for you to program against, and the best part is you don't need to download anything extra because every computer already has a JS runtime installed.
jbverschoor 3y ago
Don't forget https://fossil-scm.org/ https://fossil-scm.org/
mr_tristan 3y ago
The challenge when I see posts like this is the people in charge of building this "check it all in" ecosystem usually forget about the developer experience and basically just implement a CI system. Cool, you can 're-run' an old build cleanly, which is good, but not enough. How about commercial IDEs? Cloud environments? A lot of developer environments these days include a ton of stuff that likely doesn't make sense to check in, usually licensing config is annoying, or because you're relying on runtime services. And all this time engineers spend on their own machines is basically time wasted, which isn't really a great solution to pitch to a business. Side note: I used to work for Perforce until the private equity sale. If there was a platform to vendor everything like this, it would be Perforce, because you could already do this kind of thing for years. AFAIK not many Perforce customers ever did this, and I don't think it was because Perforce wasn't capable. It's just a subtly wicked problem. Getting this right - just check out and go across different software development stacks - requires a lot of investment. It does look like Perforce has been acquiring many other aspects related to the application lifecycle, so in theory, they should be better positioned to be the "vendor everything on our stack" solution, but I'm not convinced this is going to work out well. Cloud development environment vendors seem to be the best positioned as a product for solving this problem, because there is less of that "go figure out your DX" aspect left to the customer. But the right CDE would have to have a lot of enterprise-style controls. This is so new that I'm not sure who will get it right first, but my guess is that we'll get to a more "development to delivery" integrated environment, and away from a hodgepodge of tools managed per project.
Joel_Mckay 3y ago
True in some situations, but a fundamentally flawed approach to FOSS. Indeed, if you are statically linking noncritical code, than for maintainability it is easier to version-lock an entire OS with the build tree in a VM. Thus, the same input spits out the same audited binary objects every time. In some situations it is unavoidable (see Microsoft or FPGA ecosystem). However, a shared object library ecosystem is arguably a key part of FOSS when it works properly on *nix. As it is fundamentally important to keep versioned interoperability with other projects to minimize library RAM footprints etc. Additionally, all projects have a shared interest in maintaining each others security, rather than trying to waste resources on every application that has legacy stripped static obfuscated vulnerable leaky objects. "Reuse [shared objects], Reduce [resource costs], and Recycle [to optimize]." Sounds like familiar advice... =) like how some git solutions don't handle large binary artifacts well or simply implode. Good luck, and maybe one can prove all those cautionary stories wrong. YMMV =)
hervem 3y ago
What about working over a way to build a dev. env. correctly? ansible, container, dev-container.. Git shouldn't be used for backing all dependencies, tooling, and OS.. And while we are at it, do we track git itself in the "toolchain"?
lamontcg 3y ago
You're better off just taking your build artifacts and shoving them into some artifact repo which is an already solved problem. If you want to ship all your runtime deps use an OCI container image, or something like flatpak which allows shipping bundled libraries. Those are the artifacts which should flow through your CI/CD pipeline (which may be as simple as just an automated test stage before manual deployment if you don't like those abbreviations I just used). Beyond that there's not really much utility to being able to rebuild exactly the same thing, because that's what you already shipped. The problem becomes pulling in new stuff and bumping the build, in which case you always have the cost of integration testing all the new code together. I do fully support Gemfile.lock/Cargo.lock/flake.lock/etc files being checked into git and the build process should be bumping those and building and shipping the artifacts off to a repo -- and that's good enough without actually checking all the blobs into git. And the reason why git is bad at LFS and binary blob / artifact support is likely that everyone has adopted this method and so making git better for the use case isn't a very high priority.
- cxr 3y ago
  > You're better off just taking your build artifacts and shoving them into some artifact repo Better off how? What does "better" refer to here? Size on disk? Data transfer? Time? CPU cycles? Brain cycles? Body health? Please be specific. > there's not really much utility to being able to rebuild exactly the same thing, because that's what you already shipped This strikes me as profoundly myopic (maybe even willfully). The unspoken benefit here is related to a general understanding that it's good to be able to establish a base case showing that you can build the same thing in the first place, so that when you make slight changes to any part of it, you end up with exactly the same thing as before plus your changes, rather than being the casualty of hundreds or thousands of other uncontrolled inputs that have nothing to do with what you changed. You're casting aspersions on and summarily dismissing the basic tenet of having a control[1]. > The problem becomes pulling in new stuff and bumping the build, in which case you always have the cost of integration testing all the new code together. Just to emphasize: you are correct to say that this is a cost you always have, whether or not you're checking the results of package installation into your repo. Not checking them in doesn't ipso facto cause any issues related to upgrades to go away, and checking them in doesn't ipso facto increase the number of integration work you have to do. There's nothing magical about pulling the dependencies in over the network at the extreme last moment versus reading them from some other place on your own disk. 1. <https://en.wikipedia.org/wiki/Scientific_control https://en.wikipedia.org/wiki/Scientific_control>

59 points by crummy 3y ago

[flagged]

o11c 3y ago

> Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies and that will never happen. And here is the true root of the problem - people use build systems as if they were package managers. Use a real package manager (and install your dependencies before the build) and suddenly it is clear why dependencies do not, in fact, belong in version control.

api 3y ago
If you mean an OS package manager, now your dependencies are all really old and you have only one version to choose from. There’s a reason Rust and Go built their own dependency fetching systems and other languages like Java has had them for years.
- reactordev 3y ago
  And you should use them instead of checking in Jar files into your repo…
  - api 3y ago
    So your builds fail if a remote site goes down, and the speed of that site rate limits your CI.
    - reactordev 3y ago
      Nexus/Artifactory my friend. If your site is down, you have bigger issues on your hands. If the remote end is down, cache is there to save you.
- shortrounddev2 3y ago
  vcpkg is really awesome, but the versioning system has something to be desired
gravypod 3y ago
I don't think a package manager solves this case. If you own a repo with software in it you'll make sure to back up everything in that repo. If part of the build of your code in your repo fetches something from the network that will eventually fail and you will no longer be able to build your software again. I recently had someone ping me for a build of software that I wrote >10years ago. I couldn't build the code in the repo since I didn't contain the deps in my repo and the servers my build scripts reached out to where down.
- amluto 3y ago
  It’s worse than just that. Say I pick Fedora as my base. Now my project is a pile of SRPMs, that need a nasty bootstrap to build (okay, this is hard to avoid), and I’m stuck using the Fedora build system. Which sucks, sorry. (Okay, at least there is only one Fedora build system.) Also a whole OS is dragged along. If I want to edit a dependency, eww, I have to commit patch files and muck with .spec files. Good times. Good tooling like the author describes could exist. No distro I’ve ever used is it. Gentoo is probably closer than Fedora, as Gentoo is actually intended to build packages as opposed to just running them.

jiggawatts 3y ago

This wouldn’t be a problem if the default in VCS systems was to use something like S3 blob storage by default for large binary files. Just store the torrent-like Merkle tree hash in the VCS database.

eternityforest 3y ago
I wish packages managers had P2P fetching from machines on the LAN as a fully supported built-in feature. Unfortunately, if they did so, they'd probably use IPFS, and I don't know when they're going to fix their idle bandwidth issues, a lot of people seem to just give up and use web gateways, completely defeating the purpose of working without single points of failure.
dharmab 3y ago
https://github.com/microsoft/p4vfs https://github.com/microsoft/p4vfs

gravypod 3y ago

I think something like this, if someone could build it, would be amazing. Essentially, this would bring the benefits of a BigCo monorepo (and all the SWE-time performance benefits it has) to the rest of the world. Lots of nice things could come from tech like this being adopted. I don't think it would get mass adoption though. Git+GitHub+$PackageManager is "good enough" and this approach wouldn't be significantly better for every use case.

cxr 3y ago
> Git+GitHub+$PackageManager is "good enough" Weird way to frame it. Surely if that's good enough, then Git + repo host alone, sans package manager, satisfies the same criterion. It didn't/doesn't become more complicated by omitting package managers like npm, cargo, etc., and their associated methodology from the equation. It's the other way around. Adding their package management philosophy/methodology into the fray is strictly more complicated than not. It's extra.

api 3y ago

“Vendoring” dependencies was a good idea that fell out of favor because it wasn’t quite implemented right. It should be revisited.

rch 3y ago

I'm annoyed that ever few months I'll start something new that I know will eventually run on Kudu and Impala (per my employer), but the local build requirements are such that it's more effective to start with Postgres and figure out porting later on. As a NixOS user, I know the answer, but I just haven't allocated the time. Maybe this holiday season... Advent of Nix or something.

echelon 3y ago

This top-down, prescriptive suggestion is wrong. The truth is that it depends on project construction, language, build toolchain, operating system support, and libraries. Python projects are a particular hell as the multiple attempts to solve dependencies didn't capture transitive dependencies well. Python also builds against native dynamically linked libraries, which introduces additional hell. But that's Python. The author is trying to use ML projects on Windows, and he probably hasn't realized that academics authoring this code aren't familiar with best practices, aren't writing multi-platform support, and aren't properly and hermetically packaging their code. To compare Python to a few other languages: - Rust projects are almost always incredibly hermetic. You really only need to vendor something in if you want to make changes to it. - Modern Javascript is much better than the wild west of ten years ago. Modern tools capture repeatable builds. Don't go checking in built binaries unless you understand that your particular set of software needs it. Know the right tools for the job.

jmyeet 3y ago

No. No, they don't. Specifically, binary dependencies don't belong in a repo and you want to use binary dependencies rather than source dependencies where possible. Once again, we see 20 years of dependency systems that have failed to do what Maven established as the bare minimum. Specifically: 1. Create signed versions of dependencies so you can't rebuild a given release; 2. Allow you to specify a specific version, the latest version or even the latest minor version of a specific major version; 3. Allow you to run internal repos in corporate environments where you might want to publish private libraries; and 4. Version information is nowhere near any source code. Putting github URLs in Go source files is the most egregious example of bad dependency management from a language in recent history. Every line of source code, whether its yours or third-party comes at a cost. Depending on your toolchain, this may well increase compilation time and required resources. You want reproducible builds. If you can do that without putting every dependency in a repo then you should. If you can't then you have a bad dependency system.

shortrounddev2 3y ago
I think the only C/C++ package manager that is even close to maven is vcpkg
- jmyeet 3y ago
  Header file is both a huge strength and weakness of C/C++ and are really an anachronism now. It's very general purpose text substitution, which is very powerful, but means you have to include .h files to get types, signatures and so on. This kills binary compatibility (specificially, if you're linking to static or shared libraries, you still need a .h file). Even C++ templates aren't really much better and they're still text replacement. More modern languages have taken the use cases for .h files and incorporated them without the general purpose text substitution. Like Rust's macros a superior to #define macros. Type aliases (in various langauges) are better than #define uint32 unsigned int.
  - shortrounddev2 3y ago
    Modern C++ also has constexpr, consteval, and using aliases

justinwp 3y ago

> This is exactly how Google and Meta operate today I wish it was that great. Works until you need to have a different version than some other piece of code, import a dependency that requires 100 others, or you need to build for some other platform than the "blessed" target. I choose google3 only when I have to or when I am certain of the scope. (I am in DevRel, so I have more options than typical engineers for my work)

reactordev 3y ago

>“My background is C++ gamedev” This is why you think this way. Your proposal is not new. The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Every other language has had package managers for this very reason. Where do you draw the line? OS libs and package manager is ok but it’s not for a developer? Go learn vcpkg and come back to us when you learn why everyone does it this way.

codetrotter 3y ago
> The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Exactly. With Rust you can commit your Cargo.lock file and you will then be able to rebuild your project with the exact same version of all your deps in the future. No need to commit the deps themselves.
- kibwen 3y ago
  And cargo-vendor is available out of the box as well (although it doesn't go quite so far as to add the entire Rust compiler toolchain to your repo): https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
  - reactordev 3y ago
    Exactly, In case you need to transport your dependencies and build air-gapped (happens) or you need to make some modifications (keep them as patches please, to be re-applied). Not so you can check them all into your repo. You should .gitignore your vendor folder.
- HumanOstrich 3y ago
  Of course, just rewrite everything in Rust! It's so easy!
  - 3y ago

pmorici 3y ago

Isn't this the same kind of attitude around containers. "I don't want to think about or document the dependencies" lets just throw it all in a container full of crap that no one fully understands and it will "just work" because it worked for someone once before. One of the things that I find very useful is to start from a base install of a particular OS and then be very meticulous about documenting each package I need to install to get software to build. You can even put this into the documentation and automate checking the dependencies are there with the system package manager. The dependencies and how you check them will be different across different distros and versions but at least you had an understanding at one point to work from if you need to figure it out going forward.

drewcoo 3y ago
The putting into containers is a repeatable recipe. One would hope that everything you put in is also built from a repeatable recipe. The point is that you can reuse the container - that environment - and know that it was always built the same as any other copy. Because, especially with the way some people use package mangers, those "repeatable" recipes are not always repeatable.
lsaferite 3y ago
> lets just throw it all in a container full of crap that no one fully understands I mean, my usage of containers is very tightly controlled. I prefer to start with the scratch container in most cases. If I'm using software that needs more, then I'm very deliberate is what packages make it into the build container and the deploy container. So, maybe some just toss in the kitchen sink, others are deliberate about build and runtime deps.

malkia 3y ago

This came out this year from Perforce - https://www.perforce.com/blog/vcs/what-is-virtual-file-sync-p4vfs https://www.perforce.com/blog/vcs/what-is-virtual-file-sync-...

malkia 3y ago
There is also this - https://github.com/microsoft/p4vfs https://github.com/microsoft/p4vfs and several other solutions - just need to dig around.
- forrestthewoods 3y ago
  Microsoft also has VFSforGit. Sadly they abandoned it to pursue sparse clones. I'm not sure the full story why. :(

eternityforest 3y ago

Seems like what all we would need on the VCS side is something like Git LFS but with global chunk based deduplication, binary file patching algorithms, which serves the actual files over bittorent. The last part is essential, because GitHub LFS is too expensive for anyone to just try out on their own. But then, on the dev tool side, we would need automated ways to get all that stuff in the repo in the first place, and make sure that the IDE linting and autocomplete knows about it. I used to put my python dependencies in a thirdparty folder, and have a line that alters the sys.path to point at it. I just spent a week getting rid of all that and using PyPi, because it didn't play nice with linters and I couldn't swap different versions of dependencies with virtualenv, updating required manual work, and there was no management to make sure there wasn't version conflicts in subtle ways. I like the idea of deps in VCS, but not as much as I like using tools as the devs intended and sticking to the official workflow everyone else uses.

reactordev 3y ago
This is horrible. Just use vcpkg or any number of other c++ package managers. Pypi exists for a reason. Maven and gradle exist for a reason. Nuget exists for a reason. NPM/Yarn as well. Storing your dependencies with your code ensures you will be out of date, vulnerable to whatever vulnerabilities have been patched since then, and that your build will produce a different hash so windows defender will do a full binary scan on you every time. Not to mention an all-hands on deck weekend holiday to upgrade.
- cxr 3y ago
  > Storing your dependencies with your code ensures you will be out of date, vulnerable to whatever vulnerabilities have been patched since then I missed the part where having a copy of the dependencies you last built with means that you are required to never ever ever make use of any bugfixes that have come out since the first time those dependencies were committed.
  - eternityforest 3y ago
    If you don't have automated management, you will probably not regularly check for updates on every one of dozens of dependencies manually.

adrianmonk 3y ago

> Source code, binary assets, third-party libraries, and even compiler toolchains. Everything. How far down the stack of turtles do you go, though? Should you include the libc that the compiler requires to run? Other parts of the operating system? The kernel? An emulator for hardware that can run that kernel? Eventually, all of those things will stop being produced or easy to find. Even if you have the libraries and compiler in version control, can you build a game that ran on MS-DOS 5.x or CP/M or DEC VMS? My point is that you may want to just designate a stable interface somewhere (a language standard, some libc ABI, etc.) as the part you expect to not change. Be aware of what it is, and account for it in your plans. If a language is the interface that you expect to be stable, then don't upgrade to a compiler version that breaks compatibility. Or do upgrade, but do it in an orderly manner in conjunction with porting your code over. If you want your code to be runnable by our descendants 1000 years from now, you should probably have an incredibly simple VM that can be described in a picture and a binary that will run on that VM. (In other words, you go down so many turtles that you reach "anyone who knows how to code or build machines can implement this VM".)

lebean 3y ago
Everything! Check in your OS's system files! Build a new PC with parts identical to the one you're building on! Build many of them, one for each commit, and put them all in a warehouse!
- adrianmonk 3y ago
  I mean, if you're the military, this might be the right answer. If you're making a personal web site, probably not.
hducy4 3y ago
This is a strawman. You're extending their argument to an extreme so it sounds silly when it's not what has been proposed. The argument is clearly to keep direct dependencies required for building in source control so that if you have a working build system, you can build the software indefinitely and independently from the internet. Build systems and operating systems don't disappear overnight. Leftpad does.

matisseverduyn 3y ago

"Security" would be a useful benefit/section to add to this post: A.) If maintainers of your dependencies edited an existing/previous version, or B.) If your dependencies did not pin their dependencies. For instance, if you installed vue-cli in May of last year from NPM with --prefer-offline (using the cache / basically the same as checking in your node_modules), you were fine. But because vue-cli doesn't pin its dependencies ("node-ipc"), installing fresh/online would create WITH-LOVE-FROM-AMERICA.txt on your desktop [1], which was at the very least a scare, but for some, incredibly problematic. [1] https://github.com/vuejs/vue-cli/issues/7054 https://github.com/vuejs/vue-cli/issues/7054

brendoncarroll 3y ago

I don't think you need to go quite so far as checking gigabytes of executables into version control. If you download some dependencies at build time, that's fine as long as you know exactly what they are ahead of time. "Exactly what they are" means a hash, not a name and version tag. The dockerized build approach is actually a good strategy, unfortunately it's done by image name instead of image hash in practice. Upgrading dependencies, or otherwise resolving a name and version to a hash is a pre-source task, not a from-source task. Maybe it can be automated, and a bot can generate pull requests to bump the versions, but that happens as a proposed change to the source, not in a from-source task like build, test, publish, or deploy.

malkia 3y ago

.exe may belong in p4, .pdb's not! :)

jmisavage 3y ago

I agree in principle, but you also need the same machine too otherwise new OS and hardware might introduce issues too. I resurrected an old project that needed to convert LESS to CSS and the node version required couldn’t run on my machine. Upgrading it to a version that could introduced filesystem changes that broke the packages that it was looking for. Now just imagine businesses in the middle of a platform shift like Macs going from Intel to Apple’s own ARM chips. Eventually you’ll going to be missing something and all this work of bundling everything will end up being busy work.

spiffytech 3y ago

Is it going to be a problem for your Windows/Linux developers that someone committed a node_modules with macOS-only binaries inside?

simonw 3y ago

If you're concerned about bloating your Git repository with non-unique binary files (as you should be) a trick I've used in the past that worked really well was to have a separate project-dependencies Git repository that all of my dependencies lived in. I was working with Python, so this was effectively a Git-backed cache of the various .tar.gz and .whl files that my project depended on. This worked, and it gave me reassurance that I'd still be able to build my project if I couldn't download packages from PyPI... but to be honest these days I don't bother, because PyPI has proven itself robust enough that it doesn't feel worth the extra effort. I keep meaning to set myself up a simple S3 bucket somewhere with my PyPI dependencies mirrored there, just in case I ever need to deploy a fix in a hurry while PyPI is having an outage.

matrss 3y ago

Nix (https://nix.dev/ https://nix.dev/) can provide all of this, although in a smarter way than just through dumping everything in the VCS. Some projects use it already to provide a reproducible development environment and if done right a clean build is just a `nix-build` away.

delotrag 3y ago
Honestly surprised the article didn't mention Nix or Guix. Seems like functional package management solves the exact problems the author is worried about.
- FoodWThrow 3y ago
  The article starts with a gamedev disclaimer. Most gamedev folks would rather die on that Microsoft hill than use another OS.
HumanOstrich 3y ago
It does not provide "all of this" and would just make the experience worse for everyone.
- matrss 3y ago
  What is it missing? Using nix with nixpkgs you can pin all dependencies to a specific version identified by a hash value of that dependencies source code and dependencies, recursively building a dependency graph down to nixpkgs' bootstrap toolchains. That is deeper down than you could reasonably go by dumping everything in the VCS. Building becomes a single command that fetches everything it needs. Even the offline requirement is satisfiable if you prefetch all dependencies, it is just not happening by default (and for good reason, you might already have the toolchain from another project or some other dependencies are already on your system).

jbverschoor 3y ago

I think you forgot to include: toolchains \win arm \linux x86 \linux arm \macos x86 \macos arm What I do think is that dependencies should be versioned, and their artifacts should be immutable. Dependency management is not the only thing wrong with gamedev

TeMPOraL 3y ago
> What I do think is that dependencies should be versioned, and their artifacts should be immutable. And you should keep a copy of those artifacts, and build from it. Having a hash is not enough. People have this weird assumption that you can just fetch all your dependencies by their hashes from the cloud. But cloud is just other people's servers; no one is going to be hosting build artifacts indefinitely just for you.

dboreham 3y ago

This is both right and wrong: If you're shipping shrink-wrap product, or equivalent, then you should freeze everything in carbonite so you can later recreate the thing you released. The article is written as if this is a novel idea but it isn't. Decades ago when I worked on products shipped on CD it was standard procedure to archive the entire build machine to tape and put that in a fire safe. In fact I subsequently (decades later) worked for law firms on patent cases where they were able to get those backup tapes to prove something about the content of software shipped on a specific date. otoh for the typical present-day software project you don't want to re-create an identical build result as someone else got six months ago. For example if it's a JavaScript project then Node is going to be two versions out of date and probably full of security bugs from that time. So you actually want "code that behaves as expected but built with current dependencies and toolchain". Admittedly experience shows that for some languages this is an unreasonable expectation. Some level of ongoing maintenance to the codebase is often required just to keep it building with security-fixed dependencies.

cxr 3y ago
> otoh for the typical present-day software project you don't want to re-create an identical build result as someone else got six months ago According to whom? I absolutely want to be able to do this. > if it's a JavaScript project then Node is going to be two versions out of date and probably full of security bugs from that time Sounds like building on something as unstable as Node is a significant drawback. That's a Node problem. (I missed how we got to talking about Node, anyway—you said "JavaScript project". NodeJS does not have a monopoly on JS. It's not even a great example of a particularly good JS runtime; for the reasons you mention—instability—it's actually a pretty good example of a bad JS runtime.) Fortunately if you're doing JS, there's an incredibly stable set of interfaces available for you to program against, and the best part is you don't need to download anything extra because every computer already has a JS runtime installed.

jbverschoor 3y ago

Don't forget https://fossil-scm.org/ https://fossil-scm.org/

mr_tristan 3y ago

The challenge when I see posts like this is the people in charge of building this "check it all in" ecosystem usually forget about the developer experience and basically just implement a CI system. Cool, you can 're-run' an old build cleanly, which is good, but not enough. How about commercial IDEs? Cloud environments? A lot of developer environments these days include a ton of stuff that likely doesn't make sense to check in, usually licensing config is annoying, or because you're relying on runtime services. And all this time engineers spend on their own machines is basically time wasted, which isn't really a great solution to pitch to a business. Side note: I used to work for Perforce until the private equity sale. If there was a platform to vendor everything like this, it would be Perforce, because you could already do this kind of thing for years. AFAIK not many Perforce customers ever did this, and I don't think it was because Perforce wasn't capable. It's just a subtly wicked problem. Getting this right - just check out and go across different software development stacks - requires a lot of investment. It does look like Perforce has been acquiring many other aspects related to the application lifecycle, so in theory, they should be better positioned to be the "vendor everything on our stack" solution, but I'm not convinced this is going to work out well. Cloud development environment vendors seem to be the best positioned as a product for solving this problem, because there is less of that "go figure out your DX" aspect left to the customer. But the right CDE would have to have a lot of enterprise-style controls. This is so new that I'm not sure who will get it right first, but my guess is that we'll get to a more "development to delivery" integrated environment, and away from a hodgepodge of tools managed per project.

Joel_Mckay 3y ago

True in some situations, but a fundamentally flawed approach to FOSS. Indeed, if you are statically linking noncritical code, than for maintainability it is easier to version-lock an entire OS with the build tree in a VM. Thus, the same input spits out the same audited binary objects every time. In some situations it is unavoidable (see Microsoft or FPGA ecosystem). However, a shared object library ecosystem is arguably a key part of FOSS when it works properly on *nix. As it is fundamentally important to keep versioned interoperability with other projects to minimize library RAM footprints etc. Additionally, all projects have a shared interest in maintaining each others security, rather than trying to waste resources on every application that has legacy stripped static obfuscated vulnerable leaky objects. "Reuse [shared objects], Reduce [resource costs], and Recycle [to optimize]." Sounds like familiar advice... =) like how some git solutions don't handle large binary artifacts well or simply implode. Good luck, and maybe one can prove all those cautionary stories wrong. YMMV =)

hervem 3y ago

What about working over a way to build a dev. env. correctly? ansible, container, dev-container.. Git shouldn't be used for backing all dependencies, tooling, and OS.. And while we are at it, do we track git itself in the "toolchain"?

lamontcg 3y ago

You're better off just taking your build artifacts and shoving them into some artifact repo which is an already solved problem. If you want to ship all your runtime deps use an OCI container image, or something like flatpak which allows shipping bundled libraries. Those are the artifacts which should flow through your CI/CD pipeline (which may be as simple as just an automated test stage before manual deployment if you don't like those abbreviations I just used). Beyond that there's not really much utility to being able to rebuild exactly the same thing, because that's what you already shipped. The problem becomes pulling in new stuff and bumping the build, in which case you always have the cost of integration testing all the new code together. I do fully support Gemfile.lock/Cargo.lock/flake.lock/etc files being checked into git and the build process should be bumping those and building and shipping the artifacts off to a repo -- and that's good enough without actually checking all the blobs into git. And the reason why git is bad at LFS and binary blob / artifact support is likely that everyone has adopted this method and so making git better for the use case isn't a very high priority.

cxr 3y ago
> You're better off just taking your build artifacts and shoving them into some artifact repo Better off how? What does "better" refer to here? Size on disk? Data transfer? Time? CPU cycles? Brain cycles? Body health? Please be specific. > there's not really much utility to being able to rebuild exactly the same thing, because that's what you already shipped This strikes me as profoundly myopic (maybe even willfully). The unspoken benefit here is related to a general understanding that it's good to be able to establish a base case showing that you can build the same thing in the first place, so that when you make slight changes to any part of it, you end up with exactly the same thing as before plus your changes, rather than being the casualty of hundreds or thousands of other uncontrolled inputs that have nothing to do with what you changed. You're casting aspersions on and summarily dismissing the basic tenet of having a control[1]. > The problem becomes pulling in new stuff and bumping the build, in which case you always have the cost of integration testing all the new code together. Just to emphasize: you are correct to say that this is a cost you always have, whether or not you're checking the results of package installation into your repo. Not checking them in doesn't ipso facto cause any issues related to upgrades to go away, and checking them in doesn't ipso facto increase the number of integration work you have to do. There's nothing magical about pulling the dependencies in over the network at the extreme last moment versus reading them from some other place on your own disk. 1. <https://en.wikipedia.org/wiki/Scientific_control https://en.wikipedia.org/wiki/Scientific_control>