3 ms·

A huge part of the problem is relying on the system. Project repos should include their dependencies. The Linux model of global shared libraries is, imho, bad a

A huge part of the problem is relying on the system. Project repos should include their dependencies. The Linux model of global shared libraries is, imho, bad and wrong. Building a project shouldn’t be complicated. There should be extremely minimal branching and if checks. My blog post where I argue this in more detail: https://www.forrestthewoods.com/blog/dependencies-belong-in-version-control/ https://www.forrestthewoods.com/blog/dependencies-belong-in-...

nickelpro 3y ago
vcpkg, conan, and Fetch_Content fix all the issues you discuss in that blog with the added benefit of not being insane. You dismiss package managers out of hand and you shouldn't.
- forrestthewoods 3y ago
  > not being insane Insane? It’s what all of BigTech and every game dev does. It’s tried and true. > vcpkg, conan, and Fetch_Content fix all the issues Not really, no.
- nalllar 3y ago
  conan makes it very hard to build a project offline and with a fixed set of inputs, it makes things worse in important ways if you care about reproducibility.
jonstewart 3y ago
Ah, the bad old days, when we checked DLLs into our cvs repos and every six months the greybeard on the team would bring a case of Mountain Dew into the office on a weekend and try upgrading the dependencies, sometimes successfully, before everyone else would arrive at work on Monday morning. Fun stuff.
marcus_holmes 3y ago
Ahh the eternal cycle of static vs dynamic linking swings around again. "I need common.lib, it gets linked into my application at compile-time" "why does every binary include this common.lib code? Let's pull that out to a shared module so we only have one version of it! See? that saved 35739475749Kb of storage! Everything is smaller and sleeker now!" "which version of common.lib does this application need? Hmm, but that other application needs an older version. I'll have to hack some way of having multiple versions of the common.lib available" "ahh, this application needs version 5.2.1 but that dependency needs version 4.11.6! Two incompatible versions in the same application! Who designed this thing?" "right, let's compile whatever versions of common.lib we need into our binary. It'll make life easier and to be honest there are so many versions of common.lib around that we might as well have a separate version for every application" "Ah, small problem, I have to compile a binary for my application for each variant of each platform because common.lib actually varies per platform... hmm, wonder if I can link to the system version of that lib at run-time..." <repeat until heat death of the universe>
- forrestthewoods 3y ago
  > Ahh the eternal cycle of static vs dynamic linking swings around again. Somewhat. Although it's agnostic to static vs dynamic linking. The philosophy goes a bit deeper than that. > See? that saved 35739475749Kb of storage! Storage capacity hasn't been relevant for over a decade. Besides, we foolishly replaced "use shared modules to save storage space" with "build multi-gigabyte docker images because it's the only way to reliably launch a simple program without crashing on startup".
  - yau8edq12i 3y ago
    > Storage capacity hasn't been relevant for over a decade. Bullshit. It's especially visible with games. I cannot install more than 3-4 big games on my 512GB drive because apparently all game developers believe that "storage capacity isn't relevant".
    - forrestthewoods 3y ago
      Let me rephrase, storage capacity with respond to dynamic/shared libraries hasn't been relevant for over a decade. AAA video games have massive storage requirements primarily due to textures, and sometimes audio. Which is a totally different conversation. It's not a bunch of redundant static libraries causing your pain. $100 will get you an 8Tb spinning drive or a 1Tb SSD or possibly 2TB. You can also get an external SSD for the same price. Large game sizes is a real and annoying issue. But even 4 year old consoles launched with a 1Tb SSD.
    - junon 3y ago
      That is almost all assets and has almost nothing to do with static vs shared. In fact, making things shared usually makes them worse, because you miss out on WPLTO.
    - bitwize 3y ago
      Storage is cheap. Buy a bigger drive.
  - czarit 3y ago
theamk 3y ago
Something does not compute... problem: > Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies and that will never happen. author's "ideal" solution: > First, the user clones a random GenAI repo. This is near instantaneous as files are not prefetched. The user then invokes the build script. As files are accessed they're downloaded. The very first build may download a few hundred megabytes of data. Seems like contradiction to me? Do you want your builds to be offline or online? I have a feeling that the real problem is that (1) author never worked with build systems which supply their own compiler, like buildroot or hermetic bazel and (2) author only downloaded dependencies from third-party servers. You can solve all the problems in the blog post by (1) changing build system to download compilers and libraries and (2) adopting some sort of "binary storage" (something as simple as private ftp server + small text file checked into repo with path and checksum + few scripts to upload and download). And it would not require any new major code investment, nor new tooling, nor new VCS. The key idea is that all dependencies should always come from your local binary server, never from the third-party server. This is a bit of PITA as many modern tools default to downloading stuff from third-party servers, and it could be hard to convince them to look at your local server instead. But once you do it, you get reliability + reproducibility + sustainability, without having to reinvent the world.
bhawks 3y ago
As a Xoogler one of the things I miss the most was srcfs/CitC(client in the cloud). It worked exactly as you were describing and also brought some very useful features including transparent read access to other users clients and time machine like capabilities. After having every single edit saved for you doing period git commits feels like such a chore.
- forrestthewoods 3y ago
  That post is derived from my experience in video games and at Meta. There is a better way, and it already exists in some places! I'm shocked at how resistant people are to the idea that current popular workflows aren't actually that good and some places already do it better.
NoGravitas 3y ago
I, too, enjoy rebuilding the world from source when there's a security issue found in a commonly-used dependency.
- forrestthewoods 3y ago
  You mean all your docker containers, right?

nickelpro 3y ago

vcpkg, conan, and Fetch_Content fix all the issues you discuss in that blog with the added benefit of not being insane. You dismiss package managers out of hand and you shouldn't.

forrestthewoods 3y ago
> not being insane Insane? It’s what all of BigTech and every game dev does. It’s tried and true. > vcpkg, conan, and Fetch_Content fix all the issues Not really, no.
nalllar 3y ago
conan makes it very hard to build a project offline and with a fixed set of inputs, it makes things worse in important ways if you care about reproducibility.

jonstewart 3y ago

Ah, the bad old days, when we checked DLLs into our cvs repos and every six months the greybeard on the team would bring a case of Mountain Dew into the office on a weekend and try upgrading the dependencies, sometimes successfully, before everyone else would arrive at work on Monday morning. Fun stuff.

marcus_holmes 3y ago

Ahh the eternal cycle of static vs dynamic linking swings around again. "I need common.lib, it gets linked into my application at compile-time" "why does every binary include this common.lib code? Let's pull that out to a shared module so we only have one version of it! See? that saved 35739475749Kb of storage! Everything is smaller and sleeker now!" "which version of common.lib does this application need? Hmm, but that other application needs an older version. I'll have to hack some way of having multiple versions of the common.lib available" "ahh, this application needs version 5.2.1 but that dependency needs version 4.11.6! Two incompatible versions in the same application! Who designed this thing?" "right, let's compile whatever versions of common.lib we need into our binary. It'll make life easier and to be honest there are so many versions of common.lib around that we might as well have a separate version for every application" "Ah, small problem, I have to compile a binary for my application for each variant of each platform because common.lib actually varies per platform... hmm, wonder if I can link to the system version of that lib at run-time..." <repeat until heat death of the universe>

forrestthewoods 3y ago
> Ahh the eternal cycle of static vs dynamic linking swings around again. Somewhat. Although it's agnostic to static vs dynamic linking. The philosophy goes a bit deeper than that. > See? that saved 35739475749Kb of storage! Storage capacity hasn't been relevant for over a decade. Besides, we foolishly replaced "use shared modules to save storage space" with "build multi-gigabyte docker images because it's the only way to reliably launch a simple program without crashing on startup".
- yau8edq12i 3y ago
  > Storage capacity hasn't been relevant for over a decade. Bullshit. It's especially visible with games. I cannot install more than 3-4 big games on my 512GB drive because apparently all game developers believe that "storage capacity isn't relevant".
  - forrestthewoods 3y ago
    Let me rephrase, storage capacity with respond to dynamic/shared libraries hasn't been relevant for over a decade. AAA video games have massive storage requirements primarily due to textures, and sometimes audio. Which is a totally different conversation. It's not a bunch of redundant static libraries causing your pain. $100 will get you an 8Tb spinning drive or a 1Tb SSD or possibly 2TB. You can also get an external SSD for the same price. Large game sizes is a real and annoying issue. But even 4 year old consoles launched with a 1Tb SSD.
  - junon 3y ago
    That is almost all assets and has almost nothing to do with static vs shared. In fact, making things shared usually makes them worse, because you miss out on WPLTO.
  - bitwize 3y ago
    Storage is cheap. Buy a bigger drive.
- czarit 3y ago

theamk 3y ago

Something does not compute... problem: > Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies and that will never happen. author's "ideal" solution: > First, the user clones a random GenAI repo. This is near instantaneous as files are not prefetched. The user then invokes the build script. As files are accessed they're downloaded. The very first build may download a few hundred megabytes of data. Seems like contradiction to me? Do you want your builds to be offline or online? I have a feeling that the real problem is that (1) author never worked with build systems which supply their own compiler, like buildroot or hermetic bazel and (2) author only downloaded dependencies from third-party servers. You can solve all the problems in the blog post by (1) changing build system to download compilers and libraries and (2) adopting some sort of "binary storage" (something as simple as private ftp server + small text file checked into repo with path and checksum + few scripts to upload and download). And it would not require any new major code investment, nor new tooling, nor new VCS. The key idea is that all dependencies should always come from your local binary server, never from the third-party server. This is a bit of PITA as many modern tools default to downloading stuff from third-party servers, and it could be hard to convince them to look at your local server instead. But once you do it, you get reliability + reproducibility + sustainability, without having to reinvent the world.

bhawks 3y ago

As a Xoogler one of the things I miss the most was srcfs/CitC(client in the cloud). It worked exactly as you were describing and also brought some very useful features including transparent read access to other users clients and time machine like capabilities. After having every single edit saved for you doing period git commits feels like such a chore.

forrestthewoods 3y ago
That post is derived from my experience in video games and at Meta. There is a better way, and it already exists in some places! I'm shocked at how resistant people are to the idea that current popular workflows aren't actually that good and some places already do it better.

NoGravitas 3y ago

I, too, enjoy rebuilding the world from source when there's a security issue found in a commonly-used dependency.

forrestthewoods 3y ago
You mean all your docker containers, right?