4 ms·

>“My background is C++ gamedev” This is why you think this way. Your proposal is not new. The issue is not in the fact that you don’t have your dependencies, i

>“My background is C++ gamedev” This is why you think this way. Your proposal is not new. The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Every other language has had package managers for this very reason. Where do you draw the line? OS libs and package manager is ok but it’s not for a developer? Go learn vcpkg and come back to us when you learn why everyone does it this way.

codetrotter 3y ago
> The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Exactly. With Rust you can commit your Cargo.lock file and you will then be able to rebuild your project with the exact same version of all your deps in the future. No need to commit the deps themselves.
- kibwen 3y ago
  And cargo-vendor is available out of the box as well (although it doesn't go quite so far as to add the entire Rust compiler toolchain to your repo): https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
  - reactordev 3y ago
    Exactly, In case you need to transport your dependencies and build air-gapped (happens) or you need to make some modifications (keep them as patches please, to be re-applied). Not so you can check them all into your repo. You should .gitignore your vendor folder.
- HumanOstrich 3y ago
  Of course, just rewrite everything in Rust! It's so easy!
  - reactordev 3y ago
    That's not what was said. What was said was "This is how rust does it, take note".
TillE 3y ago
Yeah vcpkg.json is the appropriate solution. That plus one git commit hash pins and verifies everything. Make port overlays if you need em. And you really shouldn't get married to one particular version of a compiler toolchain unless you know for sure that something is going to break if you update. That just leads to a lot of annoyed programmers stuck using ancient tools for no reason.
KRAKRISMOTT 3y ago
> Go learn vcpkg and come back to us when you learn why everyone does it this way. IMO it's mostly because of a culture of laziness among system programmers. They like to kick the ball down the road and make dependency management the distro's problem. Too many embedded engineers not being paid enough. When you are building a product you shouldn't be wasting your time on this sort of "make work" that reinvents the wheel multiple times. Come to the machine learning world and everything including the kitchen sink and graphic drivers are bundled in the build because otherwise your deployment won't work and you can't afford to pay engineers that cost half a mil a year to spend their time fiddling with deployment binaries.
- moomoo3000 3y ago
  Do you have any links you can recommend for how this is done in machine learning projects?
  - KRAKRISMOTT 3y ago
    The short version is to adopt modern tooling (the vcpkg suggestion is an excellent one) and dependency management rather than using OS specific tools (unless you are on Windows). Part of the reason for this mess is because the Unix world operates on an evergreen philosophy and nothing is truly backwards compatible out of the box without manual intervention. The modern web development and machine learning world runs on the opposite doctrine that programmer time is the most expensive commodity above all else; bandwidth is cheap, storage has a negligible cost, and horizontal scaling can sometimes fix compute bound problems. Deployment processes are thus optimized for reliably reproducible builds. Docker is the classic example: bundle literally every dependency possible just to ensure that the build always succeeds, anywhere, anytime. It has its downsides but it is still one of the most widely used deployment methods for a reason. In the Windows world, you often find desktops with ten different copies of the "Windows C++/.Net redistributable" (the windows version of the C++/CLR standard library dynamically loaded artefacts) installed because each individual app have their own specific dependencies and it's better for them to bundle/specify it rather than rely on the OS to figure out what to load. The JavaScript, Julia, Rust, Go ecosystems all have first party support for pulling in driver binaries that may be hundred of gigabytes in size (because Nvidia is about as cooperative as a 3 year old child). You don't waste time fiddling with autotools and ./configure and praying that everything would run. Just run `npm install` and most if not all of the popular dependency heavy libraries would work out of the box.
hoten 3y ago
vcpkg may expire assets after 1.5 years, so to achieve long-term reproducibility you will need to cache your dependencies.... Somewhere. Not sure what the expected solution is. https://github.com/microsoft/vcpkg/pull/30546#issuecomment-1820486542 https://github.com/microsoft/vcpkg/pull/30546#issuecomment-1...
- KRAKRISMOTT 3y ago
  Microsoft needs to fund Vcpkg more, the developer experience (especially installation) still has room for improvement.
  - reactordev 3y ago
    You see the sentiment from C++ folks. "Why do I need a package manager?". So until they realize they've been hamstrung for years, it will continue to rot on the vine.
groestl 3y ago
> Every other language has had package managers for this very reason Nah. Package managers are nice, but they only solve Problem 1 (Usability). If you have any business continuity concerns, you'll at least cache and archive the dependencies, and your package management will effectively become a binary blob extension of your VCS.
forrestthewoods 3y ago
OP here. > Your proposal is not new. Good thing I never claimed it was? I merely claimed it was "the right thing". I'm exceedingly familiar with vcpkg. And many other package managers in many other languages. Some good (Rust Cargo) and some bad (Python pip/Conda). Have you ever worked in a repo that committed all of its dependencies? I have. It's quite nice! Maybe you should have an open mind and come back to us when you've tried the other side.
- reactordev 3y ago
  I don't want to download 3TB of dependencies from git. What is your gripe with vcpkg? What short-comings does it have? Do you need to keep archives of your dependencies? That's an IT problem. I simply don't agree that it's "The Right Thing". The only thing that is right is making sure you can build. Whether that's pulling dependencies from the internet. Pulling dependencies from your OS. Or Pulling dependencies off a USB stick authorized by the government for use in the clean room you must build and install the software into? I don't want a world where 1 CVE requires 100,000 commits to all the repo's of the world. I want one where 1 CVE requires 1 commit to 1 repo and everyone else get's their share. Bifurcating your software out of the gate is not a sound software development strategy. It's just laziness. If you need to build on 10 year old hardware, you can checkout 10 year old Linux. You can build for 10 year old windows. You can build for 10 year old Mac (time limiting...). You can build Win 3.1 apps today with vcpkg. Storage is cheap. Maybe not so much on Github, but Github isn't the only place you can run git. I've worked at organizations that archived and kept every build, every dependency, every configuration, for every version, since the inception of the software project. Nowhere in the repo is all of that information.
  - deleted 3y ago
    [deleted]
  - forrestthewoods 3y ago
    > I don't want to download 3TB of dependencies from git. Oh you didn’t read the post. Got it.
cxr 3y ago
> Go learn vcpkg and come back to us when you learn why everyone does it this way. What reason do you have in mind? Can you articulate it in a way that can be quantified/falsified? The last time I pressed someone on HN about this[1], it quickly devolved into vague handwaving[2] of the same sort—that we all already know why so we don't have to talk about it. (Asking because I thought I knew, and then one day I realized I couldn't actually explain it, at least not in a way that I could be confident that everyone else would actually agree with. It feels a lot like the hand of Ra[3] is at play.) I'm curious to hear from anyone who thinks they know why or can perhaps point to where they've already explained it. I'm particularly interested in responses from people who deal with NPM. (What is the "NPM hypothesis", i.e. the value proposition for why having a skeleton source tree in Git that delegates the bulk of the work to `npm install` is objectively better—or subjectively better, even? Can you state it in the form of a falsifiable claim? If you're able to state a testable hypothesis and you have tested it, can you show your work?) 1. <https://news.ycombinator.com/item?id=37605966 https://news.ycombinator.com/item?id=37605966> 2. <https://news.ycombinator.com/item?id=37607202 https://news.ycombinator.com/item?id=37607202> 3. <https://srconstantin.wordpress.com/2016/10/20/ra/ https://srconstantin.wordpress.com/2016/10/20/ra/>
- pinkgolem 3y ago
  In case you are serious, and I am not super sure about that. - depencys include blobs, or other blob like things(big code bases, think react) - there is little value in seeing changes in a blob or blob like things as changes in those have a bad signal to noise ratio. Version information (minor/major) + changelog is much more interesting for the average dev - there is a performance penalty for having to many things/big things in git - there is no nice way to keep dependencies up to date in git - the discoverability of git/GitHub has not been great You can use git as a package provider, see pip for example, but using git alone is painfull.
  - forrestthewoods 3y ago
    This is an argument that Git has serious limitation (it does). It's not an argument against the theory of dependencies in VCS.

codetrotter 3y ago

> The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Exactly. With Rust you can commit your Cargo.lock file and you will then be able to rebuild your project with the exact same version of all your deps in the future. No need to commit the deps themselves.

kibwen 3y ago
And cargo-vendor is available out of the box as well (although it doesn't go quite so far as to add the entire Rust compiler toolchain to your repo): https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
- reactordev 3y ago
  Exactly, In case you need to transport your dependencies and build air-gapped (happens) or you need to make some modifications (keep them as patches please, to be re-applied). Not so you can check them all into your repo. You should .gitignore your vendor folder.
HumanOstrich 3y ago
Of course, just rewrite everything in Rust! It's so easy!
- reactordev 3y ago
  That's not what was said. What was said was "This is how rust does it, take note".

TillE 3y ago

Yeah vcpkg.json is the appropriate solution. That plus one git commit hash pins and verifies everything. Make port overlays if you need em. And you really shouldn't get married to one particular version of a compiler toolchain unless you know for sure that something is going to break if you update. That just leads to a lot of annoyed programmers stuck using ancient tools for no reason.

KRAKRISMOTT 3y ago

> Go learn vcpkg and come back to us when you learn why everyone does it this way. IMO it's mostly because of a culture of laziness among system programmers. They like to kick the ball down the road and make dependency management the distro's problem. Too many embedded engineers not being paid enough. When you are building a product you shouldn't be wasting your time on this sort of "make work" that reinvents the wheel multiple times. Come to the machine learning world and everything including the kitchen sink and graphic drivers are bundled in the build because otherwise your deployment won't work and you can't afford to pay engineers that cost half a mil a year to spend their time fiddling with deployment binaries.

moomoo3000 3y ago
Do you have any links you can recommend for how this is done in machine learning projects?
- KRAKRISMOTT 3y ago
  The short version is to adopt modern tooling (the vcpkg suggestion is an excellent one) and dependency management rather than using OS specific tools (unless you are on Windows). Part of the reason for this mess is because the Unix world operates on an evergreen philosophy and nothing is truly backwards compatible out of the box without manual intervention. The modern web development and machine learning world runs on the opposite doctrine that programmer time is the most expensive commodity above all else; bandwidth is cheap, storage has a negligible cost, and horizontal scaling can sometimes fix compute bound problems. Deployment processes are thus optimized for reliably reproducible builds. Docker is the classic example: bundle literally every dependency possible just to ensure that the build always succeeds, anywhere, anytime. It has its downsides but it is still one of the most widely used deployment methods for a reason. In the Windows world, you often find desktops with ten different copies of the "Windows C++/.Net redistributable" (the windows version of the C++/CLR standard library dynamically loaded artefacts) installed because each individual app have their own specific dependencies and it's better for them to bundle/specify it rather than rely on the OS to figure out what to load. The JavaScript, Julia, Rust, Go ecosystems all have first party support for pulling in driver binaries that may be hundred of gigabytes in size (because Nvidia is about as cooperative as a 3 year old child). You don't waste time fiddling with autotools and ./configure and praying that everything would run. Just run `npm install` and most if not all of the popular dependency heavy libraries would work out of the box.

hoten 3y ago

vcpkg may expire assets after 1.5 years, so to achieve long-term reproducibility you will need to cache your dependencies.... Somewhere. Not sure what the expected solution is. https://github.com/microsoft/vcpkg/pull/30546#issuecomment-1820486542 https://github.com/microsoft/vcpkg/pull/30546#issuecomment-1...

KRAKRISMOTT 3y ago
Microsoft needs to fund Vcpkg more, the developer experience (especially installation) still has room for improvement.
- reactordev 3y ago
  You see the sentiment from C++ folks. "Why do I need a package manager?". So until they realize they've been hamstrung for years, it will continue to rot on the vine.

groestl 3y ago

> Every other language has had package managers for this very reason Nah. Package managers are nice, but they only solve Problem 1 (Usability). If you have any business continuity concerns, you'll at least cache and archive the dependencies, and your package management will effectively become a binary blob extension of your VCS.

forrestthewoods 3y ago

OP here. > Your proposal is not new. Good thing I never claimed it was? I merely claimed it was "the right thing". I'm exceedingly familiar with vcpkg. And many other package managers in many other languages. Some good (Rust Cargo) and some bad (Python pip/Conda). Have you ever worked in a repo that committed all of its dependencies? I have. It's quite nice! Maybe you should have an open mind and come back to us when you've tried the other side.

reactordev 3y ago
I don't want to download 3TB of dependencies from git. What is your gripe with vcpkg? What short-comings does it have? Do you need to keep archives of your dependencies? That's an IT problem. I simply don't agree that it's "The Right Thing". The only thing that is right is making sure you can build. Whether that's pulling dependencies from the internet. Pulling dependencies from your OS. Or Pulling dependencies off a USB stick authorized by the government for use in the clean room you must build and install the software into? I don't want a world where 1 CVE requires 100,000 commits to all the repo's of the world. I want one where 1 CVE requires 1 commit to 1 repo and everyone else get's their share. Bifurcating your software out of the gate is not a sound software development strategy. It's just laziness. If you need to build on 10 year old hardware, you can checkout 10 year old Linux. You can build for 10 year old windows. You can build for 10 year old Mac (time limiting...). You can build Win 3.1 apps today with vcpkg. Storage is cheap. Maybe not so much on Github, but Github isn't the only place you can run git. I've worked at organizations that archived and kept every build, every dependency, every configuration, for every version, since the inception of the software project. Nowhere in the repo is all of that information.
- deleted 3y ago
  [deleted]
- forrestthewoods 3y ago
  > I don't want to download 3TB of dependencies from git. Oh you didn’t read the post. Got it.

cxr 3y ago

> Go learn vcpkg and come back to us when you learn why everyone does it this way. What reason do you have in mind? Can you articulate it in a way that can be quantified/falsified? The last time I pressed someone on HN about this[1], it quickly devolved into vague handwaving[2] of the same sort—that we all already know why so we don't have to talk about it. (Asking because I thought I knew, and then one day I realized I couldn't actually explain it, at least not in a way that I could be confident that everyone else would actually agree with. It feels a lot like the hand of Ra[3] is at play.) I'm curious to hear from anyone who thinks they know why or can perhaps point to where they've already explained it. I'm particularly interested in responses from people who deal with NPM. (What is the "NPM hypothesis", i.e. the value proposition for why having a skeleton source tree in Git that delegates the bulk of the work to `npm install` is objectively better—or subjectively better, even? Can you state it in the form of a falsifiable claim? If you're able to state a testable hypothesis and you have tested it, can you show your work?) 1. <https://news.ycombinator.com/item?id=37605966 https://news.ycombinator.com/item?id=37605966> 2. <https://news.ycombinator.com/item?id=37607202 https://news.ycombinator.com/item?id=37607202> 3. <https://srconstantin.wordpress.com/2016/10/20/ra/ https://srconstantin.wordpress.com/2016/10/20/ra/>

pinkgolem 3y ago
In case you are serious, and I am not super sure about that. - depencys include blobs, or other blob like things(big code bases, think react) - there is little value in seeing changes in a blob or blob like things as changes in those have a bad signal to noise ratio. Version information (minor/major) + changelog is much more interesting for the average dev - there is a performance penalty for having to many things/big things in git - there is no nice way to keep dependencies up to date in git - the discoverability of git/GitHub has not been great You can use git as a package provider, see pip for example, but using git alone is painfull.
- forrestthewoods 3y ago
  This is an argument that Git has serious limitation (it does). It's not an argument against the theory of dependencies in VCS.