5 ms·

You're better off just taking your build artifacts and shoving them into some artifact repo which is an already solved problem. If you want to ship all your ru

You're better off just taking your build artifacts and shoving them into some artifact repo which is an already solved problem. If you want to ship all your runtime deps use an OCI container image, or something like flatpak which allows shipping bundled libraries. Those are the artifacts which should flow through your CI/CD pipeline (which may be as simple as just an automated test stage before manual deployment if you don't like those abbreviations I just used). Beyond that there's not really much utility to being able to rebuild exactly the same thing, because that's what you already shipped. The problem becomes pulling in new stuff and bumping the build, in which case you always have the cost of integration testing all the new code together. I do fully support Gemfile.lock/Cargo.lock/flake.lock/etc files being checked into git and the build process should be bumping those and building and shipping the artifacts off to a repo -- and that's good enough without actually checking all the blobs into git. And the reason why git is bad at LFS and binary blob / artifact support is likely that everyone has adopted this method and so making git better for the use case isn't a very high priority.

cxr 3y ago
> You're better off just taking your build artifacts and shoving them into some artifact repo Better off how? What does "better" refer to here? Size on disk? Data transfer? Time? CPU cycles? Brain cycles? Body health? Please be specific. > there's not really much utility to being able to rebuild exactly the same thing, because that's what you already shipped This strikes me as profoundly myopic (maybe even willfully). The unspoken benefit here is related to a general understanding that it's good to be able to establish a base case showing that you can build the same thing in the first place, so that when you make slight changes to any part of it, you end up with exactly the same thing as before plus your changes, rather than being the casualty of hundreds or thousands of other uncontrolled inputs that have nothing to do with what you changed. You're casting aspersions on and summarily dismissing the basic tenet of having a control[1]. > The problem becomes pulling in new stuff and bumping the build, in which case you always have the cost of integration testing all the new code together. Just to emphasize: you are correct to say that this is a cost you always have, whether or not you're checking the results of package installation into your repo. Not checking them in doesn't ipso facto cause any issues related to upgrades to go away, and checking them in doesn't ipso facto increase the number of integration work you have to do. There's nothing magical about pulling the dependencies in over the network at the extreme last moment versus reading them from some other place on your own disk. 1. <https://en.wikipedia.org/wiki/Scientific_control https://en.wikipedia.org/wiki/Scientific_control>
- lamontcg 3y ago
  > Better off how? What does "better" refer to here? Size on disk? Data transfer? Time? CPU cycles? Brain cycles? Body health? Please be specific. Getting shit done. Git doesn't support checking blobs into git very well, so you're pissing into the wind trying to do that because of philosophical opinions. It won't help you deliver your business objectives at the end of the day/week/month/year. > This strikes me as profoundly myopic (maybe even willfully). Based on a decade of experience, you do not need that incredibly strict level of control over reproducibility. The bigger problem is just that your dependencies will change with new version and you need to pull in those versions and deal with the tech debt of integration. Even if you sit down and perfectly and accurately solve the problem of exact reproducability of yesterday's builds, you will always have the problem of bumping versions and dealing with tomorrows builds. Dealing with the constant incoming flow of technical debt starts to dwarf any small problems caused by slightly inconsistent "rebuildability", while the cost of pursuing the perfect ability to rebuild starts to climb. What you're deaing with is a multi-objective optimization problem[1] where the cost function is the finite resource constraints available to you where the two problems cannot be solved 100% under those resource constraints and where overall velocity suffers if the costs become too great, so you are looking for something close to a Pareto optimal[2] tradeoff between the different objectives rather than strict perfection along any one axis. But yeah, keeping things pretty consistent is useful for dealing with tomorrow's builds as well, that is why I suggested all the different lockfile approaches. The optimization problem is nonlinear and up to a point solving yesterday's build problem does help tomorrow's build problem as well. > Just to emphasize: you are correct to say that this is a cost you always have, whether or not you're checking the results of package installation into your repo. Not checking them in doesn't ipso facto cause any issues related to upgrades to go away, and checking them in doesn't ipso facto increase the number of integration work you have to do. There's nothing magical about pulling the dependencies in over the network at the extreme last moment versus reading them from some other place on your own disk. The problem is inherently in the bumping of versions, and pulling in new code from upstream and dealing with emerging tech debt and integration problem. Once you have builds reasonably reproducible (lockfiles, etc) then just dealing with all the new code you're pulling in dwarfs the benefits of iterating on _perfect_ reproducibility of yesterday's builds. And in fact if you are checking your artifacts into git _and then not bumping them_ you are just kicking the can down the road and hiding your future tech debt. Yeah, you can try to freeze time and build on 8 year old versions of your build tools, but then one day you're going to have a business requirement that requires a newer toolchain (new O/S targets, new chipsets, whatever) and then you're stuck with solving 8 years of technical debt. > 1. <https://en.wikipedia.org/wiki/Scientific_control https://en.wikipedia.org/wiki/Scientific_control> Do you really think I'm dumb enough to need you to condescendingly cite this wikipedia page? [1] https://en.wikipedia.org/wiki/Multi-objective_optimization https://en.wikipedia.org/wiki/Multi-objective_optimization [2] https://en.wikipedia.org/wiki/Pareto_efficiency https://en.wikipedia.org/wiki/Pareto_efficiency

cxr 3y ago

> You're better off just taking your build artifacts and shoving them into some artifact repo Better off how? What does "better" refer to here? Size on disk? Data transfer? Time? CPU cycles? Brain cycles? Body health? Please be specific. > there's not really much utility to being able to rebuild exactly the same thing, because that's what you already shipped This strikes me as profoundly myopic (maybe even willfully). The unspoken benefit here is related to a general understanding that it's good to be able to establish a base case showing that you can build the same thing in the first place, so that when you make slight changes to any part of it, you end up with exactly the same thing as before plus your changes, rather than being the casualty of hundreds or thousands of other uncontrolled inputs that have nothing to do with what you changed. You're casting aspersions on and summarily dismissing the basic tenet of having a control[1]. > The problem becomes pulling in new stuff and bumping the build, in which case you always have the cost of integration testing all the new code together. Just to emphasize: you are correct to say that this is a cost you always have, whether or not you're checking the results of package installation into your repo. Not checking them in doesn't ipso facto cause any issues related to upgrades to go away, and checking them in doesn't ipso facto increase the number of integration work you have to do. There's nothing magical about pulling the dependencies in over the network at the extreme last moment versus reading them from some other place on your own disk. 1. <https://en.wikipedia.org/wiki/Scientific_control https://en.wikipedia.org/wiki/Scientific_control>

lamontcg 3y ago
> Better off how? What does "better" refer to here? Size on disk? Data transfer? Time? CPU cycles? Brain cycles? Body health? Please be specific. Getting shit done. Git doesn't support checking blobs into git very well, so you're pissing into the wind trying to do that because of philosophical opinions. It won't help you deliver your business objectives at the end of the day/week/month/year. > This strikes me as profoundly myopic (maybe even willfully). Based on a decade of experience, you do not need that incredibly strict level of control over reproducibility. The bigger problem is just that your dependencies will change with new version and you need to pull in those versions and deal with the tech debt of integration. Even if you sit down and perfectly and accurately solve the problem of exact reproducability of yesterday's builds, you will always have the problem of bumping versions and dealing with tomorrows builds. Dealing with the constant incoming flow of technical debt starts to dwarf any small problems caused by slightly inconsistent "rebuildability", while the cost of pursuing the perfect ability to rebuild starts to climb. What you're deaing with is a multi-objective optimization problem[1] where the cost function is the finite resource constraints available to you where the two problems cannot be solved 100% under those resource constraints and where overall velocity suffers if the costs become too great, so you are looking for something close to a Pareto optimal[2] tradeoff between the different objectives rather than strict perfection along any one axis. But yeah, keeping things pretty consistent is useful for dealing with tomorrow's builds as well, that is why I suggested all the different lockfile approaches. The optimization problem is nonlinear and up to a point solving yesterday's build problem does help tomorrow's build problem as well. > Just to emphasize: you are correct to say that this is a cost you always have, whether or not you're checking the results of package installation into your repo. Not checking them in doesn't ipso facto cause any issues related to upgrades to go away, and checking them in doesn't ipso facto increase the number of integration work you have to do. There's nothing magical about pulling the dependencies in over the network at the extreme last moment versus reading them from some other place on your own disk. The problem is inherently in the bumping of versions, and pulling in new code from upstream and dealing with emerging tech debt and integration problem. Once you have builds reasonably reproducible (lockfiles, etc) then just dealing with all the new code you're pulling in dwarfs the benefits of iterating on _perfect_ reproducibility of yesterday's builds. And in fact if you are checking your artifacts into git _and then not bumping them_ you are just kicking the can down the road and hiding your future tech debt. Yeah, you can try to freeze time and build on 8 year old versions of your build tools, but then one day you're going to have a business requirement that requires a newer toolchain (new O/S targets, new chipsets, whatever) and then you're stuck with solving 8 years of technical debt. > 1. <https://en.wikipedia.org/wiki/Scientific_control https://en.wikipedia.org/wiki/Scientific_control> Do you really think I'm dumb enough to need you to condescendingly cite this wikipedia page? [1] https://en.wikipedia.org/wiki/Multi-objective_optimization https://en.wikipedia.org/wiki/Multi-objective_optimization [2] https://en.wikipedia.org/wiki/Pareto_efficiency https://en.wikipedia.org/wiki/Pareto_efficiency