6 ms·

Great! This is my current build script: BUILD_TIME=$(date +"%Y%m%d.%H%M%S") CommitHash=$(git describe | cut -d- -f3) GoVersion=$(go version | cut -c 14- | cu

by hbbio 5y ago

Great! This is my current build script: BUILD_TIME=$(date +"%Y%m%d.%H%M%S") CommitHash=$(git describe | cut -d- -f3) GoVersion=$(go version | cut -c 14- | cut -d' ' -f1) GitTag=$(git describe | cut -d- -f1) FLAG="-X $TRG_PKG.BuildTime=$BUILD_TIME" FLAG="$FLAG -X $TRG_PKG.CommitHash=$CommitHash" FLAG="$FLAG -X $TRG_PKG.GoVersion=$GoVersion" FLAG="$FLAG -X $TRG_PKG.GitTag=$GitTag" go build -v -ldflags "$FLAG"

rollcat 5y ago
Please consider omitting build time from your artifacts. It makes build reproducibility a nightmare; people are coming up with frameworks to lie to build systems about the current time. https://reproducible-builds.org https://reproducible-builds.org
- ainar-g 5y ago
  Alternatively, just make it configurable: buildtime="${BUILD_TIME:-$( date -u +%FT%TZ%z )}" # Use $buildtime in your go build command. This will only set the variable buildtime if BUILD_TIME isn't set already. Making reproducing a build easy: just define BUILD_TIME to be equal to the one of the binary you want to reproduce.
  - Foxboron 5y ago
    Please read the link. Reproducible Builds has standardized "SOURCE_DATE_EPOCH" for exactly this usecase. Introducing other variable names i just going to cause more problems. https://reproducible-builds.org/docs/source-date-epoch/ https://reproducible-builds.org/docs/source-date-epoch/
    - ainar-g 5y ago
      My message was more about how you don't necessarily need to exclude the date information as opposed to exact variable names. But thanks for the info.
    - jatone 5y ago
      no its not. build tools can use whatever names they want for environment variables. sharing tends to cause more issues then having namespaced ones due to unintended side effects.
    - laumars 5y ago
      It’s a proposed convention. A specification. But it’s not a standard.
      - capableweb 5y ago
        Foxboron wrote "Reproducible Builds has standardized". Going to the website, they write "SOURCE_DATE_EPOCH is a standardised environment variable" and it links to https://reproducible-builds.org/specs/source-date-epoch/ https://reproducible-builds.org/specs/source-date-epoch/ which is a specification to standardize something. So it's accurate to say that "Reproducible Builds has standardized" that since they only talk about their own organization, not that it has been accepted to IETF or something like that.
        Foxboron 5y ago
        Man, if people just took the time to read the link and not the comment :)
        password4321 5y ago
        It usually takes less than 30 seconds to copy+paste quote the portion of the linked content that's actually relevant. Doing so greatly improves the effectiveness of communication.
        laumars 5y ago
        I actually did and the link just references a specification, not a standard. Something being “standardised” requires more than just a specification and just because the specification throws the term “standard” around it doesn’t mean it has been formally standardised. This doesn’t mean people shouldn’t still follow the specification. Plenty of specifications exist that have not been standardised. For example, there is an RFC specification for CSVs but not formal standard. So by all means push this specification as a de facto standard but please don’t mislead people saying it has been standardised when it has not.
        laumars 5y ago
        That’s a specification. Just because the specification says it’s standardised it doesn’t mean it is. Where’s the corresponding approval from ISO, ANSI or other standards authority? There’s nothing wrong with a specification or even a reference implementation but they are not themselves standards. And that’s exactly what this is: a widely respected specification. You shouldn’t have to lie about this being standardised for it still to be relevant.
  - thiht 5y ago
    I do that in all my Makefiles and it works great.
- wahern 5y ago
  For timestamps I like to use something like, TZ=UTC git log -n1 --format='%cd' --date='format-local:%Y-%m-%dT%H:%M:%S' There's probably a better way to do that, but the basic gist is to derive the most recent commit date.
- WJW 5y ago
  It seems that a proper reproducible build checker should omit this kind of metadata from the portions of an executable that it verifies to be the same.
  - verytrivial 5y ago
    Proper does not mean an endless list of exceptions to deal with the endless number of ways to exclude effectively random, non-functionally relevant information in the binaries. Because that where reproducible builds started. Projects not interested in reproducibility can/should content themselves to live in the non-reproducible set instead of expecting someone else to monkey patch for them!
  - smitop 5y ago
    It's hard to statically determine what parts of an executable affect the running of it. Many executable have timestamps in them that do affect how they run, so you can't just ignore all timestamps if you want reproducible builds. See https://reproducible-builds.org/docs/timestamps/ https://reproducible-builds.org/docs/timestamps/
  - akavel 5y ago
    Will 2 binaries differing by timestamp hash the same? do you suggest that file hashing algorithms skip date bytes in a binary? The KISS solution is to not vary date in the binary.
    - capableweb 5y ago
      Not sure why the timestamp of when the build happened should even be logged. Include the git repository, the sha and if on CI, the build number/hash and you're pretty much covered.
- matheusd 5y ago
  There are also cases where multiple VCS tools are used or where you might want to build from an archived full copy of source+vendored dependencies and might not even have an underlying VCS to speak of. Reproducible releases should probably _not_ have these commit metadata built in (e.g. imagine reproducible building this 5 years in the future where you've moved off git). Though this is quite useful in debug builds where you might be have installed a number of different builds in the wild and want to be able to quickly figure out the exact version of the code a given binary was generated from.
  - grumbel 5y ago
    I would include the commit metadata, otherwise you have no easy way to figure out where that source came from. The issue of out-of-VCS builds can be solved by putting the commit-hash into the directory/tarball name or a VERSION file, e.g. $PROJECTNAME-$(git describe --tags | sed "s/^v//"). When converting VCSs most converters will preserve the old hash in some form, e.g. by appending it to the commit message. Github downloads however remain a problem, as they just include the branch name, not the commit hash.
- arpanetus 5y ago
  can nixos/nix package manager be somehow related?
- jerf 5y ago
  Knowing where my build artifact came from and when is at least a good two orders of magnitude more important to me than my own binaries being reproducible. Part of the reason why "reproducibility" has taken so long to appear is precisely that its use cases take a long time and a lot of scale to appear. I approve of distros pursuing this goal, but haven't got a lot of use of it for the stuff I build, which is why I have a very similar build script. Please don't reply with a simple recitation of the putative benefits of reproducibility (I'm well aware) or some wild "what if" scenario relating to my own binaries that will occur once every few centuries on average. I have had multiple real problems in my life where not knowing the exact source checkout and time something was built has been a problem. I've only really once had a problem where I would have felt better if I could have reproduced something exactly, to be sure that the source and random unattributed binary were in fact the same, and even that ended up only as a "I would have felt better." (And now that I think about it, if said unattributed binary had just said what commit it was built out of that would have been fine too....) I am much more interested in solving the problems that actually manifest than problems that someone somewhere hypothesizes I may someday have.
  - xyzzyz 5y ago
    > I have had multiple real problems in my life where not knowing the exact source checkout and time something was built has been a problem. Once you know the exact source, what does it matter when the binary was built?
    - mentat 5y ago
      Yes, exactly. That OP needs to know the time is the problem being solved.
    - ashtonkem 5y ago
      Changes in the build pipeline won’t be encoded in the git sha, since usually those are stored separately. And if you pull in a non-reproducible library, time will matter a lot. This of course could be fixed with a Nix type system (or even Nix itself), where the inputs of everything are hashed. But that’s a much more radical change than what OP is doing.
- Steltek 5y ago
  Isn't this only a concern for OSS projects where reproducibility is a concern? At my proprietary coding job, reproducibility is a nonstarter anyway and build time is super important for debugging day-to-day (or even minute to minute) problems.
  - er4hn 5y ago
    It's a concern for private industry as well. The solarwinds and other recent supply chain attack involved injecting additional code into a build environment. Reproducible builds could have been done in multiple places as an auditing mechanism to catch this. Another place where reproducible builds is useful in industry is being able to reproduce debug symbols for something after it has shipped. Sure you can rely on always saving debug symbols in an artifact repo, but if those age out and you get a customer case... what ya gonna do?
AtNightWeCode 5y ago
Never include things that change on every build in the binaries. It is bad practice and you will regret it. You can let your CI/CD software keep track of it or write the info to a separated log file.

by hbbio 5y ago

rollcat 5y ago

Please consider omitting build time from your artifacts. It makes build reproducibility a nightmare; people are coming up with frameworks to lie to build systems about the current time. https://reproducible-builds.org https://reproducible-builds.org

ainar-g 5y ago
Alternatively, just make it configurable: buildtime="${BUILD_TIME:-$( date -u +%FT%TZ%z )}" # Use $buildtime in your go build command. This will only set the variable buildtime if BUILD_TIME isn't set already. Making reproducing a build easy: just define BUILD_TIME to be equal to the one of the binary you want to reproduce.
- Foxboron 5y ago
  Please read the link. Reproducible Builds has standardized "SOURCE_DATE_EPOCH" for exactly this usecase. Introducing other variable names i just going to cause more problems. https://reproducible-builds.org/docs/source-date-epoch/ https://reproducible-builds.org/docs/source-date-epoch/
  - ainar-g 5y ago
    My message was more about how you don't necessarily need to exclude the date information as opposed to exact variable names. But thanks for the info.
  - jatone 5y ago
    no its not. build tools can use whatever names they want for environment variables. sharing tends to cause more issues then having namespaced ones due to unintended side effects.
  - laumars 5y ago
    It’s a proposed convention. A specification. But it’s not a standard.
    - capableweb 5y ago
      Foxboron wrote "Reproducible Builds has standardized". Going to the website, they write "SOURCE_DATE_EPOCH is a standardised environment variable" and it links to https://reproducible-builds.org/specs/source-date-epoch/ https://reproducible-builds.org/specs/source-date-epoch/ which is a specification to standardize something. So it's accurate to say that "Reproducible Builds has standardized" that since they only talk about their own organization, not that it has been accepted to IETF or something like that.
      - Foxboron 5y ago
        Man, if people just took the time to read the link and not the comment :)
        password4321 5y ago
        It usually takes less than 30 seconds to copy+paste quote the portion of the linked content that's actually relevant. Doing so greatly improves the effectiveness of communication.
        laumars 5y ago
        I actually did and the link just references a specification, not a standard. Something being “standardised” requires more than just a specification and just because the specification throws the term “standard” around it doesn’t mean it has been formally standardised. This doesn’t mean people shouldn’t still follow the specification. Plenty of specifications exist that have not been standardised. For example, there is an RFC specification for CSVs but not formal standard. So by all means push this specification as a de facto standard but please don’t mislead people saying it has been standardised when it has not.
      - laumars 5y ago
        That’s a specification. Just because the specification says it’s standardised it doesn’t mean it is. Where’s the corresponding approval from ISO, ANSI or other standards authority? There’s nothing wrong with a specification or even a reference implementation but they are not themselves standards. And that’s exactly what this is: a widely respected specification. You shouldn’t have to lie about this being standardised for it still to be relevant.
- thiht 5y ago
  I do that in all my Makefiles and it works great.
wahern 5y ago
For timestamps I like to use something like, TZ=UTC git log -n1 --format='%cd' --date='format-local:%Y-%m-%dT%H:%M:%S' There's probably a better way to do that, but the basic gist is to derive the most recent commit date.
WJW 5y ago
It seems that a proper reproducible build checker should omit this kind of metadata from the portions of an executable that it verifies to be the same.
- verytrivial 5y ago
  Proper does not mean an endless list of exceptions to deal with the endless number of ways to exclude effectively random, non-functionally relevant information in the binaries. Because that where reproducible builds started. Projects not interested in reproducibility can/should content themselves to live in the non-reproducible set instead of expecting someone else to monkey patch for them!
- smitop 5y ago
  It's hard to statically determine what parts of an executable affect the running of it. Many executable have timestamps in them that do affect how they run, so you can't just ignore all timestamps if you want reproducible builds. See https://reproducible-builds.org/docs/timestamps/ https://reproducible-builds.org/docs/timestamps/
- akavel 5y ago
  Will 2 binaries differing by timestamp hash the same? do you suggest that file hashing algorithms skip date bytes in a binary? The KISS solution is to not vary date in the binary.
  - capableweb 5y ago
    Not sure why the timestamp of when the build happened should even be logged. Include the git repository, the sha and if on CI, the build number/hash and you're pretty much covered.
matheusd 5y ago
There are also cases where multiple VCS tools are used or where you might want to build from an archived full copy of source+vendored dependencies and might not even have an underlying VCS to speak of. Reproducible releases should probably _not_ have these commit metadata built in (e.g. imagine reproducible building this 5 years in the future where you've moved off git). Though this is quite useful in debug builds where you might be have installed a number of different builds in the wild and want to be able to quickly figure out the exact version of the code a given binary was generated from.
- grumbel 5y ago
  I would include the commit metadata, otherwise you have no easy way to figure out where that source came from. The issue of out-of-VCS builds can be solved by putting the commit-hash into the directory/tarball name or a VERSION file, e.g. $PROJECTNAME-$(git describe --tags | sed "s/^v//"). When converting VCSs most converters will preserve the old hash in some form, e.g. by appending it to the commit message. Github downloads however remain a problem, as they just include the branch name, not the commit hash.
arpanetus 5y ago
can nixos/nix package manager be somehow related?
jerf 5y ago
Knowing where my build artifact came from and when is at least a good two orders of magnitude more important to me than my own binaries being reproducible. Part of the reason why "reproducibility" has taken so long to appear is precisely that its use cases take a long time and a lot of scale to appear. I approve of distros pursuing this goal, but haven't got a lot of use of it for the stuff I build, which is why I have a very similar build script. Please don't reply with a simple recitation of the putative benefits of reproducibility (I'm well aware) or some wild "what if" scenario relating to my own binaries that will occur once every few centuries on average. I have had multiple real problems in my life where not knowing the exact source checkout and time something was built has been a problem. I've only really once had a problem where I would have felt better if I could have reproduced something exactly, to be sure that the source and random unattributed binary were in fact the same, and even that ended up only as a "I would have felt better." (And now that I think about it, if said unattributed binary had just said what commit it was built out of that would have been fine too....) I am much more interested in solving the problems that actually manifest than problems that someone somewhere hypothesizes I may someday have.
- xyzzyz 5y ago
  > I have had multiple real problems in my life where not knowing the exact source checkout and time something was built has been a problem. Once you know the exact source, what does it matter when the binary was built?
  - mentat 5y ago
    Yes, exactly. That OP needs to know the time is the problem being solved.
  - ashtonkem 5y ago
    Changes in the build pipeline won’t be encoded in the git sha, since usually those are stored separately. And if you pull in a non-reproducible library, time will matter a lot. This of course could be fixed with a Nix type system (or even Nix itself), where the inputs of everything are hashed. But that’s a much more radical change than what OP is doing.
Steltek 5y ago
Isn't this only a concern for OSS projects where reproducibility is a concern? At my proprietary coding job, reproducibility is a nonstarter anyway and build time is super important for debugging day-to-day (or even minute to minute) problems.
- er4hn 5y ago
  It's a concern for private industry as well. The solarwinds and other recent supply chain attack involved injecting additional code into a build environment. Reproducible builds could have been done in multiple places as an auditing mechanism to catch this. Another place where reproducible builds is useful in industry is being able to reproduce debug symbols for something after it has shipped. Sure you can rely on always saving debug symbols in an artifact repo, but if those age out and you get a customer case... what ya gonna do?

AtNightWeCode 5y ago

Never include things that change on every build in the binaries. It is bad practice and you will regret it. You can let your CI/CD software keep track of it or write the info to a separated log file.