3 ms·

Automatically publishing your build artifacts

23 points by agateau 5y ago

dnsmichi 5y ago
Hi, interesting post, thanks. Sharing how you can do it with GitLab :) Start with defining artifacts in the job config, and download them directly in the MR during the review & QA process. https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html build-job: script: - ./build-dist-artifacts.sh artifacts: paths: - dist/* Engineers can use `filter pipelines` to find specific branches, tags, etc. they are looking for. Large binary files may consume lots of storage, and need regular cleanup. `expire_in` allows to control the cleanup in GitLab. For older builds, you can always retry the build job/pipeline, and generate artifacts on demand, i.e. when debugging a problem between older release versions. This is helpful for tarballs, also RPM/DEB packages, etc - anything which requires time and knowledge to build manually on a local development environment. With GitLab API access, you can integrate the job artifacts into more automated workflows or custom index websites of your choice, leaving the storage as SSoT in GitLab. https://docs.gitlab.com/ee/api/job_artifacts.html#download-a-single-artifact-file-from-specific-tag-or-branch https://docs.gitlab.com/ee/api/job_artifacts.html#download-a... The job artifacts can be put into a cloud object storage, like S3, too. https://docs.gitlab.com/ee/administration/job_artifacts.html#using-object-storage https://docs.gitlab.com/ee/administration/job_artifacts.html... Last but but not least: If you prefer building your own file index based on smaller sized artifacts, you could use GitLab Pages and follow this post: https://forum.gitlab.com/t/how-to-allow-directory-listing-on-gitlab-com-pages/20625/7 https://forum.gitlab.com/t/how-to-allow-directory-listing-on... to publish the artifacts and create an html index. I've done a similar approach to publish custom code coverage reports in CI/CD in a past workshop: https://gitlab.com/gitlab-de/workshops/ci-monitoring-webcast-2020/-/blob/main/.gitlab-ci.yml#L80 https://gitlab.com/gitlab-de/workshops/ci-monitoring-webcast... - can be handy for reviews and QA checks too.
techplex 5y ago
I often "publish" artifacts internally in github actions with the actions/upload-artifact action. https://github.com/actions/upload-artifact https://github.com/actions/upload-artifact //.github/workflows/main.yml name: Main on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: self-hosted steps: - uses: actions/checkout@v2 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.16 - name: Test run: go test -v ./... - name: Build Discover Command run: go build -o discover cmd/discover/main.go - name: Upload Build Artifacts uses: actions/upload-artifact@v2 with: name: discover path: discover For internal tooling we often publish tagged artifacts to releases on the repo using a workflow that is triggered when someone creates a release. The creation of the release makes a new tag and triggers the build. //.github/workflows/release.yml name: Release on: release: types: - published jobs: build: runs-on: self-hosted env: GOPRIVATE: "github.com/OUR-ORG-NAME/\*" NAME: deploy-${{ github.event.release.tag_name }}-${{ matrix.GOOS }}-${{ matrix.GOARCH }}${{ matrix.EXTENSION }} strategy: matrix: GOOS: [ windows, linux, darwin ] GOARCH: [ amd64, 386 ] exclude: # excludes 32bit on macOS - GOOS: darwin GOARCH: 386 include: # includes a new variable for windows builds - GOOS: windows EXTENSION: ".exe" steps: # Runs a single command using the runners shell - name: Print Info run: echo '${{ toJSON(github.event.release) }}' - uses: actions/checkout@v2 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.16 # From https://github.com/mvdan/github-actions-golang/blob/master/README.md - name: Configure git for private modules run: | git config --global \ url."https://${{ secrets.GHUSER }}:${{ secrets.GHTOKEN }}@github.com".insteadOf \ "https://github.com" - name: Build ${{ env.NAME }} Command run: | GOOS=${{ matrix.GOOS }} \ GOARCH=${{ matrix.GOARCH }} \ go build -o ${{ env.NAME }} cmd/main.go - name: Upload Release Asset - ${{ matrix.GOOS }} / ${{ matrix.GOARCH }} uses: actions/upload-release-asset@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: upload_url: ${{ github.event.release.upload_url }} asset_path: ./${{ env.NAME }} asset_name: ${{ env.NAME }} asset_content_type: application/octet-stream edit: code formatting
arwineap 5y ago
If you don't publish your build artifacts, what are you deploying?
- GauntletWizard 5y ago
  An incredible amount of companies have a "release" build pipeline; They don't simply release some sufficiently tested version of master, but cut a branch and muck with things until they get a "release". This is the source of many, many bugs in software that is shipped, and many many regressions as "release" branches are reused inappropriately.
  - deckard1 5y ago
    oof. I've been there. One day my manager asked me if I could cherry-pick a large amount of code from master into their Frankenstein "release" branch. I thought about it for a minute trying to figure out a polite way to say "you all are fucking insane, you know?" before settling on "okay, I can do that but if this breaks it's on you." I'm pretty sure if you have releases and they involve half a dozen "hot fixes" then you're doing CI/CD wrong. But what do I know. Everyone is still drinking that agile kool-aid.

23 points by agateau 5y ago

Hi, interesting post, thanks. Sharing how you can do it with GitLab :) Start with defining artifacts in the job config, and download them directly in the MR during the review & QA process. https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html build-job: script: - ./build-dist-artifacts.sh artifacts: paths: - dist/* Engineers can use `filter pipelines` to find specific branches, tags, etc. they are looking for. Large binary files may consume lots of storage, and need regular cleanup. `expire_in` allows to control the cleanup in GitLab. For older builds, you can always retry the build job/pipeline, and generate artifacts on demand, i.e. when debugging a problem between older release versions. This is helpful for tarballs, also RPM/DEB packages, etc - anything which requires time and knowledge to build manually on a local development environment. With GitLab API access, you can integrate the job artifacts into more automated workflows or custom index websites of your choice, leaving the storage as SSoT in GitLab. https://docs.gitlab.com/ee/api/job_artifacts.html#download-a-single-artifact-file-from-specific-tag-or-branch https://docs.gitlab.com/ee/api/job_artifacts.html#download-a... The job artifacts can be put into a cloud object storage, like S3, too. https://docs.gitlab.com/ee/administration/job_artifacts.html#using-object-storage https://docs.gitlab.com/ee/administration/job_artifacts.html... Last but but not least: If you prefer building your own file index based on smaller sized artifacts, you could use GitLab Pages and follow this post: https://forum.gitlab.com/t/how-to-allow-directory-listing-on-gitlab-com-pages/20625/7 https://forum.gitlab.com/t/how-to-allow-directory-listing-on... to publish the artifacts and create an html index. I've done a similar approach to publish custom code coverage reports in CI/CD in a past workshop: https://gitlab.com/gitlab-de/workshops/ci-monitoring-webcast-2020/-/blob/main/.gitlab-ci.yml#L80 https://gitlab.com/gitlab-de/workshops/ci-monitoring-webcast... - can be handy for reviews and QA checks too.

techplex 5y ago

I often "publish" artifacts internally in github actions with the actions/upload-artifact action. https://github.com/actions/upload-artifact https://github.com/actions/upload-artifact //.github/workflows/main.yml name: Main on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: self-hosted steps: - uses: actions/checkout@v2 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.16 - name: Test run: go test -v ./... - name: Build Discover Command run: go build -o discover cmd/discover/main.go - name: Upload Build Artifacts uses: actions/upload-artifact@v2 with: name: discover path: discover For internal tooling we often publish tagged artifacts to releases on the repo using a workflow that is triggered when someone creates a release. The creation of the release makes a new tag and triggers the build. //.github/workflows/release.yml name: Release on: release: types: - published jobs: build: runs-on: self-hosted env: GOPRIVATE: "github.com/OUR-ORG-NAME/\*" NAME: deploy-${{ github.event.release.tag_name }}-${{ matrix.GOOS }}-${{ matrix.GOARCH }}${{ matrix.EXTENSION }} strategy: matrix: GOOS: [ windows, linux, darwin ] GOARCH: [ amd64, 386 ] exclude: # excludes 32bit on macOS - GOOS: darwin GOARCH: 386 include: # includes a new variable for windows builds - GOOS: windows EXTENSION: ".exe" steps: # Runs a single command using the runners shell - name: Print Info run: echo '${{ toJSON(github.event.release) }}' - uses: actions/checkout@v2 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.16 # From https://github.com/mvdan/github-actions-golang/blob/master/README.md - name: Configure git for private modules run: | git config --global \ url."https://${{ secrets.GHUSER }}:${{ secrets.GHTOKEN }}@github.com".insteadOf \ "https://github.com" - name: Build ${{ env.NAME }} Command run: | GOOS=${{ matrix.GOOS }} \ GOARCH=${{ matrix.GOARCH }} \ go build -o ${{ env.NAME }} cmd/main.go - name: Upload Release Asset - ${{ matrix.GOOS }} / ${{ matrix.GOARCH }} uses: actions/upload-release-asset@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: upload_url: ${{ github.event.release.upload_url }} asset_path: ./${{ env.NAME }} asset_name: ${{ env.NAME }} asset_content_type: application/octet-stream edit: code formatting

arwineap 5y ago

If you don't publish your build artifacts, what are you deploying?

GauntletWizard 5y ago
An incredible amount of companies have a "release" build pipeline; They don't simply release some sufficiently tested version of master, but cut a branch and muck with things until they get a "release". This is the source of many, many bugs in software that is shipped, and many many regressions as "release" branches are reused inappropriately.
- deckard1 5y ago
  oof. I've been there. One day my manager asked me if I could cherry-pick a large amount of code from master into their Frankenstein "release" branch. I thought about it for a minute trying to figure out a polite way to say "you all are fucking insane, you know?" before settling on "okay, I can do that but if this breaks it's on you." I'm pretty sure if you have releases and they involve half a dozen "hot fixes" then you're doing CI/CD wrong. But what do I know. Everyone is still drinking that agile kool-aid.