4 ms·

I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've activel

by scubbo 3y ago

I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)

mdaniel 3y ago
My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?
- shepherdjerred 3y ago
  Amazon internal tools for building codes are _amazing_. Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number. Everything from the CI/CD to the code review tool to your local builds use the same build configuration with Brazil. All software packages in Brazil are built from source on Amazon's gigantic fleet of build servers. Builds are cached, so even though Amazon builds its own version of Make, Java, etc., these are all built and cached by the build servers and downloaded. A simple Java application at Amazon might have hundreds of dependencies (because you'll need to build Java from scratch), but since this is all cached you don't have to wait very long. Lastly, you have Pipelines which is their internal CI/CD tool which integrates naturally with Brazil + the build fleet. It can deploy to their internal fleet with Apollo, or to AWS Lambda, S3 buckets, etc. In all, everything is just very well integrated. I haven't seen anything come close to what you get internally at Amazon.
  - mdaniel 3y ago
    so what I'm hearing is that app-1.0 needs app-1.0-runtime-build-20240410 which was, itself, built from a base of runtime-y-2.0 and layering library-x-1.11 upon it, kind of like # in some "app-runtimes" project, they assemble your app's runtime cat > Dockerfile <<FOO FROM public.ecr.aws/runtimes/runtime-y:2.0 ADD https://cache.example/library-x/1.1/library-x-1.1.jar FOO tar -cf - Dockerfile | podman build -t public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 - # then you consume it in your project cat > Dockerfile <<FOO FROM public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 ADD ./app-1.0.jar FOO cat > .gitlab-ci.yml <<'YML' # you can also distribute artifacts other than just docker images # https://docs.gitlab.com/ee/user/packages/package_registry/supported_package_managers.html cook image: stage: package script: # or this https://docs.gitlab.com/ee/topics/autodevops/customize.html#customize-buildpacks-with-cloud-native-buildpacks - podman build -t $CI_REGISTRY_IMAGE . # https://docs.gitlab.com/ee/user/packages/#container-registry is built in - podman push $CI_REGISTRY_IMAGE review env: stage: staging script: auto-devops deploy # for free: https://docs.gitlab.com/ee/ci/review_apps/index.html environment: name: review/${CI_COMMIT_REF_SLUG} url: https://${CI_ENVIRONMENT_SLUG}.int.example on_stop: teardown-review teardown-review: stage: staging script: auto-devops stop when: manual environment: name: review/${CI_COMMIT_REF_SLUG} action: stop ... etc ... YML and then, yadda, yadda, blue-green, incremental rollout <https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml#L81-84 https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/...>, feature flags <https://docs.gitlab.com/ee/operations/feature_flags.html https://docs.gitlab.com/ee/operations/feature_flags.html>, error capture <https://docs.gitlab.com/ee/operations/error_tracking.html#integrated-error-tracking https://docs.gitlab.com/ee/operations/error_tracking.html#in...>, project-managed provisioning <https://docs.gitlab.com/ee/user/infrastructure/iac/#integrate-your-project-with-terraform https://docs.gitlab.com/ee/user/infrastructure/iac/#integrat...>, on call management <https://docs.gitlab.com/ee/operations/incident_management/ https://docs.gitlab.com/ee/operations/incident_management/>, on call runbooks <https://docs.gitlab.com/ee/user/project/clusters/runbooks/index.html https://docs.gitlab.com/ee/user/project/clusters/runbooks/in...> you can orchestrate all that from ~~Slack~~ Chime :-D if you're into that kind of thing https://docs.gitlab.com/ee/ci/chatops/ https://docs.gitlab.com/ee/ci/chatops/
sdesol 3y ago
> (I would be _delighted_ for someone to "Umm actually" me by providing a great product!) I think the issue is, nobody would be willing to pay for a good solution since they usually come with a steep maintenance cost. I wouldn't be surprised if the in-house CD team at Amazon were putting out fires every week/month behind the scene.
xyzzy_plugh 3y ago
NixOS/nixpkgs is about the closest thing you'll find in the wild. You have to squint a bit, I'll admit.
tripdout 3y ago
Is it true that teams don't do branches in source control at all? Just publishing a CR?
- scubbo 3y ago
  Yep! Well, a branch is created under-the-hood, and you can still check it out locally if you want to - but the standard code contribution workflow doesn't require you to think or work in those terms at all.
zaphirplane 3y ago
AWS has services around pipeline and deploy, right ?

by scubbo 3y ago

mdaniel 3y ago

My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?

shepherdjerred 3y ago
Amazon internal tools for building codes are _amazing_. Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number. Everything from the CI/CD to the code review tool to your local builds use the same build configuration with Brazil. All software packages in Brazil are built from source on Amazon's gigantic fleet of build servers. Builds are cached, so even though Amazon builds its own version of Make, Java, etc., these are all built and cached by the build servers and downloaded. A simple Java application at Amazon might have hundreds of dependencies (because you'll need to build Java from scratch), but since this is all cached you don't have to wait very long. Lastly, you have Pipelines which is their internal CI/CD tool which integrates naturally with Brazil + the build fleet. It can deploy to their internal fleet with Apollo, or to AWS Lambda, S3 buckets, etc. In all, everything is just very well integrated. I haven't seen anything come close to what you get internally at Amazon.
- mdaniel 3y ago
  so what I'm hearing is that app-1.0 needs app-1.0-runtime-build-20240410 which was, itself, built from a base of runtime-y-2.0 and layering library-x-1.11 upon it, kind of like # in some "app-runtimes" project, they assemble your app's runtime cat > Dockerfile <<FOO FROM public.ecr.aws/runtimes/runtime-y:2.0 ADD https://cache.example/library-x/1.1/library-x-1.1.jar FOO tar -cf - Dockerfile | podman build -t public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 - # then you consume it in your project cat > Dockerfile <<FOO FROM public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 ADD ./app-1.0.jar FOO cat > .gitlab-ci.yml <<'YML' # you can also distribute artifacts other than just docker images # https://docs.gitlab.com/ee/user/packages/package_registry/supported_package_managers.html cook image: stage: package script: # or this https://docs.gitlab.com/ee/topics/autodevops/customize.html#customize-buildpacks-with-cloud-native-buildpacks - podman build -t $CI_REGISTRY_IMAGE . # https://docs.gitlab.com/ee/user/packages/#container-registry is built in - podman push $CI_REGISTRY_IMAGE review env: stage: staging script: auto-devops deploy # for free: https://docs.gitlab.com/ee/ci/review_apps/index.html environment: name: review/${CI_COMMIT_REF_SLUG} url: https://${CI_ENVIRONMENT_SLUG}.int.example on_stop: teardown-review teardown-review: stage: staging script: auto-devops stop when: manual environment: name: review/${CI_COMMIT_REF_SLUG} action: stop ... etc ... YML and then, yadda, yadda, blue-green, incremental rollout <https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml#L81-84 https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/...>, feature flags <https://docs.gitlab.com/ee/operations/feature_flags.html https://docs.gitlab.com/ee/operations/feature_flags.html>, error capture <https://docs.gitlab.com/ee/operations/error_tracking.html#integrated-error-tracking https://docs.gitlab.com/ee/operations/error_tracking.html#in...>, project-managed provisioning <https://docs.gitlab.com/ee/user/infrastructure/iac/#integrate-your-project-with-terraform https://docs.gitlab.com/ee/user/infrastructure/iac/#integrat...>, on call management <https://docs.gitlab.com/ee/operations/incident_management/ https://docs.gitlab.com/ee/operations/incident_management/>, on call runbooks <https://docs.gitlab.com/ee/user/project/clusters/runbooks/index.html https://docs.gitlab.com/ee/user/project/clusters/runbooks/in...> you can orchestrate all that from ~~Slack~~ Chime :-D if you're into that kind of thing https://docs.gitlab.com/ee/ci/chatops/ https://docs.gitlab.com/ee/ci/chatops/

sdesol 3y ago

> (I would be _delighted_ for someone to "Umm actually" me by providing a great product!) I think the issue is, nobody would be willing to pay for a good solution since they usually come with a steep maintenance cost. I wouldn't be surprised if the in-house CD team at Amazon were putting out fires every week/month behind the scene.

xyzzy_plugh 3y ago

NixOS/nixpkgs is about the closest thing you'll find in the wild. You have to squint a bit, I'll admit.

tripdout 3y ago

Is it true that teams don't do branches in source control at all? Just publishing a CR?

scubbo 3y ago
Yep! Well, a branch is created under-the-hood, and you can still check it out locally if you want to - but the standard code contribution workflow doesn't require you to think or work in those terms at all.

zaphirplane 3y ago

AWS has services around pipeline and deploy, right ?